2024-04-26T23:19:13.395 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-26T23:19:13.395 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/dbench}} --name rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi --owner scheduled_rishabh@teuthology --archive /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675308 --verbose -- /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675308/orig.config.yaml 2024-04-26T23:19:13.454 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-26T23:19:13.556 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675308 branch: wip-rishabh-testing-20240426.111959 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/dbench}} email: null first_in_suite: false job_id: '7675308' kernel: client: branch: testing flavor: default kdb: true sha1: 806ec36b21129f113ce356c007dd59694d340b37 ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-rishabh-testing-20240426.111959 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 subvols: create: 2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported install: ceph: flavor: default sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - wsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-rishabh-testing-20240426.111959 sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 owner: scheduled_rishabh@teuthology priority: 50 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 816 sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 sleep_before_teardown: 0 subset: 166/512 suite: fs suite_branch: wip-rishabh-testing-20240426.111959 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_bc612ffb9c1d5bc27536bc917a705262a36d4387/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi072.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJjeDVEDNe31QcgWdx6OLTtV1zAuB+DPHQaJ6j6g+/nAjD/dAadH8llC8ys0ClkWFVLrJJhBtkVWheGRksqmgtw= smithi076.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEgNrepYTRHeeVzBI/2EyIIxQBwE9tGQOIgZ019w8WakdlkOJkd1u3IajgCWpBvUlces3W9hZ7ta9NssCwE0E3M= smithi136.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHVtaGF43LF2b17nJkPADW9nsUJW4D5atFZYbgIcPrLZEgZwPauh8XXD1V0qvGK42lngRI4D2Aj4wyEu35o1OKw= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/dbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-26_19:30:57 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-26T23:19:13.557 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_bc612ffb9c1d5bc27536bc917a705262a36d4387/qa; will attempt to use it 2024-04-26T23:19:13.557 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_bc612ffb9c1d5bc27536bc917a705262a36d4387/qa/tasks 2024-04-26T23:19:13.558 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-26T23:19:13.564 INFO:teuthology.task.internal:Checking packages... 2024-04-26T23:19:13.596 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'bc612ffb9c1d5bc27536bc917a705262a36d4387' 2024-04-26T23:19:13.596 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-26T23:19:13.596 INFO:teuthology.packaging:ref: None 2024-04-26T23:19:13.596 INFO:teuthology.packaging:tag: None 2024-04-26T23:19:13.596 INFO:teuthology.packaging:branch: wip-rishabh-testing-20240426.111959 2024-04-26T23:19:13.596 INFO:teuthology.packaging:sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T23:19:13.597 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-rishabh-testing-20240426.111959 2024-04-26T23:19:13.834 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3306.gbc612ffb 2024-04-26T23:19:13.836 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-26T23:19:13.849 INFO:teuthology.task.internal:no buildpackages task found 2024-04-26T23:19:13.849 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-26T23:19:13.867 INFO:teuthology.task.internal:Saving configuration 2024-04-26T23:19:13.895 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-26T23:19:13.907 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T23:19:13.933 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi072.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675308', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 23:13:52.638673', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJjeDVEDNe31QcgWdx6OLTtV1zAuB+DPHQaJ6j6g+/nAjD/dAadH8llC8ys0ClkWFVLrJJhBtkVWheGRksqmgtw='} 2024-04-26T23:19:13.955 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi076.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675308', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 23:13:52.645357', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEgNrepYTRHeeVzBI/2EyIIxQBwE9tGQOIgZ019w8WakdlkOJkd1u3IajgCWpBvUlces3W9hZ7ta9NssCwE0E3M='} 2024-04-26T23:19:13.978 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi136.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675308', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 23:13:52.641960', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHVtaGF43LF2b17nJkPADW9nsUJW4D5atFZYbgIcPrLZEgZwPauh8XXD1V0qvGK42lngRI4D2Aj4wyEu35o1OKw='} 2024-04-26T23:19:13.978 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-26T23:19:14.074 INFO:teuthology.task.internal:roles: ubuntu@smithi072.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-26T23:19:14.074 INFO:teuthology.task.internal:roles: ubuntu@smithi076.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-26T23:19:14.074 INFO:teuthology.task.internal:roles: ubuntu@smithi136.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-26T23:19:14.075 INFO:teuthology.run_tasks:Running task console_log... 2024-04-26T23:19:14.205 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fda21201160>, signals=[15]) 2024-04-26T23:19:14.205 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-26T23:19:14.218 INFO:teuthology.task.internal:Opening connections... 2024-04-26T23:19:14.218 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi072.front.sepia.ceph.com 2024-04-26T23:19:14.220 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T23:19:14.305 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi076.front.sepia.ceph.com 2024-04-26T23:19:14.306 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T23:19:14.386 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi136.front.sepia.ceph.com 2024-04-26T23:19:14.387 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T23:19:14.462 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-26T23:19:14.472 DEBUG:teuthology.orchestra.run.smithi072:> uname -m 2024-04-26T23:19:14.490 INFO:teuthology.orchestra.run.smithi072.stdout:x86_64 2024-04-26T23:19:14.491 DEBUG:teuthology.orchestra.run.smithi072:> cat /etc/os-release 2024-04-26T23:19:14.547 INFO:teuthology.orchestra.run.smithi072.stdout:NAME="CentOS Stream" 2024-04-26T23:19:14.547 INFO:teuthology.orchestra.run.smithi072.stdout:VERSION="9" 2024-04-26T23:19:14.547 INFO:teuthology.orchestra.run.smithi072.stdout:ID="centos" 2024-04-26T23:19:14.547 INFO:teuthology.orchestra.run.smithi072.stdout:ID_LIKE="rhel fedora" 2024-04-26T23:19:14.547 INFO:teuthology.orchestra.run.smithi072.stdout:VERSION_ID="9" 2024-04-26T23:19:14.548 INFO:teuthology.orchestra.run.smithi072.stdout:PLATFORM_ID="platform:el9" 2024-04-26T23:19:14.548 INFO:teuthology.orchestra.run.smithi072.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-26T23:19:14.548 INFO:teuthology.orchestra.run.smithi072.stdout:ANSI_COLOR="0;31" 2024-04-26T23:19:14.548 INFO:teuthology.orchestra.run.smithi072.stdout:LOGO="fedora-logo-icon" 2024-04-26T23:19:14.548 INFO:teuthology.orchestra.run.smithi072.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-26T23:19:14.548 INFO:teuthology.orchestra.run.smithi072.stdout:HOME_URL="https://centos.org/" 2024-04-26T23:19:14.548 INFO:teuthology.orchestra.run.smithi072.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-26T23:19:14.548 INFO:teuthology.orchestra.run.smithi072.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-26T23:19:14.548 INFO:teuthology.orchestra.run.smithi072.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-26T23:19:14.549 INFO:teuthology.lock.ops:Updating smithi072.front.sepia.ceph.com on lock server 2024-04-26T23:19:14.574 DEBUG:teuthology.orchestra.run.smithi076:> uname -m 2024-04-26T23:19:14.591 INFO:teuthology.orchestra.run.smithi076.stdout:x86_64 2024-04-26T23:19:14.591 DEBUG:teuthology.orchestra.run.smithi076:> cat /etc/os-release 2024-04-26T23:19:14.649 INFO:teuthology.orchestra.run.smithi076.stdout:NAME="CentOS Stream" 2024-04-26T23:19:14.650 INFO:teuthology.orchestra.run.smithi076.stdout:VERSION="9" 2024-04-26T23:19:14.650 INFO:teuthology.orchestra.run.smithi076.stdout:ID="centos" 2024-04-26T23:19:14.650 INFO:teuthology.orchestra.run.smithi076.stdout:ID_LIKE="rhel fedora" 2024-04-26T23:19:14.651 INFO:teuthology.orchestra.run.smithi076.stdout:VERSION_ID="9" 2024-04-26T23:19:14.651 INFO:teuthology.orchestra.run.smithi076.stdout:PLATFORM_ID="platform:el9" 2024-04-26T23:19:14.651 INFO:teuthology.orchestra.run.smithi076.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-26T23:19:14.651 INFO:teuthology.orchestra.run.smithi076.stdout:ANSI_COLOR="0;31" 2024-04-26T23:19:14.651 INFO:teuthology.orchestra.run.smithi076.stdout:LOGO="fedora-logo-icon" 2024-04-26T23:19:14.651 INFO:teuthology.orchestra.run.smithi076.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-26T23:19:14.651 INFO:teuthology.orchestra.run.smithi076.stdout:HOME_URL="https://centos.org/" 2024-04-26T23:19:14.652 INFO:teuthology.orchestra.run.smithi076.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-26T23:19:14.652 INFO:teuthology.orchestra.run.smithi076.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-26T23:19:14.652 INFO:teuthology.orchestra.run.smithi076.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-26T23:19:14.652 INFO:teuthology.lock.ops:Updating smithi076.front.sepia.ceph.com on lock server 2024-04-26T23:19:14.673 DEBUG:teuthology.orchestra.run.smithi136:> uname -m 2024-04-26T23:19:14.691 INFO:teuthology.orchestra.run.smithi136.stdout:x86_64 2024-04-26T23:19:14.691 DEBUG:teuthology.orchestra.run.smithi136:> cat /etc/os-release 2024-04-26T23:19:14.749 INFO:teuthology.orchestra.run.smithi136.stdout:NAME="CentOS Stream" 2024-04-26T23:19:14.749 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION="9" 2024-04-26T23:19:14.749 INFO:teuthology.orchestra.run.smithi136.stdout:ID="centos" 2024-04-26T23:19:14.749 INFO:teuthology.orchestra.run.smithi136.stdout:ID_LIKE="rhel fedora" 2024-04-26T23:19:14.749 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION_ID="9" 2024-04-26T23:19:14.749 INFO:teuthology.orchestra.run.smithi136.stdout:PLATFORM_ID="platform:el9" 2024-04-26T23:19:14.749 INFO:teuthology.orchestra.run.smithi136.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-26T23:19:14.749 INFO:teuthology.orchestra.run.smithi136.stdout:ANSI_COLOR="0;31" 2024-04-26T23:19:14.750 INFO:teuthology.orchestra.run.smithi136.stdout:LOGO="fedora-logo-icon" 2024-04-26T23:19:14.750 INFO:teuthology.orchestra.run.smithi136.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-26T23:19:14.750 INFO:teuthology.orchestra.run.smithi136.stdout:HOME_URL="https://centos.org/" 2024-04-26T23:19:14.750 INFO:teuthology.orchestra.run.smithi136.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-26T23:19:14.750 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-26T23:19:14.750 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-26T23:19:14.750 INFO:teuthology.lock.ops:Updating smithi136.front.sepia.ceph.com on lock server 2024-04-26T23:19:14.765 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-26T23:19:14.780 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-26T23:19:14.790 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-26T23:19:14.790 DEBUG:teuthology.orchestra.run.smithi072:> test '!' -e /home/ubuntu/cephtest 2024-04-26T23:19:14.793 DEBUG:teuthology.orchestra.run.smithi076:> test '!' -e /home/ubuntu/cephtest 2024-04-26T23:19:14.795 DEBUG:teuthology.orchestra.run.smithi136:> test '!' -e /home/ubuntu/cephtest 2024-04-26T23:19:14.811 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-26T23:19:14.853 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-26T23:19:14.853 DEBUG:teuthology.orchestra.run.smithi072:> test -z $(ls -A /var/lib/ceph) 2024-04-26T23:19:14.856 DEBUG:teuthology.orchestra.run.smithi076:> test -z $(ls -A /var/lib/ceph) 2024-04-26T23:19:14.858 DEBUG:teuthology.orchestra.run.smithi136:> test -z $(ls -A /var/lib/ceph) 2024-04-26T23:19:14.965 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-26T23:19:15.036 INFO:teuthology.run_tasks:Running task kernel... 2024-04-26T23:19:15.056 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '806ec36b21129f113ce356c007dd59694d340b37'} 2024-04-26T23:19:15.056 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-04-26T23:19:15.056 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-04-26T23:19:15.057 DEBUG:teuthology.orchestra.run.smithi072:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T23:19:15.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T23:19:15.075 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-04-26T23:19:15.251 DEBUG:teuthology.task.kernel:sha1 for client.0 is 806ec36b21129f113ce356c007dd59694d340b37 2024-04-26T23:19:15.251 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "806ec36b21129f113ce356c007dd59694d340b37"... 2024-04-26T23:19:15.251 DEBUG:teuthology.orchestra.run.smithi072:> uname -r 2024-04-26T23:19:15.269 INFO:teuthology.orchestra.run.smithi072.stdout:5.14.0-437.el9.x86_64 2024-04-26T23:19:15.269 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs 806ec36b21129f113ce356c007dd59694d340b37 2024-04-26T23:19:15.269 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-04-26T23:19:15.270 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum upgrade -y linux-firmware 2024-04-26T23:19:18.880 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - BaseOS 6.5 MB/s | 8.0 MB 00:01 2024-04-26T23:19:38.472 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - AppStream 1.0 MB/s | 19 MB 00:18 2024-04-26T23:19:42.558 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.2 MB 00:00 2024-04-26T23:19:44.219 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 16 kB 00:00 2024-04-26T23:19:46.425 INFO:teuthology.orchestra.run.smithi072.stdout:Extra Packages for Enterprise Linux 10 MB/s | 21 MB 00:02 2024-04-26T23:19:51.432 INFO:teuthology.orchestra.run.smithi072.stdout:lab-extras 42 kB/s | 1.7 kB 00:00 2024-04-26T23:19:53.398 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-04-26T23:19:53.399 INFO:teuthology.orchestra.run.smithi072.stdout:Nothing to do. 2024-04-26T23:19:53.399 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-04-26T23:19:53.527 INFO:teuthology.task.kernel:Downloading kernel 806ec36b21129f113ce356c007dd59694d340b37 on client.0... 2024-04-26T23:19:53.527 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=806ec36b21129f113ce356c007dd59694d340b37 2024-04-26T23:19:53.753 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ 2024-04-26T23:19:53.753 DEBUG:teuthology.orchestra.run.smithi072:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ --input-file=- 2024-04-26T23:19:53.756 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi072.front.sepia.ceph.com to complete... 2024-04-26T23:20:12.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26 23:20:12 URL:https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm [357367792/357367792] -> "/tmp/kernel.x86_64.rpm" [1] 2024-04-26T23:20:12.632 INFO:teuthology.orchestra.run.smithi072.stderr:FINISHED --2024-04-26 23:20:12-- 2024-04-26T23:20:12.632 INFO:teuthology.orchestra.run.smithi072.stderr:Total wall clock time: 19s 2024-04-26T23:20:12.632 INFO:teuthology.orchestra.run.smithi072.stderr:Downloaded: 1 files, 341M in 18s (18.6 MB/s) 2024-04-26T23:20:12.634 INFO:teuthology.task.kernel:Installing kernel 806ec36b21129f113ce356c007dd59694d340b37 on client.0... 2024-04-26T23:20:12.634 DEBUG:teuthology.orchestra.run.smithi072:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-04-26T23:20:14.728 INFO:teuthology.orchestra.run.smithi072.stdout:Verifying... ######################################## 2024-04-26T23:20:14.880 INFO:teuthology.orchestra.run.smithi072.stdout:Preparing... ######################################## 2024-04-26T23:20:14.982 INFO:teuthology.orchestra.run.smithi072.stdout:Updating / installing... 2024-04-26T23:20:22.873 INFO:teuthology.orchestra.run.smithi072.stdout:kernel-6.9.0_rc3_g806ec36b2112-1 ######################################## 2024-04-26T23:21:18.580 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi072.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-04-26T23:21:18.581 DEBUG:teuthology.orchestra.run.smithi072:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-04-26T23:21:18.674 INFO:teuthology.orchestra.run.smithi072.stdout:/boot/System.map-6.9.0-rc3-g806ec36b2112 2024-04-26T23:21:18.674 INFO:teuthology.orchestra.run.smithi072.stdout:/boot/config-6.9.0-rc3-g806ec36b2112 2024-04-26T23:21:18.674 INFO:teuthology.orchestra.run.smithi072.stdout:/boot/vmlinuz-6.9.0-rc3-g806ec36b2112 2024-04-26T23:21:18.674 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/System.map 2024-04-26T23:21:18.674 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/config 2024-04-26T23:21:18.674 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel 2024-04-26T23:21:18.674 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch 2024-04-26T23:21:18.675 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86 2024-04-26T23:21:18.675 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto 2024-04-26T23:21:18.675 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-04-26T23:21:18.675 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-04-26T23:21:18.675 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-04-26T23:21:18.675 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-04-26T23:21:18.675 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-04-26T23:21:18.675 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-04-26T23:21:18.675 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-04-26T23:21:18.675 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32c-intel.ko 2024-04-26T23:21:18.675 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-04-26T23:21:18.675 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-04-26T23:21:18.675 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-04-26T23:21:18.676 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-04-26T23:21:18.676 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx2.ko 2024-04-26T23:21:18.676 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-04-26T23:21:18.676 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-04-26T23:21:18.676 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-04-26T23:21:18.676 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-04-26T23:21:18.676 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-04-26T23:21:18.676 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-04-26T23:21:18.676 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-04-26T23:21:18.676 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel 2024-04-26T23:21:18.676 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu 2024-04-26T23:21:18.676 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce 2024-04-26T23:21:18.676 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-04-26T23:21:18.676 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm 2024-04-26T23:21:18.676 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-amd.ko 2024-04-26T23:21:18.676 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-intel.ko 2024-04-26T23:21:18.677 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm.ko 2024-04-26T23:21:18.677 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto 2024-04-26T23:21:18.677 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ansi_cprng.ko 2024-04-26T23:21:18.677 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/anubis.ko 2024-04-26T23:21:18.677 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/arc4.ko 2024-04-26T23:21:18.677 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx 2024-04-26T23:21:18.677 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_memcpy.ko 2024-04-26T23:21:18.677 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_pq.ko 2024-04-26T23:21:18.677 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_raid6_recov.ko 2024-04-26T23:21:18.677 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_tx.ko 2024-04-26T23:21:18.677 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_xor.ko 2024-04-26T23:21:18.677 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/raid6test.ko 2024-04-26T23:21:18.677 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authenc.ko 2024-04-26T23:21:18.677 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authencesn.ko 2024-04-26T23:21:18.677 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blake2b_generic.ko 2024-04-26T23:21:18.678 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_common.ko 2024-04-26T23:21:18.678 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_generic.ko 2024-04-26T23:21:18.678 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/camellia_generic.ko 2024-04-26T23:21:18.678 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast5_generic.ko 2024-04-26T23:21:18.678 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast6_generic.ko 2024-04-26T23:21:18.678 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast_common.ko 2024-04-26T23:21:18.678 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ccm.ko 2024-04-26T23:21:18.678 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cmac.ko 2024-04-26T23:21:18.678 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crc32_generic.ko 2024-04-26T23:21:18.678 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_engine.ko 2024-04-26T23:21:18.678 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_user.ko 2024-04-26T23:21:18.678 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cts.ko 2024-04-26T23:21:18.678 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/des_generic.ko 2024-04-26T23:21:18.679 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/dh_generic.ko 2024-04-26T23:21:18.679 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/echainiv.ko 2024-04-26T23:21:18.679 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/essiv.ko 2024-04-26T23:21:18.679 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/fcrypt.ko 2024-04-26T23:21:18.679 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/khazad.ko 2024-04-26T23:21:18.679 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4.ko 2024-04-26T23:21:18.679 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4hc.ko 2024-04-26T23:21:18.679 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/md4.ko 2024-04-26T23:21:18.679 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/michael_mic.ko 2024-04-26T23:21:18.679 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcbc.ko 2024-04-26T23:21:18.679 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcrypt.ko 2024-04-26T23:21:18.679 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/rmd160.ko 2024-04-26T23:21:18.679 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/seed.ko 2024-04-26T23:21:18.679 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/serpent_generic.ko 2024-04-26T23:21:18.680 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tcrypt.ko 2024-04-26T23:21:18.680 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tea.ko 2024-04-26T23:21:18.680 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_common.ko 2024-04-26T23:21:18.680 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_generic.ko 2024-04-26T23:21:18.680 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/vmac.ko 2024-04-26T23:21:18.680 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/wp512.ko 2024-04-26T23:21:18.680 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xcbc.ko 2024-04-26T23:21:18.680 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xor.ko 2024-04-26T23:21:18.680 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xxhash_generic.ko 2024-04-26T23:21:18.680 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers 2024-04-26T23:21:18.680 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi 2024-04-26T23:21:18.680 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_ipmi.ko 2024-04-26T23:21:18.680 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_pad.ko 2024-04-26T23:21:18.680 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/ec_sys.ko 2024-04-26T23:21:18.680 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/platform_profile.ko 2024-04-26T23:21:18.680 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbs.ko 2024-04-26T23:21:18.681 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbshc.ko 2024-04-26T23:21:18.681 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/video.ko 2024-04-26T23:21:18.681 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata 2024-04-26T23:21:18.681 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/acard-ahci.ko 2024-04-26T23:21:18.681 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ahci_platform.ko 2024-04-26T23:21:18.681 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ata_generic.ko 2024-04-26T23:21:18.681 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/libahci_platform.ko 2024-04-26T23:21:18.681 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_acpi.ko 2024-04-26T23:21:18.681 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ali.ko 2024-04-26T23:21:18.681 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_amd.ko 2024-04-26T23:21:18.681 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_artop.ko 2024-04-26T23:21:18.681 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atiixp.ko 2024-04-26T23:21:18.681 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atp867x.ko 2024-04-26T23:21:18.681 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd640.ko 2024-04-26T23:21:18.681 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd64x.ko 2024-04-26T23:21:18.682 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cypress.ko 2024-04-26T23:21:18.682 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_efar.ko 2024-04-26T23:21:18.682 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt366.ko 2024-04-26T23:21:18.682 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt37x.ko 2024-04-26T23:21:18.682 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x2n.ko 2024-04-26T23:21:18.682 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x3.ko 2024-04-26T23:21:18.682 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it8213.ko 2024-04-26T23:21:18.682 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it821x.ko 2024-04-26T23:21:18.682 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_jmicron.ko 2024-04-26T23:21:18.682 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_marvell.ko 2024-04-26T23:21:18.682 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_mpiix.ko 2024-04-26T23:21:18.682 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_netcell.ko 2024-04-26T23:21:18.682 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ninja32.ko 2024-04-26T23:21:18.683 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87410.ko 2024-04-26T23:21:18.683 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87415.ko 2024-04-26T23:21:18.683 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_oldpiix.ko 2024-04-26T23:21:18.683 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_opti.ko 2024-04-26T23:21:18.683 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_optidma.ko 2024-04-26T23:21:18.683 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pcmcia.ko 2024-04-26T23:21:18.683 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc2027x.ko 2024-04-26T23:21:18.683 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-04-26T23:21:18.683 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_piccolo.ko 2024-04-26T23:21:18.683 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_rdc.ko 2024-04-26T23:21:18.683 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sch.ko 2024-04-26T23:21:18.683 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_serverworks.ko 2024-04-26T23:21:18.683 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sil680.ko 2024-04-26T23:21:18.683 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sis.ko 2024-04-26T23:21:18.683 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sl82c105.ko 2024-04-26T23:21:18.684 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_triflex.ko 2024-04-26T23:21:18.684 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_via.ko 2024-04-26T23:21:18.684 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pdc_adma.ko 2024-04-26T23:21:18.684 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_inic162x.ko 2024-04-26T23:21:18.684 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_mv.ko 2024-04-26T23:21:18.684 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_nv.ko 2024-04-26T23:21:18.684 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_promise.ko 2024-04-26T23:21:18.684 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_qstor.ko 2024-04-26T23:21:18.684 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil.ko 2024-04-26T23:21:18.684 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil24.ko 2024-04-26T23:21:18.684 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sis.ko 2024-04-26T23:21:18.684 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_svw.ko 2024-04-26T23:21:18.684 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sx4.ko 2024-04-26T23:21:18.684 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_uli.ko 2024-04-26T23:21:18.685 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_via.ko 2024-04-26T23:21:18.685 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_vsc.ko 2024-04-26T23:21:18.685 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay 2024-04-26T23:21:18.685 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay/ks0108.ko 2024-04-26T23:21:18.685 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base 2024-04-26T23:21:18.685 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap 2024-04-26T23:21:18.685 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-i2c.ko 2024-04-26T23:21:18.685 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-mmio.ko 2024-04-26T23:21:18.685 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma 2024-04-26T23:21:18.685 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma/bcma.ko 2024-04-26T23:21:18.685 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block 2024-04-26T23:21:18.685 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe 2024-04-26T23:21:18.685 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe/aoe.ko 2024-04-26T23:21:18.685 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/brd.ko 2024-04-26T23:21:18.686 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd 2024-04-26T23:21:18.686 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd/drbd.ko 2024-04-26T23:21:18.686 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/floppy.ko 2024-04-26T23:21:18.686 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/loop.ko 2024-04-26T23:21:18.686 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx 2024-04-26T23:21:18.686 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-04-26T23:21:18.686 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/nbd.ko 2024-04-26T23:21:18.686 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk 2024-04-26T23:21:18.686 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk/null_blk.ko 2024-04-26T23:21:18.686 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/pktcdvd.ko 2024-04-26T23:21:18.686 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/rbd.ko 2024-04-26T23:21:18.686 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/virtio_blk.ko 2024-04-26T23:21:18.686 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback 2024-04-26T23:21:18.686 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-04-26T23:21:18.686 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkfront.ko 2024-04-26T23:21:18.687 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram 2024-04-26T23:21:18.687 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram/zram.ko 2024-04-26T23:21:18.687 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char 2024-04-26T23:21:18.687 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hangcheck-timer.ko 2024-04-26T23:21:18.687 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random 2024-04-26T23:21:18.687 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/amd-rng.ko 2024-04-26T23:21:18.687 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/intel-rng.ko 2024-04-26T23:21:18.687 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-04-26T23:21:18.687 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/via-rng.ko 2024-04-26T23:21:18.687 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/virtio-rng.ko 2024-04-26T23:21:18.687 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi 2024-04-26T23:21:18.687 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-04-26T23:21:18.687 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-04-26T23:21:18.687 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-04-26T23:21:18.687 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_si.ko 2024-04-26T23:21:18.687 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-04-26T23:21:18.688 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-04-26T23:21:18.688 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/lp.ko 2024-04-26T23:21:18.688 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave 2024-04-26T23:21:18.688 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave/mwave.ko 2024-04-26T23:21:18.688 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ppdev.ko 2024-04-26T23:21:18.688 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tlclk.ko 2024-04-26T23:21:18.688 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm 2024-04-26T23:21:18.688 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm.ko 2024-04-26T23:21:18.688 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_atmel.ko 2024-04-26T23:21:18.688 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_infineon.ko 2024-04-26T23:21:18.688 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_nsc.ko 2024-04-26T23:21:18.688 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis.ko 2024-04-26T23:21:18.688 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-04-26T23:21:18.688 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/uv_mmtimer.ko 2024-04-26T23:21:18.688 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/virtio_console.ko 2024-04-26T23:21:18.688 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq 2024-04-26T23:21:18.689 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-04-26T23:21:18.689 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-04-26T23:21:18.689 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/p4-clockmod.ko 2024-04-26T23:21:18.689 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-04-26T23:21:18.689 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/powernow-k8.ko 2024-04-26T23:21:18.689 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/speedstep-lib.ko 2024-04-26T23:21:18.689 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto 2024-04-26T23:21:18.689 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp 2024-04-26T23:21:18.689 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-04-26T23:21:18.689 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp.ko 2024-04-26T23:21:18.689 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel 2024-04-26T23:21:18.689 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat 2024-04-26T23:21:18.689 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common 2024-04-26T23:21:18.689 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-04-26T23:21:18.689 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-04-26T23:21:18.690 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-04-26T23:21:18.690 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-aes.ko 2024-04-26T23:21:18.690 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-sha.ko 2024-04-26T23:21:18.690 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio 2024-04-26T23:21:18.690 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-04-26T23:21:18.690 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca 2024-04-26T23:21:18.690 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca/dca.ko 2024-04-26T23:21:18.690 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq 2024-04-26T23:21:18.690 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-04-26T23:21:18.690 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma 2024-04-26T23:21:18.690 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw 2024-04-26T23:21:18.690 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw/dw_dmac.ko 2024-04-26T23:21:18.690 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat 2024-04-26T23:21:18.690 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat/ioatdma.ko 2024-04-26T23:21:18.691 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac 2024-04-26T23:21:18.691 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/amd64_edac.ko 2024-04-26T23:21:18.691 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/e752x_edac.ko 2024-04-26T23:21:18.691 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/edac_mce_amd.ko 2024-04-26T23:21:18.691 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3000_edac.ko 2024-04-26T23:21:18.691 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3200_edac.ko 2024-04-26T23:21:18.691 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5100_edac.ko 2024-04-26T23:21:18.691 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5400_edac.ko 2024-04-26T23:21:18.691 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7300_edac.ko 2024-04-26T23:21:18.691 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7core_edac.ko 2024-04-26T23:21:18.691 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i82975x_edac.ko 2024-04-26T23:21:18.691 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/ie31200_edac.ko 2024-04-26T23:21:18.691 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/sb_edac.ko 2024-04-26T23:21:18.691 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/x38_edac.ko 2024-04-26T23:21:18.691 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware 2024-04-26T23:21:18.692 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/edd.ko 2024-04-26T23:21:18.692 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/iscsi_ibft.ko 2024-04-26T23:21:18.692 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio 2024-04-26T23:21:18.692 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-ich.ko 2024-04-26T23:21:18.692 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-viperboard.ko 2024-04-26T23:21:18.692 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid 2024-04-26T23:21:18.692 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-appleir.ko 2024-04-26T23:21:18.692 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-aureal.ko 2024-04-26T23:21:18.692 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-axff.ko 2024-04-26T23:21:18.692 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-dr.ko 2024-04-26T23:21:18.692 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elecom.ko 2024-04-26T23:21:18.692 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elo.ko 2024-04-26T23:21:18.692 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-emsff.ko 2024-04-26T23:21:18.692 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gaff.ko 2024-04-26T23:21:18.692 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gt683r.ko 2024-04-26T23:21:18.693 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gyration.ko 2024-04-26T23:21:18.693 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-kbd.ko 2024-04-26T23:21:18.693 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-mouse.ko 2024-04-26T23:21:18.693 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtekff.ko 2024-04-26T23:21:18.693 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-hyperv.ko 2024-04-26T23:21:18.693 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-icade.ko 2024-04-26T23:21:18.693 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-keytouch.ko 2024-04-26T23:21:18.693 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-kye.ko 2024-04-26T23:21:18.693 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lcpower.ko 2024-04-26T23:21:18.693 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-led.ko 2024-04-26T23:21:18.693 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lenovo.ko 2024-04-26T23:21:18.693 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-dj.ko 2024-04-26T23:21:18.693 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-04-26T23:21:18.693 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-multitouch.ko 2024-04-26T23:21:18.693 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-ortek.ko 2024-04-26T23:21:18.694 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-penmount.ko 2024-04-26T23:21:18.694 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-petalynx.ko 2024-04-26T23:21:18.694 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-picolcd.ko 2024-04-26T23:21:18.694 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-pl.ko 2024-04-26T23:21:18.694 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-plantronics.ko 2024-04-26T23:21:18.694 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-primax.ko 2024-04-26T23:21:18.694 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-rmi.ko 2024-04-26T23:21:18.694 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-arvo.ko 2024-04-26T23:21:18.694 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-common.ko 2024-04-26T23:21:18.694 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-isku.ko 2024-04-26T23:21:18.694 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kone.ko 2024-04-26T23:21:18.694 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-04-26T23:21:18.694 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-konepure.ko 2024-04-26T23:21:18.694 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-04-26T23:21:18.695 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-lua.ko 2024-04-26T23:21:18.695 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-pyra.ko 2024-04-26T23:21:18.695 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-ryos.ko 2024-04-26T23:21:18.695 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-savu.ko 2024-04-26T23:21:18.695 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat.ko 2024-04-26T23:21:18.695 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-saitek.ko 2024-04-26T23:21:18.695 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-samsung.ko 2024-04-26T23:21:18.695 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sensor-hub.ko 2024-04-26T23:21:18.695 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sjoy.ko 2024-04-26T23:21:18.695 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sony.ko 2024-04-26T23:21:18.695 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-speedlink.ko 2024-04-26T23:21:18.695 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-steelseries.ko 2024-04-26T23:21:18.695 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sunplus.ko 2024-04-26T23:21:18.695 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-thrustmaster.ko 2024-04-26T23:21:18.696 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tivo.ko 2024-04-26T23:21:18.696 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tmff.ko 2024-04-26T23:21:18.696 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-topseed.ko 2024-04-26T23:21:18.696 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-twinhan.ko 2024-04-26T23:21:18.696 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-uclogic.ko 2024-04-26T23:21:18.696 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-waltop.ko 2024-04-26T23:21:18.696 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-wiimote.ko 2024-04-26T23:21:18.696 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-xinmo.ko 2024-04-26T23:21:18.696 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zpff.ko 2024-04-26T23:21:18.696 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zydacron.ko 2024-04-26T23:21:18.696 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/uhid.ko 2024-04-26T23:21:18.696 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/wacom.ko 2024-04-26T23:21:18.696 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv 2024-04-26T23:21:18.696 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_balloon.ko 2024-04-26T23:21:18.696 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_utils.ko 2024-04-26T23:21:18.697 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_vmbus.ko 2024-04-26T23:21:18.697 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon 2024-04-26T23:21:18.697 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru.ko 2024-04-26T23:21:18.697 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru3.ko 2024-04-26T23:21:18.697 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/acpi_power_meter.ko 2024-04-26T23:21:18.697 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7414.ko 2024-04-26T23:21:18.697 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7418.ko 2024-04-26T23:21:18.697 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adc128d818.ko 2024-04-26T23:21:18.697 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1025.ko 2024-04-26T23:21:18.697 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1026.ko 2024-04-26T23:21:18.697 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1029.ko 2024-04-26T23:21:18.697 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1031.ko 2024-04-26T23:21:18.697 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm9240.ko 2024-04-26T23:21:18.697 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ads7828.ko 2024-04-26T23:21:18.697 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7410.ko 2024-04-26T23:21:18.698 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7411.ko 2024-04-26T23:21:18.698 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7462.ko 2024-04-26T23:21:18.698 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7470.ko 2024-04-26T23:21:18.698 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7475.ko 2024-04-26T23:21:18.698 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7x10.ko 2024-04-26T23:21:18.698 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/amc6821.ko 2024-04-26T23:21:18.698 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/applesmc.ko 2024-04-26T23:21:18.698 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asb100.ko 2024-04-26T23:21:18.698 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asc7621.ko 2024-04-26T23:21:18.698 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asus_atk0110.ko 2024-04-26T23:21:18.698 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/atxp1.ko 2024-04-26T23:21:18.698 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/coretemp.ko 2024-04-26T23:21:18.698 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-04-26T23:21:18.699 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dme1737.ko 2024-04-26T23:21:18.699 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds1621.ko 2024-04-26T23:21:18.699 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds620.ko 2024-04-26T23:21:18.699 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc1403.ko 2024-04-26T23:21:18.699 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc6w201.ko 2024-04-26T23:21:18.699 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71805f.ko 2024-04-26T23:21:18.699 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71882fg.ko 2024-04-26T23:21:18.699 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f75375s.ko 2024-04-26T23:21:18.699 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fam15h_power.ko 2024-04-26T23:21:18.699 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fschmd.ko 2024-04-26T23:21:18.699 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g760a.ko 2024-04-26T23:21:18.699 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g762.ko 2024-04-26T23:21:18.699 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl518sm.ko 2024-04-26T23:21:18.699 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl520sm.ko 2024-04-26T23:21:18.700 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/hwmon-vid.ko 2024-04-26T23:21:18.700 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5500_temp.ko 2024-04-26T23:21:18.700 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5k_amb.ko 2024-04-26T23:21:18.700 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmaem.ko 2024-04-26T23:21:18.700 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmpex.ko 2024-04-26T23:21:18.700 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina209.ko 2024-04-26T23:21:18.700 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina2xx.ko 2024-04-26T23:21:18.700 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/it87.ko 2024-04-26T23:21:18.700 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k10temp.ko 2024-04-26T23:21:18.700 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k8temp.ko 2024-04-26T23:21:18.700 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lineage-pem.ko 2024-04-26T23:21:18.700 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm63.ko 2024-04-26T23:21:18.700 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm73.ko 2024-04-26T23:21:18.700 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm75.ko 2024-04-26T23:21:18.701 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm77.ko 2024-04-26T23:21:18.701 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm78.ko 2024-04-26T23:21:18.701 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm80.ko 2024-04-26T23:21:18.701 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm83.ko 2024-04-26T23:21:18.701 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm85.ko 2024-04-26T23:21:18.701 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm87.ko 2024-04-26T23:21:18.701 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm90.ko 2024-04-26T23:21:18.701 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm92.ko 2024-04-26T23:21:18.701 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm93.ko 2024-04-26T23:21:18.701 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95234.ko 2024-04-26T23:21:18.701 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95241.ko 2024-04-26T23:21:18.701 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95245.ko 2024-04-26T23:21:18.701 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc2945.ko 2024-04-26T23:21:18.701 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4151.ko 2024-04-26T23:21:18.701 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4215.ko 2024-04-26T23:21:18.701 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4222.ko 2024-04-26T23:21:18.702 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4245.ko 2024-04-26T23:21:18.702 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4260.ko 2024-04-26T23:21:18.702 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4261.ko 2024-04-26T23:21:18.702 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max16065.ko 2024-04-26T23:21:18.702 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1619.ko 2024-04-26T23:21:18.702 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1668.ko 2024-04-26T23:21:18.702 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max197.ko 2024-04-26T23:21:18.702 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6639.ko 2024-04-26T23:21:18.702 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6650.ko 2024-04-26T23:21:18.702 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6697.ko 2024-04-26T23:21:18.702 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/mcp3021.ko 2024-04-26T23:21:18.702 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6683.ko 2024-04-26T23:21:18.702 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775-core.ko 2024-04-26T23:21:18.703 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775.ko 2024-04-26T23:21:18.703 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct7802.ko 2024-04-26T23:21:18.703 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87360.ko 2024-04-26T23:21:18.703 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87427.ko 2024-04-26T23:21:18.703 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pcf8591.ko 2024-04-26T23:21:18.703 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus 2024-04-26T23:21:18.703 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-04-26T23:21:18.703 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-04-26T23:21:18.703 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-04-26T23:21:18.703 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max16064.ko 2024-04-26T23:21:18.703 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max34440.ko 2024-04-26T23:21:18.703 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max8688.ko 2024-04-26T23:21:18.703 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-04-26T23:21:18.703 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-04-26T23:21:18.704 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-04-26T23:21:18.704 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-04-26T23:21:18.704 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-04-26T23:21:18.704 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-04-26T23:21:18.704 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/powr1220.ko 2024-04-26T23:21:18.704 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5627.ko 2024-04-26T23:21:18.704 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5636.ko 2024-04-26T23:21:18.704 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch56xx-common.ko 2024-04-26T23:21:18.704 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht15.ko 2024-04-26T23:21:18.704 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht21.ko 2024-04-26T23:21:18.704 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/shtc1.ko 2024-04-26T23:21:18.704 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sis5595.ko 2024-04-26T23:21:18.704 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47b397.ko 2024-04-26T23:21:18.704 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m1.ko 2024-04-26T23:21:18.704 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m192.ko 2024-04-26T23:21:18.705 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/thmc50.ko 2024-04-26T23:21:18.705 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp102.ko 2024-04-26T23:21:18.705 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp103.ko 2024-04-26T23:21:18.705 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp401.ko 2024-04-26T23:21:18.705 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp421.ko 2024-04-26T23:21:18.705 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via-cputemp.ko 2024-04-26T23:21:18.705 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via686a.ko 2024-04-26T23:21:18.705 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt1211.ko 2024-04-26T23:21:18.705 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt8231.ko 2024-04-26T23:21:18.705 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627ehf.ko 2024-04-26T23:21:18.705 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627hf.ko 2024-04-26T23:21:18.705 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83781d.ko 2024-04-26T23:21:18.705 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83791d.ko 2024-04-26T23:21:18.705 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83792d.ko 2024-04-26T23:21:18.705 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83793.ko 2024-04-26T23:21:18.706 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83795.ko 2024-04-26T23:21:18.706 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l785ts.ko 2024-04-26T23:21:18.706 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l786ng.ko 2024-04-26T23:21:18.706 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c 2024-04-26T23:21:18.706 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos 2024-04-26T23:21:18.706 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-04-26T23:21:18.706 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-04-26T23:21:18.706 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses 2024-04-26T23:21:18.706 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-04-26T23:21:18.706 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-04-26T23:21:18.706 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-04-26T23:21:18.706 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-04-26T23:21:18.706 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-04-26T23:21:18.707 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-04-26T23:21:18.707 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-04-26T23:21:18.707 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-04-26T23:21:18.707 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-i801.ko 2024-04-26T23:21:18.707 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-isch.ko 2024-04-26T23:21:18.707 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-04-26T23:21:18.707 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-04-26T23:21:18.707 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-04-26T23:21:18.707 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-parport.ko 2024-04-26T23:21:18.707 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-04-26T23:21:18.707 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-04-26T23:21:18.707 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-04-26T23:21:18.707 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-04-26T23:21:18.707 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-04-26T23:21:18.708 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-04-26T23:21:18.708 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-via.ko 2024-04-26T23:21:18.708 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-04-26T23:21:18.708 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-04-26T23:21:18.708 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-dev.ko 2024-04-26T23:21:18.708 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-mux.ko 2024-04-26T23:21:18.708 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-04-26T23:21:18.708 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-smbus.ko 2024-04-26T23:21:18.708 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-stub.ko 2024-04-26T23:21:18.708 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband 2024-04-26T23:21:18.708 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core 2024-04-26T23:21:18.708 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_cm.ko 2024-04-26T23:21:18.708 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_core.ko 2024-04-26T23:21:18.708 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_umad.ko 2024-04-26T23:21:18.708 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-04-26T23:21:18.709 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/iw_cm.ko 2024-04-26T23:21:18.709 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_cm.ko 2024-04-26T23:21:18.709 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-04-26T23:21:18.709 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw 2024-04-26T23:21:18.709 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4 2024-04-26T23:21:18.709 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-04-26T23:21:18.709 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4 2024-04-26T23:21:18.709 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-04-26T23:21:18.709 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5 2024-04-26T23:21:18.709 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-04-26T23:21:18.709 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca 2024-04-26T23:21:18.709 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-04-26T23:21:18.709 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp 2024-04-26T23:21:18.709 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib 2024-04-26T23:21:18.709 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-04-26T23:21:18.710 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser 2024-04-26T23:21:18.710 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-04-26T23:21:18.710 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert 2024-04-26T23:21:18.710 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-04-26T23:21:18.710 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp 2024-04-26T23:21:18.710 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-04-26T23:21:18.710 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt 2024-04-26T23:21:18.710 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-04-26T23:21:18.710 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input 2024-04-26T23:21:18.710 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport 2024-04-26T23:21:18.710 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-04-26T23:21:18.710 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/fm801-gp.ko 2024-04-26T23:21:18.710 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/gameport.ko 2024-04-26T23:21:18.711 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/joydev.ko 2024-04-26T23:21:18.711 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard 2024-04-26T23:21:18.711 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard/gpio_keys.ko 2024-04-26T23:21:18.711 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse 2024-04-26T23:21:18.711 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/appletouch.ko 2024-04-26T23:21:18.711 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/bcm5974.ko 2024-04-26T23:21:18.711 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/cyapatp.ko 2024-04-26T23:21:18.711 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/elan_i2c.ko 2024-04-26T23:21:18.711 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/sermouse.ko 2024-04-26T23:21:18.711 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-04-26T23:21:18.711 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_usb.ko 2024-04-26T23:21:18.711 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/vsxxxaa.ko 2024-04-26T23:21:18.711 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4 2024-04-26T23:21:18.711 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4/rmi_core.ko 2024-04-26T23:21:18.711 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio 2024-04-26T23:21:18.712 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/altera_ps2.ko 2024-04-26T23:21:18.712 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/arc_ps2.ko 2024-04-26T23:21:18.712 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-04-26T23:21:18.712 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/serio_raw.ko 2024-04-26T23:21:18.712 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/sparse-keymap.ko 2024-04-26T23:21:18.712 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds 2024-04-26T23:21:18.712 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-blinkm.ko 2024-04-26T23:21:18.712 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lm3530.ko 2024-04-26T23:21:18.712 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lp3944.ko 2024-04-26T23:21:18.712 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-ss4200.ko 2024-04-26T23:21:18.712 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger 2024-04-26T23:21:18.712 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-04-26T23:21:18.712 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-04-26T23:21:18.712 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-04-26T23:21:18.712 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-04-26T23:21:18.713 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-04-26T23:21:18.713 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-04-26T23:21:18.713 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-04-26T23:21:18.713 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-04-26T23:21:18.713 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-04-26T23:21:18.713 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md 2024-04-26T23:21:18.713 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache 2024-04-26T23:21:18.713 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache/bcache.ko 2024-04-26T23:21:18.713 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-bio-prison.ko 2024-04-26T23:21:18.713 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache-smq.ko 2024-04-26T23:21:18.713 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache.ko 2024-04-26T23:21:18.713 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-crypt.ko 2024-04-26T23:21:18.713 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-delay.ko 2024-04-26T23:21:18.713 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-flakey.ko 2024-04-26T23:21:18.714 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-log-userspace.ko 2024-04-26T23:21:18.714 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-multipath.ko 2024-04-26T23:21:18.714 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-queue-length.ko 2024-04-26T23:21:18.714 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-raid.ko 2024-04-26T23:21:18.714 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-round-robin.ko 2024-04-26T23:21:18.714 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-service-time.ko 2024-04-26T23:21:18.714 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-switch.ko 2024-04-26T23:21:18.714 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-thin-pool.ko 2024-04-26T23:21:18.714 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-verity.ko 2024-04-26T23:21:18.714 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data 2024-04-26T23:21:18.714 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-04-26T23:21:18.714 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid0.ko 2024-04-26T23:21:18.714 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid1.ko 2024-04-26T23:21:18.714 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid10.ko 2024-04-26T23:21:18.715 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid456.ko 2024-04-26T23:21:18.715 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message 2024-04-26T23:21:18.715 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion 2024-04-26T23:21:18.715 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptbase.ko 2024-04-26T23:21:18.715 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptctl.ko 2024-04-26T23:21:18.715 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptfc.ko 2024-04-26T23:21:18.715 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptlan.ko 2024-04-26T23:21:18.715 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptsas.ko 2024-04-26T23:21:18.715 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptscsih.ko 2024-04-26T23:21:18.715 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptspi.ko 2024-04-26T23:21:18.715 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd 2024-04-26T23:21:18.715 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_ich.ko 2024-04-26T23:21:18.715 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_sch.ko 2024-04-26T23:21:18.715 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/mfd-core.ko 2024-04-26T23:21:18.716 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/sm501.ko 2024-04-26T23:21:18.716 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/viperboard.ko 2024-04-26T23:21:18.716 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/vx855.ko 2024-04-26T23:21:18.716 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/wl1273-core.ko 2024-04-26T23:21:18.716 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc 2024-04-26T23:21:18.716 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl 2024-04-26T23:21:18.716 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-04-26T23:21:18.716 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds9802als.ko 2024-04-26T23:21:18.716 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds990x.ko 2024-04-26T23:21:18.716 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/bh1770glc.ko 2024-04-26T23:21:18.716 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710 2024-04-26T23:21:18.716 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710/cb710.ko 2024-04-26T23:21:18.716 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom 2024-04-26T23:21:18.716 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/at24.ko 2024-04-26T23:21:18.716 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-04-26T23:21:18.717 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/max6875.ko 2024-04-26T23:21:18.717 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/enclosure.ko 2024-04-26T23:21:18.717 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/hpilo.ko 2024-04-26T23:21:18.717 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29003.ko 2024-04-26T23:21:18.717 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29020.ko 2024-04-26T23:21:18.717 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d 2024-04-26T23:21:18.717 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-04-26T23:21:18.717 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-04-26T23:21:18.717 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei 2024-04-26T23:21:18.717 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-me.ko 2024-04-26T23:21:18.717 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-txe.ko 2024-04-26T23:21:18.717 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei.ko 2024-04-26T23:21:18.717 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru 2024-04-26T23:21:18.717 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru/gru.ko 2024-04-26T23:21:18.718 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp 2024-04-26T23:21:18.718 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xp.ko 2024-04-26T23:21:18.718 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpc.ko 2024-04-26T23:21:18.718 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-04-26T23:21:18.718 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_7xx1.ko 2024-04-26T23:21:18.718 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_core.ko 2024-04-26T23:21:18.718 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tsl2550.ko 2024-04-26T23:21:18.718 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_balloon.ko 2024-04-26T23:21:18.718 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci 2024-04-26T23:21:18.718 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-04-26T23:21:18.718 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net 2024-04-26T23:21:18.718 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding 2024-04-26T23:21:18.718 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding/bonding.ko 2024-04-26T23:21:18.718 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa 2024-04-26T23:21:18.719 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53 2024-04-26T23:21:18.719 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53/b53_common.ko 2024-04-26T23:21:18.719 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/bcm-sf2.ko 2024-04-26T23:21:18.719 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6060.ko 2024-04-26T23:21:18.719 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx 2024-04-26T23:21:18.719 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-04-26T23:21:18.719 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dummy.ko 2024-04-26T23:21:18.719 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/eql.ko 2024-04-26T23:21:18.719 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet 2024-04-26T23:21:18.719 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com 2024-04-26T23:21:18.719 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-04-26T23:21:18.719 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-04-26T23:21:18.719 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-04-26T23:21:18.719 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-04-26T23:21:18.719 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390 2024-04-26T23:21:18.720 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/8390.ko 2024-04-26T23:21:18.720 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-04-26T23:21:18.720 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-04-26T23:21:18.720 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-04-26T23:21:18.720 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec 2024-04-26T23:21:18.720 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-04-26T23:21:18.720 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere 2024-04-26T23:21:18.720 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere/et131x.ko 2024-04-26T23:21:18.720 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon 2024-04-26T23:21:18.720 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-04-26T23:21:18.720 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera 2024-04-26T23:21:18.720 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-04-26T23:21:18.720 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd 2024-04-26T23:21:18.720 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-04-26T23:21:18.721 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-04-26T23:21:18.721 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-04-26T23:21:18.721 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros 2024-04-26T23:21:18.721 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx 2024-04-26T23:21:18.721 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-04-26T23:21:18.721 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c 2024-04-26T23:21:18.721 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-04-26T23:21:18.721 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e 2024-04-26T23:21:18.721 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-04-26T23:21:18.721 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx 2024-04-26T23:21:18.721 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-04-26T23:21:18.721 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-04-26T23:21:18.721 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom 2024-04-26T23:21:18.721 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-04-26T23:21:18.721 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-04-26T23:21:18.722 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-04-26T23:21:18.722 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-04-26T23:21:18.722 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-04-26T23:21:18.722 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet 2024-04-26T23:21:18.722 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-04-26T23:21:18.722 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-04-26T23:21:18.722 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade 2024-04-26T23:21:18.722 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna 2024-04-26T23:21:18.722 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-04-26T23:21:18.722 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio 2024-04-26T23:21:18.722 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb 2024-04-26T23:21:18.722 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-04-26T23:21:18.722 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-04-26T23:21:18.723 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-04-26T23:21:18.723 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-04-26T23:21:18.723 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-04-26T23:21:18.723 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-04-26T23:21:18.723 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-04-26T23:21:18.723 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-04-26T23:21:18.723 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-04-26T23:21:18.723 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco 2024-04-26T23:21:18.723 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic 2024-04-26T23:21:18.723 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-04-26T23:21:18.723 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec 2024-04-26T23:21:18.723 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip 2024-04-26T23:21:18.723 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-04-26T23:21:18.723 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-04-26T23:21:18.723 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-04-26T23:21:18.724 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-04-26T23:21:18.724 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-04-26T23:21:18.724 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-04-26T23:21:18.724 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink 2024-04-26T23:21:18.724 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-04-26T23:21:18.724 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-04-26T23:21:18.724 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dnet.ko 2024-04-26T23:21:18.724 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex 2024-04-26T23:21:18.724 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet 2024-04-26T23:21:18.724 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-04-26T23:21:18.724 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ethoc.ko 2024-04-26T23:21:18.724 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/fealnx.ko 2024-04-26T23:21:18.724 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel 2024-04-26T23:21:18.724 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e100.ko 2024-04-26T23:21:18.725 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000 2024-04-26T23:21:18.725 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-04-26T23:21:18.725 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e 2024-04-26T23:21:18.725 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-04-26T23:21:18.725 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k 2024-04-26T23:21:18.725 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-04-26T23:21:18.725 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e 2024-04-26T23:21:18.725 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-04-26T23:21:18.725 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf 2024-04-26T23:21:18.725 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-04-26T23:21:18.725 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb 2024-04-26T23:21:18.725 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-04-26T23:21:18.725 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf 2024-04-26T23:21:18.725 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-04-26T23:21:18.725 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe 2024-04-26T23:21:18.726 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-04-26T23:21:18.726 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf 2024-04-26T23:21:18.726 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-04-26T23:21:18.726 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/jme.ko 2024-04-26T23:21:18.726 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell 2024-04-26T23:21:18.726 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-04-26T23:21:18.726 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/skge.ko 2024-04-26T23:21:18.726 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-04-26T23:21:18.726 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox 2024-04-26T23:21:18.726 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4 2024-04-26T23:21:18.726 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-04-26T23:21:18.726 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-04-26T23:21:18.726 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5 2024-04-26T23:21:18.726 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-04-26T23:21:18.727 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-04-26T23:21:18.727 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel 2024-04-26T23:21:18.727 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-04-26T23:21:18.727 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom 2024-04-26T23:21:18.727 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge 2024-04-26T23:21:18.727 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-04-26T23:21:18.727 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi 2024-04-26T23:21:18.727 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-04-26T23:21:18.727 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-04-26T23:21:18.727 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion 2024-04-26T23:21:18.727 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-04-26T23:21:18.727 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia 2024-04-26T23:21:18.727 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-04-26T23:21:18.727 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines 2024-04-26T23:21:18.728 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-04-26T23:21:18.728 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-04-26T23:21:18.728 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic 2024-04-26T23:21:18.728 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen 2024-04-26T23:21:18.728 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-04-26T23:21:18.728 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-04-26T23:21:18.728 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-04-26T23:21:18.728 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-04-26T23:21:18.728 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc 2024-04-26T23:21:18.728 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-04-26T23:21:18.728 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek 2024-04-26T23:21:18.728 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-04-26T23:21:18.728 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-04-26T23:21:18.728 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/atp.ko 2024-04-26T23:21:18.729 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-04-26T23:21:18.729 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker 2024-04-26T23:21:18.729 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-04-26T23:21:18.729 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc 2024-04-26T23:21:18.729 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-04-26T23:21:18.729 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan 2024-04-26T23:21:18.729 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-04-26T23:21:18.729 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis 2024-04-26T23:21:18.729 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis190.ko 2024-04-26T23:21:18.729 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis900.ko 2024-04-26T23:21:18.729 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc 2024-04-26T23:21:18.729 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-04-26T23:21:18.729 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-04-26T23:21:18.729 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-04-26T23:21:18.730 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-04-26T23:21:18.730 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro 2024-04-26T23:21:18.730 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac 2024-04-26T23:21:18.730 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-04-26T23:21:18.730 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-04-26T23:21:18.730 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun 2024-04-26T23:21:18.730 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/cassini.ko 2024-04-26T23:21:18.730 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/niu.ko 2024-04-26T23:21:18.730 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sungem.ko 2024-04-26T23:21:18.730 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-04-26T23:21:18.730 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti 2024-04-26T23:21:18.730 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-04-26T23:21:18.730 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti 2024-04-26T23:21:18.730 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti/tlan.ko 2024-04-26T23:21:18.731 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via 2024-04-26T23:21:18.731 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-04-26T23:21:18.731 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-04-26T23:21:18.731 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet 2024-04-26T23:21:18.731 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-04-26T23:21:18.731 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-04-26T23:21:18.731 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom 2024-04-26T23:21:18.731 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-04-26T23:21:18.731 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/geneve.ko 2024-04-26T23:21:18.731 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv 2024-04-26T23:21:18.731 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-04-26T23:21:18.731 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154 2024-04-26T23:21:18.731 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154/fakelb.ko 2024-04-26T23:21:18.731 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ifb.ko 2024-04-26T23:21:18.732 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan 2024-04-26T23:21:18.732 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan/ipvlan.ko 2024-04-26T23:21:18.732 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvlan.ko 2024-04-26T23:21:18.732 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvtap.ko 2024-04-26T23:21:18.732 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio 2024-04-26T23:21:18.732 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio.ko 2024-04-26T23:21:18.732 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-04-26T23:21:18.732 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-04-26T23:21:18.732 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-regmap.ko 2024-04-26T23:21:18.732 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mii.ko 2024-04-26T23:21:18.732 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/net_failover.ko 2024-04-26T23:21:18.732 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/netconsole.ko 2024-04-26T23:21:18.732 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/nlmon.ko 2024-04-26T23:21:18.733 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs 2024-04-26T23:21:18.733 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs-lynx.ko 2024-04-26T23:21:18.733 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-04-26T23:21:18.733 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy 2024-04-26T23:21:18.733 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/amd.ko 2024-04-26T23:21:18.733 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-04-26T23:21:18.733 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-04-26T23:21:18.733 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm7xxx.ko 2024-04-26T23:21:18.733 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm87xx.ko 2024-04-26T23:21:18.733 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/broadcom.ko 2024-04-26T23:21:18.733 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/cicada.ko 2024-04-26T23:21:18.733 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/davicom.ko 2024-04-26T23:21:18.733 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/dp83640.ko 2024-04-26T23:21:18.733 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/et1011c.ko 2024-04-26T23:21:18.733 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/icplus.ko 2024-04-26T23:21:18.734 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/lxt.ko 2024-04-26T23:21:18.734 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/marvell.ko 2024-04-26T23:21:18.734 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/micrel.ko 2024-04-26T23:21:18.734 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/national.ko 2024-04-26T23:21:18.734 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/phylink.ko 2024-04-26T23:21:18.734 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/qsemi.ko 2024-04-26T23:21:18.734 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/realtek.ko 2024-04-26T23:21:18.734 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/smsc.ko 2024-04-26T23:21:18.734 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/ste10Xp.ko 2024-04-26T23:21:18.734 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/vitesse.ko 2024-04-26T23:21:18.734 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp 2024-04-26T23:21:18.734 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/bsd_comp.ko 2024-04-26T23:21:18.734 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_async.ko 2024-04-26T23:21:18.735 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_deflate.ko 2024-04-26T23:21:18.735 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_generic.ko 2024-04-26T23:21:18.735 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_mppe.ko 2024-04-26T23:21:18.735 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_synctty.ko 2024-04-26T23:21:18.735 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppoe.ko 2024-04-26T23:21:18.735 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppox.ko 2024-04-26T23:21:18.735 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pptp.ko 2024-04-26T23:21:18.735 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip 2024-04-26T23:21:18.735 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slhc.ko 2024-04-26T23:21:18.735 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slip.ko 2024-04-26T23:21:18.735 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/sungem_phy.ko 2024-04-26T23:21:18.735 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tap.ko 2024-04-26T23:21:18.735 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team 2024-04-26T23:21:18.735 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team.ko 2024-04-26T23:21:18.735 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_activebackup.ko 2024-04-26T23:21:18.736 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_broadcast.ko 2024-04-26T23:21:18.736 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-04-26T23:21:18.736 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_random.ko 2024-04-26T23:21:18.736 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-04-26T23:21:18.736 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tun.ko 2024-04-26T23:21:18.736 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/veth.ko 2024-04-26T23:21:18.736 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/virtio_net.ko 2024-04-26T23:21:18.736 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3 2024-04-26T23:21:18.736 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-04-26T23:21:18.736 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan 2024-04-26T23:21:18.736 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan/vxlan.ko 2024-04-26T23:21:18.736 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback 2024-04-26T23:21:18.736 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback/xen-netback.ko 2024-04-26T23:21:18.736 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netfront.ko 2024-04-26T23:21:18.737 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb 2024-04-26T23:21:18.737 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb/ntb.ko 2024-04-26T23:21:18.737 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme 2024-04-26T23:21:18.737 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host 2024-04-26T23:21:18.737 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme-core.ko 2024-04-26T23:21:18.737 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme.ko 2024-04-26T23:21:18.737 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport 2024-04-26T23:21:18.737 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport.ko 2024-04-26T23:21:18.737 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_cs.ko 2024-04-26T23:21:18.737 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_pc.ko 2024-04-26T23:21:18.737 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_serial.ko 2024-04-26T23:21:18.737 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci 2024-04-26T23:21:18.737 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug 2024-04-26T23:21:18.737 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-04-26T23:21:18.738 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie 2024-04-26T23:21:18.738 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie/aer_inject.ko 2024-04-26T23:21:18.738 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/xen-pcifront.ko 2024-04-26T23:21:18.738 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia 2024-04-26T23:21:18.738 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/i82092.ko 2024-04-26T23:21:18.738 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/pd6729.ko 2024-04-26T23:21:18.738 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/yenta_socket.ko 2024-04-26T23:21:18.738 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl 2024-04-26T23:21:18.738 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel 2024-04-26T23:21:18.738 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-04-26T23:21:18.738 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform 2024-04-26T23:21:18.738 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86 2024-04-26T23:21:18.738 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acer-wmi.ko 2024-04-26T23:21:18.738 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acerhdf.ko 2024-04-26T23:21:18.739 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/apple-gmux.ko 2024-04-26T23:21:18.739 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-laptop.ko 2024-04-26T23:21:18.739 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-04-26T23:21:18.739 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-wmi.ko 2024-04-26T23:21:18.739 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/classmate-laptop.ko 2024-04-26T23:21:18.739 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-04-26T23:21:18.739 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-04-26T23:21:18.739 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-04-26T23:21:18.739 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-04-26T23:21:18.739 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/hdaps.ko 2024-04-26T23:21:18.739 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel 2024-04-26T23:21:18.739 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-04-26T23:21:18.739 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel_ips.ko 2024-04-26T23:21:18.739 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/msi-wmi.ko 2024-04-26T23:21:18.740 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/mxm-wmi.ko 2024-04-26T23:21:18.740 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-04-26T23:21:18.740 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-laptop.ko 2024-04-26T23:21:18.740 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-q10.ko 2024-04-26T23:21:18.740 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/topstar-laptop.ko 2024-04-26T23:21:18.740 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-04-26T23:21:18.740 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_haps.ko 2024-04-26T23:21:18.740 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi-bmof.ko 2024-04-26T23:21:18.740 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi.ko 2024-04-26T23:21:18.740 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap 2024-04-26T23:21:18.740 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_common.ko 2024-04-26T23:21:18.740 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_msr.ko 2024-04-26T23:21:18.740 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps 2024-04-26T23:21:18.740 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients 2024-04-26T23:21:18.741 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-gpio.ko 2024-04-26T23:21:18.741 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-ldisc.ko 2024-04-26T23:21:18.741 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps_parport.ko 2024-04-26T23:21:18.741 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/pps_core.ko 2024-04-26T23:21:18.741 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp 2024-04-26T23:21:18.741 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp.ko 2024-04-26T23:21:18.741 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp_kvm.ko 2024-04-26T23:21:18.741 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras 2024-04-26T23:21:18.741 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd 2024-04-26T23:21:18.741 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl 2024-04-26T23:21:18.741 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-04-26T23:21:18.741 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/fmpm.ko 2024-04-26T23:21:18.741 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc 2024-04-26T23:21:18.742 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-bq32k.ko 2024-04-26T23:21:18.742 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1286.ko 2024-04-26T23:21:18.742 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1307.ko 2024-04-26T23:21:18.742 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1374.ko 2024-04-26T23:21:18.742 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1511.ko 2024-04-26T23:21:18.742 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1553.ko 2024-04-26T23:21:18.742 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1672.ko 2024-04-26T23:21:18.742 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1742.ko 2024-04-26T23:21:18.742 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds2404.ko 2024-04-26T23:21:18.742 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds3232.ko 2024-04-26T23:21:18.742 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-em3027.ko 2024-04-26T23:21:18.742 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-fm3130.ko 2024-04-26T23:21:18.742 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl12022.ko 2024-04-26T23:21:18.742 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl1208.ko 2024-04-26T23:21:18.743 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m41t80.ko 2024-04-26T23:21:18.743 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t35.ko 2024-04-26T23:21:18.743 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t59.ko 2024-04-26T23:21:18.743 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-max6900.ko 2024-04-26T23:21:18.743 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-msm6242.ko 2024-04-26T23:21:18.743 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf2127.ko 2024-04-26T23:21:18.743 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf85063.ko 2024-04-26T23:21:18.743 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8523.ko 2024-04-26T23:21:18.743 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8563.ko 2024-04-26T23:21:18.743 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8583.ko 2024-04-26T23:21:18.743 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rp5c01.ko 2024-04-26T23:21:18.743 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rs5c372.ko 2024-04-26T23:21:18.743 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-04-26T23:21:18.743 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8025.ko 2024-04-26T23:21:18.743 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8581.ko 2024-04-26T23:21:18.744 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-04-26T23:21:18.744 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-x1205.ko 2024-04-26T23:21:18.744 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi 2024-04-26T23:21:18.744 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-9xxx.ko 2024-04-26T23:21:18.744 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-sas.ko 2024-04-26T23:21:18.744 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-xxxx.ko 2024-04-26T23:21:18.744 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/BusLogic.ko 2024-04-26T23:21:18.744 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/a100u2w.ko 2024-04-26T23:21:18.744 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid 2024-04-26T23:21:18.744 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid/aacraid.ko 2024-04-26T23:21:18.744 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/advansys.ko 2024-04-26T23:21:18.744 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx 2024-04-26T23:21:18.744 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-04-26T23:21:18.744 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-04-26T23:21:18.744 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/am53c974.ko 2024-04-26T23:21:18.745 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr 2024-04-26T23:21:18.745 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-04-26T23:21:18.745 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/atp870u.ko 2024-04-26T23:21:18.745 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi 2024-04-26T23:21:18.745 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-04-26T23:21:18.745 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa 2024-04-26T23:21:18.745 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa/bfa.ko 2024-04-26T23:21:18.745 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc 2024-04-26T23:21:18.745 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-04-26T23:21:18.745 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i 2024-04-26T23:21:18.745 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-04-26T23:21:18.745 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ch.ko 2024-04-26T23:21:18.745 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor 2024-04-26T23:21:18.745 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor/csiostor.ko 2024-04-26T23:21:18.745 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi 2024-04-26T23:21:18.746 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i 2024-04-26T23:21:18.746 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-04-26T23:21:18.746 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i 2024-04-26T23:21:18.746 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-04-26T23:21:18.746 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-04-26T23:21:18.746 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dc395x.ko 2024-04-26T23:21:18.746 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler 2024-04-26T23:21:18.746 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-04-26T23:21:18.746 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-04-26T23:21:18.746 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-04-26T23:21:18.746 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-04-26T23:21:18.746 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dmx3191d.ko 2024-04-26T23:21:18.746 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r 2024-04-26T23:21:18.746 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r/esas2r.ko 2024-04-26T23:21:18.747 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esp_scsi.ko 2024-04-26T23:21:18.747 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe 2024-04-26T23:21:18.747 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/fcoe.ko 2024-04-26T23:21:18.747 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-04-26T23:21:18.747 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic 2024-04-26T23:21:18.747 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic/fnic.ko 2024-04-26T23:21:18.747 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hpsa.ko 2024-04-26T23:21:18.747 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hptiop.ko 2024-04-26T23:21:18.747 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hv_storvsc.ko 2024-04-26T23:21:18.747 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/initio.ko 2024-04-26T23:21:18.747 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ipr.ko 2024-04-26T23:21:18.747 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ips.ko 2024-04-26T23:21:18.747 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci 2024-04-26T23:21:18.747 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci/isci.ko 2024-04-26T23:21:18.747 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-04-26T23:21:18.748 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_tcp.ko 2024-04-26T23:21:18.748 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc 2024-04-26T23:21:18.748 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc/libfc.ko 2024-04-26T23:21:18.748 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi.ko 2024-04-26T23:21:18.748 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi_tcp.ko 2024-04-26T23:21:18.748 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas 2024-04-26T23:21:18.748 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas/libsas.ko 2024-04-26T23:21:18.748 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc 2024-04-26T23:21:18.748 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc/lpfc.ko 2024-04-26T23:21:18.748 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid 2024-04-26T23:21:18.748 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid.ko 2024-04-26T23:21:18.748 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-04-26T23:21:18.748 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-04-26T23:21:18.748 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-04-26T23:21:18.748 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas 2024-04-26T23:21:18.748 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-04-26T23:21:18.749 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas 2024-04-26T23:21:18.749 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas/mvsas.ko 2024-04-26T23:21:18.749 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvumi.ko 2024-04-26T23:21:18.749 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001 2024-04-26T23:21:18.749 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-04-26T23:21:18.749 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pmcraid.ko 2024-04-26T23:21:18.749 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla1280.ko 2024-04-26T23:21:18.749 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx 2024-04-26T23:21:18.749 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-04-26T23:21:18.749 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-04-26T23:21:18.749 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx 2024-04-26T23:21:18.749 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-04-26T23:21:18.749 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/raid_class.ko 2024-04-26T23:21:18.749 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_debug.ko 2024-04-26T23:21:18.749 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_fc.ko 2024-04-26T23:21:18.750 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-04-26T23:21:18.750 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_sas.ko 2024-04-26T23:21:18.750 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_spi.ko 2024-04-26T23:21:18.750 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_srp.ko 2024-04-26T23:21:18.750 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ses.ko 2024-04-26T23:21:18.750 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/st.ko 2024-04-26T23:21:18.750 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/stex.ko 2024-04-26T23:21:18.750 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2 2024-04-26T23:21:18.750 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-04-26T23:21:18.750 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/virtio_scsi.ko 2024-04-26T23:21:18.750 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/vmw_pvscsi.ko 2024-04-26T23:21:18.750 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/wd719x.ko 2024-04-26T23:21:18.750 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb 2024-04-26T23:21:18.750 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb/ssb.ko 2024-04-26T23:21:18.750 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target 2024-04-26T23:21:18.751 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi 2024-04-26T23:21:18.751 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-04-26T23:21:18.751 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback 2024-04-26T23:21:18.751 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback/tcm_loop.ko 2024-04-26T23:21:18.751 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_file.ko 2024-04-26T23:21:18.751 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_iblock.ko 2024-04-26T23:21:18.751 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_mod.ko 2024-04-26T23:21:18.751 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_pscsi.ko 2024-04-26T23:21:18.751 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_user.ko 2024-04-26T23:21:18.751 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc 2024-04-26T23:21:18.751 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-04-26T23:21:18.751 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal 2024-04-26T23:21:18.751 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel 2024-04-26T23:21:18.751 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal 2024-04-26T23:21:18.751 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-04-26T23:21:18.751 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-04-26T23:21:18.752 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-04-26T23:21:18.752 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-04-26T23:21:18.752 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-04-26T23:21:18.752 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-04-26T23:21:18.752 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-04-26T23:21:18.752 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-04-26T23:21:18.752 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-04-26T23:21:18.752 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-04-26T23:21:18.752 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-04-26T23:21:18.752 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-04-26T23:21:18.752 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-04-26T23:21:18.752 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-04-26T23:21:18.752 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-04-26T23:21:18.752 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-04-26T23:21:18.753 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-04-26T23:21:18.753 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-04-26T23:21:18.753 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty 2024-04-26T23:21:18.753 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless 2024-04-26T23:21:18.753 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-04-26T23:21:18.753 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_gsm.ko 2024-04-26T23:21:18.753 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_hdlc.ko 2024-04-26T23:21:18.753 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/nozomi.ko 2024-04-26T23:21:18.753 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial 2024-04-26T23:21:18.753 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250 2024-04-26T23:21:18.753 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-04-26T23:21:18.753 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/arc_uart.ko 2024-04-26T23:21:18.753 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm 2024-04-26T23:21:18.753 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm/jsm.ko 2024-04-26T23:21:18.753 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs 2024-04-26T23:21:18.754 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core 2024-04-26T23:21:18.754 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core/ufshcd-core.ko 2024-04-26T23:21:18.754 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host 2024-04-26T23:21:18.754 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-04-26T23:21:18.754 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio 2024-04-26T23:21:18.754 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio.ko 2024-04-26T23:21:18.754 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_aec.ko 2024-04-26T23:21:18.754 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_cif.ko 2024-04-26T23:21:18.754 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_pci_generic.ko 2024-04-26T23:21:18.754 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_sercos3.ko 2024-04-26T23:21:18.754 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb 2024-04-26T23:21:18.754 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class 2024-04-26T23:21:18.754 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-acm.ko 2024-04-26T23:21:18.754 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-wdm.ko 2024-04-26T23:21:18.755 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usblp.ko 2024-04-26T23:21:18.755 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usbtmc.ko 2024-04-26T23:21:18.755 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host 2024-04-26T23:21:18.755 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host/sl811-hcd.ko 2024-04-26T23:21:18.755 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image 2024-04-26T23:21:18.755 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/mdc800.ko 2024-04-26T23:21:18.755 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/microtek.ko 2024-04-26T23:21:18.755 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc 2024-04-26T23:21:18.755 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/adutux.ko 2024-04-26T23:21:18.755 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/appledisplay.ko 2024-04-26T23:21:18.755 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi26.ko 2024-04-26T23:21:18.755 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi62.ko 2024-04-26T23:21:18.755 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ezusb.ko 2024-04-26T23:21:18.755 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/idmouse.ko 2024-04-26T23:21:18.756 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/iowarrior.ko 2024-04-26T23:21:18.756 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/isight_firmware.ko 2024-04-26T23:21:18.756 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ldusb.ko 2024-04-26T23:21:18.756 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/legousbtower.ko 2024-04-26T23:21:18.756 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga 2024-04-26T23:21:18.756 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-04-26T23:21:18.756 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/trancevibrator.ko 2024-04-26T23:21:18.756 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usb3503.ko 2024-04-26T23:21:18.756 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usblcd.ko 2024-04-26T23:21:18.756 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usbsevseg.ko 2024-04-26T23:21:18.756 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/uss720.ko 2024-04-26T23:21:18.756 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/yurex.ko 2024-04-26T23:21:18.756 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy 2024-04-26T23:21:18.756 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy/phy-generic.ko 2024-04-26T23:21:18.756 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial 2024-04-26T23:21:18.756 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/aircable.ko 2024-04-26T23:21:18.757 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ark3116.ko 2024-04-26T23:21:18.757 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/belkin_sa.ko 2024-04-26T23:21:18.757 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ch341.ko 2024-04-26T23:21:18.757 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cp210x.ko 2024-04-26T23:21:18.757 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cyberjack.ko 2024-04-26T23:21:18.757 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cypress_m8.ko 2024-04-26T23:21:18.757 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/digi_acceleport.ko 2024-04-26T23:21:18.757 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/empeg.ko 2024-04-26T23:21:18.757 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ftdi_sio.ko 2024-04-26T23:21:18.757 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/garmin_gps.ko 2024-04-26T23:21:18.757 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_edgeport.ko 2024-04-26T23:21:18.757 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_ti.ko 2024-04-26T23:21:18.757 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipaq.ko 2024-04-26T23:21:18.757 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipw.ko 2024-04-26T23:21:18.757 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ir-usb.ko 2024-04-26T23:21:18.758 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-04-26T23:21:18.758 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan.ko 2024-04-26T23:21:18.758 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan_pda.ko 2024-04-26T23:21:18.758 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kl5kusb105.ko 2024-04-26T23:21:18.758 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kobil_sct.ko 2024-04-26T23:21:18.758 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mct_u232.ko 2024-04-26T23:21:18.758 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7720.ko 2024-04-26T23:21:18.758 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7840.ko 2024-04-26T23:21:18.758 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/navman.ko 2024-04-26T23:21:18.758 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/omninet.ko 2024-04-26T23:21:18.758 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/opticon.ko 2024-04-26T23:21:18.758 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/option.ko 2024-04-26T23:21:18.758 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/oti6858.ko 2024-04-26T23:21:18.758 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/pl2303.ko 2024-04-26T23:21:18.759 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcaux.ko 2024-04-26T23:21:18.759 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcserial.ko 2024-04-26T23:21:18.759 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/quatech2.ko 2024-04-26T23:21:18.759 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/safe_serial.ko 2024-04-26T23:21:18.759 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/sierra.ko 2024-04-26T23:21:18.759 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/spcp8x5.ko 2024-04-26T23:21:18.759 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ssu100.ko 2024-04-26T23:21:18.759 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/symbolserial.ko 2024-04-26T23:21:18.759 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-04-26T23:21:18.759 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-04-26T23:21:18.759 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_debug.ko 2024-04-26T23:21:18.759 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_wwan.ko 2024-04-26T23:21:18.759 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/visor.ko 2024-04-26T23:21:18.759 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/whiteheat.ko 2024-04-26T23:21:18.759 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/xsens_mt.ko 2024-04-26T23:21:18.760 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage 2024-04-26T23:21:18.760 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/uas.ko 2024-04-26T23:21:18.760 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-alauda.ko 2024-04-26T23:21:18.760 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-cypress.ko 2024-04-26T23:21:18.760 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-datafab.ko 2024-04-26T23:21:18.760 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-04-26T23:21:18.760 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-freecom.ko 2024-04-26T23:21:18.760 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-isd200.ko 2024-04-26T23:21:18.760 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-04-26T23:21:18.760 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-karma.ko 2024-04-26T23:21:18.760 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-onetouch.ko 2024-04-26T23:21:18.760 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-realtek.ko 2024-04-26T23:21:18.760 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr09.ko 2024-04-26T23:21:18.760 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr55.ko 2024-04-26T23:21:18.760 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-usbat.ko 2024-04-26T23:21:18.761 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/usb-storage.ko 2024-04-26T23:21:18.761 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip 2024-04-26T23:21:18.761 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-core.ko 2024-04-26T23:21:18.761 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-host.ko 2024-04-26T23:21:18.761 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-04-26T23:21:18.761 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio 2024-04-26T23:21:18.761 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci 2024-04-26T23:21:18.761 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-04-26T23:21:18.761 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci.ko 2024-04-26T23:21:18.761 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio.ko 2024-04-26T23:21:18.761 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-04-26T23:21:18.761 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost 2024-04-26T23:21:18.761 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost.ko 2024-04-26T23:21:18.761 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_iotlb.ko 2024-04-26T23:21:18.762 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_net.ko 2024-04-26T23:21:18.762 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_scsi.ko 2024-04-26T23:21:18.762 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video 2024-04-26T23:21:18.762 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight 2024-04-26T23:21:18.762 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/apple_bl.ko 2024-04-26T23:21:18.762 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/lcd.ko 2024-04-26T23:21:18.762 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/platform_lcd.ko 2024-04-26T23:21:18.762 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio 2024-04-26T23:21:18.762 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio.ko 2024-04-26T23:21:18.762 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_balloon.ko 2024-04-26T23:21:18.762 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_mmio.ko 2024-04-26T23:21:18.762 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci.ko 2024-04-26T23:21:18.762 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-04-26T23:21:18.762 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-04-26T23:21:18.763 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_ring.ko 2024-04-26T23:21:18.763 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1 2024-04-26T23:21:18.763 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters 2024-04-26T23:21:18.763 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2482.ko 2024-04-26T23:21:18.763 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2490.ko 2024-04-26T23:21:18.763 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves 2024-04-26T23:21:18.763 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-04-26T23:21:18.763 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-04-26T23:21:18.763 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-04-26T23:21:18.763 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-04-26T23:21:18.763 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-04-26T23:21:18.763 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-04-26T23:21:18.763 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-04-26T23:21:18.763 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-04-26T23:21:18.764 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-04-26T23:21:18.764 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_smem.ko 2024-04-26T23:21:18.764 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_therm.ko 2024-04-26T23:21:18.764 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/wire.ko 2024-04-26T23:21:18.764 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog 2024-04-26T23:21:18.764 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim1535_wdt.ko 2024-04-26T23:21:18.764 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim7101_wdt.ko 2024-04-26T23:21:18.764 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/f71808e_wdt.ko 2024-04-26T23:21:18.764 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/hpwdt.ko 2024-04-26T23:21:18.764 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/i6300esb.ko 2024-04-26T23:21:18.764 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-04-26T23:21:18.764 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_wdt.ko 2024-04-26T23:21:18.764 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ib700wdt.ko 2024-04-26T23:21:18.764 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ibmasr.ko 2024-04-26T23:21:18.764 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-04-26T23:21:18.764 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it8712f_wdt.ko 2024-04-26T23:21:18.765 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it87_wdt.ko 2024-04-26T23:21:18.765 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/machzwd.ko 2024-04-26T23:21:18.765 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/nv_tco.ko 2024-04-26T23:21:18.765 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_pci.ko 2024-04-26T23:21:18.765 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_usb.ko 2024-04-26T23:21:18.765 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-04-26T23:21:18.765 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sch311x_wdt.ko 2024-04-26T23:21:18.765 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/softdog.ko 2024-04-26T23:21:18.765 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sp5100_tco.ko 2024-04-26T23:21:18.765 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/via_wdt.ko 2024-04-26T23:21:18.765 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-04-26T23:21:18.765 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83877f_wdt.ko 2024-04-26T23:21:18.765 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83977f_wdt.ko 2024-04-26T23:21:18.765 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/wdt_pci.ko 2024-04-26T23:21:18.765 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/xen_wdt.ko 2024-04-26T23:21:18.766 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen 2024-04-26T23:21:18.766 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-acpi-processor.ko 2024-04-26T23:21:18.766 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-evtchn.ko 2024-04-26T23:21:18.766 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntalloc.ko 2024-04-26T23:21:18.766 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntdev.ko 2024-04-26T23:21:18.766 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback 2024-04-26T23:21:18.766 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-04-26T23:21:18.766 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-privcmd.ko 2024-04-26T23:21:18.766 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs 2024-04-26T23:21:18.766 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs/xenfs.ko 2024-04-26T23:21:18.766 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs 2024-04-26T23:21:18.766 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p 2024-04-26T23:21:18.766 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p/9p.ko 2024-04-26T23:21:18.766 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs 2024-04-26T23:21:18.767 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs/affs.ko 2024-04-26T23:21:18.767 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs 2024-04-26T23:21:18.767 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs/befs.ko 2024-04-26T23:21:18.767 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/binfmt_misc.ko 2024-04-26T23:21:18.767 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs 2024-04-26T23:21:18.767 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs/btrfs.ko 2024-04-26T23:21:18.767 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles 2024-04-26T23:21:18.767 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles/cachefiles.ko 2024-04-26T23:21:18.767 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph 2024-04-26T23:21:18.767 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph/ceph.ko 2024-04-26T23:21:18.767 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda 2024-04-26T23:21:18.767 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda/coda.ko 2024-04-26T23:21:18.767 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs 2024-04-26T23:21:18.767 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs/cramfs.ko 2024-04-26T23:21:18.768 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm 2024-04-26T23:21:18.768 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm/dlm.ko 2024-04-26T23:21:18.768 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs 2024-04-26T23:21:18.768 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs/ecryptfs.ko 2024-04-26T23:21:18.768 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs 2024-04-26T23:21:18.768 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs/f2fs.ko 2024-04-26T23:21:18.768 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat 2024-04-26T23:21:18.768 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/fat.ko 2024-04-26T23:21:18.768 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/msdos.ko 2024-04-26T23:21:18.768 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/vfat.ko 2024-04-26T23:21:18.768 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse 2024-04-26T23:21:18.768 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/cuse.ko 2024-04-26T23:21:18.768 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/fuse.ko 2024-04-26T23:21:18.768 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2 2024-04-26T23:21:18.768 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2/gfs2.ko 2024-04-26T23:21:18.768 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs 2024-04-26T23:21:18.769 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs/hfs.ko 2024-04-26T23:21:18.769 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus 2024-04-26T23:21:18.769 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus/hfsplus.ko 2024-04-26T23:21:18.769 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs 2024-04-26T23:21:18.769 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs/isofs.ko 2024-04-26T23:21:18.769 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs 2024-04-26T23:21:18.769 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs/jfs.ko 2024-04-26T23:21:18.769 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd 2024-04-26T23:21:18.769 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd/lockd.ko 2024-04-26T23:21:18.769 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix 2024-04-26T23:21:18.769 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix/minix.ko 2024-04-26T23:21:18.769 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs 2024-04-26T23:21:18.769 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs/netfs.ko 2024-04-26T23:21:18.769 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs 2024-04-26T23:21:18.770 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout 2024-04-26T23:21:18.770 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-04-26T23:21:18.770 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout 2024-04-26T23:21:18.770 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-04-26T23:21:18.770 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout 2024-04-26T23:21:18.770 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-04-26T23:21:18.770 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfs.ko 2024-04-26T23:21:18.770 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv3.ko 2024-04-26T23:21:18.770 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv4.ko 2024-04-26T23:21:18.770 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common 2024-04-26T23:21:18.770 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/grace.ko 2024-04-26T23:21:18.770 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/nfs_acl.ko 2024-04-26T23:21:18.770 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd 2024-04-26T23:21:18.770 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd/nfsd.ko 2024-04-26T23:21:18.771 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2 2024-04-26T23:21:18.771 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2/nilfs2.ko 2024-04-26T23:21:18.771 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls 2024-04-26T23:21:18.771 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-celtic.ko 2024-04-26T23:21:18.771 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-centeuro.ko 2024-04-26T23:21:18.771 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-croatian.ko 2024-04-26T23:21:18.771 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-cyrillic.ko 2024-04-26T23:21:18.771 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-gaelic.ko 2024-04-26T23:21:18.771 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-greek.ko 2024-04-26T23:21:18.771 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-iceland.ko 2024-04-26T23:21:18.771 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-inuit.ko 2024-04-26T23:21:18.771 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-roman.ko 2024-04-26T23:21:18.771 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-romanian.ko 2024-04-26T23:21:18.771 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-turkish.ko 2024-04-26T23:21:18.771 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1250.ko 2024-04-26T23:21:18.771 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1251.ko 2024-04-26T23:21:18.772 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1255.ko 2024-04-26T23:21:18.772 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp737.ko 2024-04-26T23:21:18.772 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp775.ko 2024-04-26T23:21:18.772 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp850.ko 2024-04-26T23:21:18.772 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp852.ko 2024-04-26T23:21:18.772 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp855.ko 2024-04-26T23:21:18.772 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp857.ko 2024-04-26T23:21:18.772 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp860.ko 2024-04-26T23:21:18.772 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp861.ko 2024-04-26T23:21:18.772 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp862.ko 2024-04-26T23:21:18.772 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp863.ko 2024-04-26T23:21:18.772 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp864.ko 2024-04-26T23:21:18.772 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp865.ko 2024-04-26T23:21:18.772 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp866.ko 2024-04-26T23:21:18.772 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp869.ko 2024-04-26T23:21:18.773 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp874.ko 2024-04-26T23:21:18.773 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp932.ko 2024-04-26T23:21:18.773 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp936.ko 2024-04-26T23:21:18.773 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp949.ko 2024-04-26T23:21:18.773 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp950.ko 2024-04-26T23:21:18.773 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_euc-jp.ko 2024-04-26T23:21:18.773 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-1.ko 2024-04-26T23:21:18.773 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-13.ko 2024-04-26T23:21:18.773 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-14.ko 2024-04-26T23:21:18.773 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-15.ko 2024-04-26T23:21:18.773 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-2.ko 2024-04-26T23:21:18.773 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-3.ko 2024-04-26T23:21:18.773 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-4.ko 2024-04-26T23:21:18.773 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-5.ko 2024-04-26T23:21:18.773 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-6.ko 2024-04-26T23:21:18.774 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-7.ko 2024-04-26T23:21:18.774 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-9.ko 2024-04-26T23:21:18.774 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-r.ko 2024-04-26T23:21:18.774 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-ru.ko 2024-04-26T23:21:18.774 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-u.ko 2024-04-26T23:21:18.774 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_ucs2_utils.ko 2024-04-26T23:21:18.774 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_utf8.ko 2024-04-26T23:21:18.774 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2 2024-04-26T23:21:18.774 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster 2024-04-26T23:21:18.774 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-04-26T23:21:18.774 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm 2024-04-26T23:21:18.774 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-04-26T23:21:18.774 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs 2024-04-26T23:21:18.774 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-04-26T23:21:18.775 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2.ko 2024-04-26T23:21:18.775 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-04-26T23:21:18.775 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-04-26T23:21:18.775 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-04-26T23:21:18.775 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs 2024-04-26T23:21:18.775 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs/overlay.ko 2024-04-26T23:21:18.775 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore 2024-04-26T23:21:18.775 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore/ramoops.ko 2024-04-26T23:21:18.775 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs 2024-04-26T23:21:18.775 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs/reiserfs.ko 2024-04-26T23:21:18.775 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs 2024-04-26T23:21:18.775 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs/romfs.ko 2024-04-26T23:21:18.775 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb 2024-04-26T23:21:18.775 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client 2024-04-26T23:21:18.775 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client/cifs.ko 2024-04-26T23:21:18.776 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common 2024-04-26T23:21:18.776 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_arc4.ko 2024-04-26T23:21:18.776 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_md4.ko 2024-04-26T23:21:18.776 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs 2024-04-26T23:21:18.776 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs/squashfs.ko 2024-04-26T23:21:18.776 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv 2024-04-26T23:21:18.776 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv/sysv.ko 2024-04-26T23:21:18.776 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf 2024-04-26T23:21:18.776 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf/udf.ko 2024-04-26T23:21:18.776 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs 2024-04-26T23:21:18.776 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs/ufs.ko 2024-04-26T23:21:18.776 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs 2024-04-26T23:21:18.776 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs/xfs.ko 2024-04-26T23:21:18.776 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel 2024-04-26T23:21:18.776 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu 2024-04-26T23:21:18.777 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu/rcutorture.ko 2024-04-26T23:21:18.777 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/torture.ko 2024-04-26T23:21:18.777 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace 2024-04-26T23:21:18.777 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-04-26T23:21:18.777 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib 2024-04-26T23:21:18.777 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/asn1_encoder.ko 2024-04-26T23:21:18.777 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc-itu-t.ko 2024-04-26T23:21:18.777 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc8.ko 2024-04-26T23:21:18.777 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto 2024-04-26T23:21:18.777 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libarc4.ko 2024-04-26T23:21:18.777 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libdes.ko 2024-04-26T23:21:18.777 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/libcrc32c.ko 2024-04-26T23:21:18.777 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lru_cache.ko 2024-04-26T23:21:18.777 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4 2024-04-26T23:21:18.777 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4_compress.ko 2024-04-26T23:21:18.778 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4hc_compress.ko 2024-04-26T23:21:18.778 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math 2024-04-26T23:21:18.778 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math/cordic.ko 2024-04-26T23:21:18.778 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6 2024-04-26T23:21:18.778 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6/raid6_pq.ko 2024-04-26T23:21:18.778 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon 2024-04-26T23:21:18.778 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon/reed_solomon.ko 2024-04-26T23:21:18.778 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_bm.ko 2024-04-26T23:21:18.778 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_fsm.ko 2024-04-26T23:21:18.778 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_kmp.ko 2024-04-26T23:21:18.778 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd 2024-04-26T23:21:18.778 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd/zstd_compress.ko 2024-04-26T23:21:18.778 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm 2024-04-26T23:21:18.778 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm/hwpoison-inject.ko 2024-04-26T23:21:18.779 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net 2024-04-26T23:21:18.779 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan 2024-04-26T23:21:18.779 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/6lowpan.ko 2024-04-26T23:21:18.779 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_dest.ko 2024-04-26T23:21:18.779 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_fragment.ko 2024-04-26T23:21:18.779 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_hop.ko 2024-04-26T23:21:18.779 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_ipv6.ko 2024-04-26T23:21:18.779 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_mobility.ko 2024-04-26T23:21:18.779 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_routing.ko 2024-04-26T23:21:18.779 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_udp.ko 2024-04-26T23:21:18.779 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802 2024-04-26T23:21:18.779 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/garp.ko 2024-04-26T23:21:18.779 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/mrp.ko 2024-04-26T23:21:18.779 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/p8022.ko 2024-04-26T23:21:18.779 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/psnap.ko 2024-04-26T23:21:18.780 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/stp.ko 2024-04-26T23:21:18.780 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q 2024-04-26T23:21:18.780 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q/8021q.ko 2024-04-26T23:21:18.780 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p 2024-04-26T23:21:18.780 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet.ko 2024-04-26T23:21:18.780 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_fd.ko 2024-04-26T23:21:18.780 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_rdma.ko 2024-04-26T23:21:18.780 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_virtio.ko 2024-04-26T23:21:18.780 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv 2024-04-26T23:21:18.780 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv/batman-adv.ko 2024-04-26T23:21:18.780 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge 2024-04-26T23:21:18.780 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/br_netfilter.ko 2024-04-26T23:21:18.780 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/bridge.ko 2024-04-26T23:21:18.780 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter 2024-04-26T23:21:18.780 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-04-26T23:21:18.781 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_among.ko 2024-04-26T23:21:18.781 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arp.ko 2024-04-26T23:21:18.781 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-04-26T23:21:18.781 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-04-26T23:21:18.781 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip.ko 2024-04-26T23:21:18.781 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-04-26T23:21:18.781 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_limit.ko 2024-04-26T23:21:18.781 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_log.ko 2024-04-26T23:21:18.781 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark.ko 2024-04-26T23:21:18.781 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-04-26T23:21:18.781 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-04-26T23:21:18.781 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-04-26T23:21:18.781 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-04-26T23:21:18.781 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_snat.ko 2024-04-26T23:21:18.781 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_stp.ko 2024-04-26T23:21:18.782 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-04-26T23:21:18.782 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-04-26T23:21:18.782 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-04-26T23:21:18.782 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-04-26T23:21:18.782 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtables.ko 2024-04-26T23:21:18.782 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-04-26T23:21:18.782 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-04-26T23:21:18.782 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-04-26T23:21:18.782 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph 2024-04-26T23:21:18.782 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph/libceph.ko 2024-04-26T23:21:18.782 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core 2024-04-26T23:21:18.782 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/failover.ko 2024-04-26T23:21:18.782 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/pktgen.ko 2024-04-26T23:21:18.782 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp 2024-04-26T23:21:18.783 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp.ko 2024-04-26T23:21:18.783 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_diag.ko 2024-04-26T23:21:18.783 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv4.ko 2024-04-26T23:21:18.783 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv6.ko 2024-04-26T23:21:18.783 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver 2024-04-26T23:21:18.783 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver/dns_resolver.ko 2024-04-26T23:21:18.783 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa 2024-04-26T23:21:18.783 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/dsa_core.ko 2024-04-26T23:21:18.783 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_brcm.ko 2024-04-26T23:21:18.783 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_dsa.ko 2024-04-26T23:21:18.783 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_none.ko 2024-04-26T23:21:18.783 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_trailer.ko 2024-04-26T23:21:18.783 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154 2024-04-26T23:21:18.783 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan 2024-04-26T23:21:18.783 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-04-26T23:21:18.784 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154.ko 2024-04-26T23:21:18.784 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154_socket.ko 2024-04-26T23:21:18.784 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4 2024-04-26T23:21:18.784 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ah4.ko 2024-04-26T23:21:18.784 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/esp4.ko 2024-04-26T23:21:18.784 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/fou.ko 2024-04-26T23:21:18.784 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/gre.ko 2024-04-26T23:21:18.784 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/inet_diag.ko 2024-04-26T23:21:18.784 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_gre.ko 2024-04-26T23:21:18.784 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_tunnel.ko 2024-04-26T23:21:18.784 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_vti.ko 2024-04-26T23:21:18.784 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipcomp.ko 2024-04-26T23:21:18.784 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipip.ko 2024-04-26T23:21:18.784 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter 2024-04-26T23:21:18.784 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arp_tables.ko 2024-04-26T23:21:18.785 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-04-26T23:21:18.785 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-04-26T23:21:18.785 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-04-26T23:21:18.785 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-04-26T23:21:18.785 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-04-26T23:21:18.785 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-04-26T23:21:18.785 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-04-26T23:21:18.785 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-04-26T23:21:18.785 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-04-26T23:21:18.785 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_security.ko 2024-04-26T23:21:18.785 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-04-26T23:21:18.785 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-04-26T23:21:18.785 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-04-26T23:21:18.785 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-04-26T23:21:18.785 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-04-26T23:21:18.786 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-04-26T23:21:18.786 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-04-26T23:21:18.786 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-04-26T23:21:18.786 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-04-26T23:21:18.786 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-04-26T23:21:18.786 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_bic.ko 2024-04-26T23:21:18.786 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_dctcp.ko 2024-04-26T23:21:18.786 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_diag.ko 2024-04-26T23:21:18.786 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_highspeed.ko 2024-04-26T23:21:18.786 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_htcp.ko 2024-04-26T23:21:18.786 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_hybla.ko 2024-04-26T23:21:18.786 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_illinois.ko 2024-04-26T23:21:18.786 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_lp.ko 2024-04-26T23:21:18.786 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_scalable.ko 2024-04-26T23:21:18.786 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_vegas.ko 2024-04-26T23:21:18.787 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_veno.ko 2024-04-26T23:21:18.787 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_westwood.ko 2024-04-26T23:21:18.787 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_yeah.ko 2024-04-26T23:21:18.787 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tunnel4.ko 2024-04-26T23:21:18.787 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_diag.ko 2024-04-26T23:21:18.787 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_tunnel.ko 2024-04-26T23:21:18.787 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/xfrm4_tunnel.ko 2024-04-26T23:21:18.787 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6 2024-04-26T23:21:18.787 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ah6.ko 2024-04-26T23:21:18.787 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/esp6.ko 2024-04-26T23:21:18.787 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/fou6.ko 2024-04-26T23:21:18.787 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_tunnel.ko 2024-04-26T23:21:18.787 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-04-26T23:21:18.787 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_vti.ko 2024-04-26T23:21:18.787 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ipcomp6.ko 2024-04-26T23:21:18.788 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter 2024-04-26T23:21:18.788 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-04-26T23:21:18.788 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-04-26T23:21:18.788 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-04-26T23:21:18.788 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-04-26T23:21:18.788 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-04-26T23:21:18.788 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-04-26T23:21:18.788 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-04-26T23:21:18.788 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-04-26T23:21:18.788 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-04-26T23:21:18.788 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-04-26T23:21:18.788 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-04-26T23:21:18.788 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-04-26T23:21:18.788 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-04-26T23:21:18.788 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-04-26T23:21:18.789 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-04-26T23:21:18.789 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-04-26T23:21:18.789 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-04-26T23:21:18.789 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-04-26T23:21:18.789 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-04-26T23:21:18.789 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-04-26T23:21:18.789 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-04-26T23:21:18.789 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-04-26T23:21:18.789 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-04-26T23:21:18.789 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-04-26T23:21:18.789 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/sit.ko 2024-04-26T23:21:18.789 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/tunnel6.ko 2024-04-26T23:21:18.789 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/xfrm6_tunnel.ko 2024-04-26T23:21:18.789 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key 2024-04-26T23:21:18.789 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key/af_key.ko 2024-04-26T23:21:18.790 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp 2024-04-26T23:21:18.790 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_core.ko 2024-04-26T23:21:18.790 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_debugfs.ko 2024-04-26T23:21:18.790 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_eth.ko 2024-04-26T23:21:18.790 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip.ko 2024-04-26T23:21:18.790 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip6.ko 2024-04-26T23:21:18.790 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_netlink.ko 2024-04-26T23:21:18.790 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ppp.ko 2024-04-26T23:21:18.790 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc 2024-04-26T23:21:18.790 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc/llc.ko 2024-04-26T23:21:18.790 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154 2024-04-26T23:21:18.790 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154/mac802154.ko 2024-04-26T23:21:18.790 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls 2024-04-26T23:21:18.790 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls/mpls_gso.ko 2024-04-26T23:21:18.791 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter 2024-04-26T23:21:18.791 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset 2024-04-26T23:21:18.791 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set.ko 2024-04-26T23:21:18.791 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-04-26T23:21:18.791 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-04-26T23:21:18.791 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-04-26T23:21:18.791 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-04-26T23:21:18.791 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-04-26T23:21:18.791 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-04-26T23:21:18.791 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-04-26T23:21:18.791 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-04-26T23:21:18.791 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-04-26T23:21:18.791 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-04-26T23:21:18.791 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-04-26T23:21:18.791 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-04-26T23:21:18.792 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-04-26T23:21:18.792 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-04-26T23:21:18.792 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-04-26T23:21:18.792 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs 2024-04-26T23:21:18.792 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs.ko 2024-04-26T23:21:18.792 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-04-26T23:21:18.792 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-04-26T23:21:18.792 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-04-26T23:21:18.792 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-04-26T23:21:18.792 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-04-26T23:21:18.792 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-04-26T23:21:18.792 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-04-26T23:21:18.792 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-04-26T23:21:18.792 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-04-26T23:21:18.793 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-04-26T23:21:18.793 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-04-26T23:21:18.793 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-04-26T23:21:18.793 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-04-26T23:21:18.793 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conncount.ko 2024-04-26T23:21:18.793 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack.ko 2024-04-26T23:21:18.793 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-04-26T23:21:18.793 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-04-26T23:21:18.793 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-04-26T23:21:18.793 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_h323.ko 2024-04-26T23:21:18.793 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_irc.ko 2024-04-26T23:21:18.793 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-04-26T23:21:18.793 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-04-26T23:21:18.793 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-04-26T23:21:18.793 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sane.ko 2024-04-26T23:21:18.794 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sip.ko 2024-04-26T23:21:18.794 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-04-26T23:21:18.794 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-04-26T23:21:18.794 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_dup_netdev.ko 2024-04-26T23:21:18.794 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table.ko 2024-04-26T23:21:18.794 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table_inet.ko 2024-04-26T23:21:18.794 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_log_syslog.ko 2024-04-26T23:21:18.794 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat.ko 2024-04-26T23:21:18.794 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_amanda.ko 2024-04-26T23:21:18.794 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_ftp.ko 2024-04-26T23:21:18.794 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_irc.ko 2024-04-26T23:21:18.794 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_sip.ko 2024-04-26T23:21:18.794 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_tftp.ko 2024-04-26T23:21:18.795 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_synproxy_core.ko 2024-04-26T23:21:18.795 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_tables.ko 2024-04-26T23:21:18.795 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink.ko 2024-04-26T23:21:18.795 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_acct.ko 2024-04-26T23:21:18.795 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_log.ko 2024-04-26T23:21:18.795 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_osf.ko 2024-04-26T23:21:18.795 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_queue.ko 2024-04-26T23:21:18.795 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_chain_nat.ko 2024-04-26T23:21:18.795 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_compat.ko 2024-04-26T23:21:18.795 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_ct.ko 2024-04-26T23:21:18.795 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_dup_netdev.ko 2024-04-26T23:21:18.795 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fib.ko 2024-04-26T23:21:18.795 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fwd_netdev.ko 2024-04-26T23:21:18.795 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_hash.ko 2024-04-26T23:21:18.795 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_limit.ko 2024-04-26T23:21:18.796 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_log.ko 2024-04-26T23:21:18.796 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_masq.ko 2024-04-26T23:21:18.796 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_nat.ko 2024-04-26T23:21:18.796 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_numgen.ko 2024-04-26T23:21:18.796 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_queue.ko 2024-04-26T23:21:18.796 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_quota.ko 2024-04-26T23:21:18.796 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_redir.ko 2024-04-26T23:21:18.796 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject.ko 2024-04-26T23:21:18.796 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject_inet.ko 2024-04-26T23:21:18.796 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_socket.ko 2024-04-26T23:21:18.796 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_tunnel.ko 2024-04-26T23:21:18.796 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_AUDIT.ko 2024-04-26T23:21:18.796 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CHECKSUM.ko 2024-04-26T23:21:18.796 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CLASSIFY.ko 2024-04-26T23:21:18.796 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-04-26T23:21:18.797 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CT.ko 2024-04-26T23:21:18.797 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_DSCP.ko 2024-04-26T23:21:18.797 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HL.ko 2024-04-26T23:21:18.797 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HMARK.ko 2024-04-26T23:21:18.797 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_IDLETIMER.ko 2024-04-26T23:21:18.797 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LED.ko 2024-04-26T23:21:18.797 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LOG.ko 2024-04-26T23:21:18.797 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_MASQUERADE.ko 2024-04-26T23:21:18.797 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NETMAP.ko 2024-04-26T23:21:18.797 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFLOG.ko 2024-04-26T23:21:18.797 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFQUEUE.ko 2024-04-26T23:21:18.797 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_RATEEST.ko 2024-04-26T23:21:18.797 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_REDIRECT.ko 2024-04-26T23:21:18.797 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_SECMARK.ko 2024-04-26T23:21:18.798 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPMSS.ko 2024-04-26T23:21:18.798 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-04-26T23:21:18.798 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TEE.ko 2024-04-26T23:21:18.798 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TPROXY.ko 2024-04-26T23:21:18.798 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TRACE.ko 2024-04-26T23:21:18.798 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_addrtype.ko 2024-04-26T23:21:18.798 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_bpf.ko 2024-04-26T23:21:18.798 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cgroup.ko 2024-04-26T23:21:18.798 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cluster.ko 2024-04-26T23:21:18.798 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_comment.ko 2024-04-26T23:21:18.798 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connbytes.ko 2024-04-26T23:21:18.798 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlabel.ko 2024-04-26T23:21:18.798 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlimit.ko 2024-04-26T23:21:18.798 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connmark.ko 2024-04-26T23:21:18.798 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_conntrack.ko 2024-04-26T23:21:18.799 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cpu.ko 2024-04-26T23:21:18.799 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dccp.ko 2024-04-26T23:21:18.799 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_devgroup.ko 2024-04-26T23:21:18.799 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dscp.ko 2024-04-26T23:21:18.799 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ecn.ko 2024-04-26T23:21:18.799 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_esp.ko 2024-04-26T23:21:18.799 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hashlimit.ko 2024-04-26T23:21:18.799 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_helper.ko 2024-04-26T23:21:18.799 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hl.ko 2024-04-26T23:21:18.799 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipcomp.ko 2024-04-26T23:21:18.799 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_iprange.ko 2024-04-26T23:21:18.799 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipvs.ko 2024-04-26T23:21:18.799 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_l2tp.ko 2024-04-26T23:21:18.799 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_length.ko 2024-04-26T23:21:18.799 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_limit.ko 2024-04-26T23:21:18.800 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mac.ko 2024-04-26T23:21:18.800 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mark.ko 2024-04-26T23:21:18.800 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_multiport.ko 2024-04-26T23:21:18.800 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nat.ko 2024-04-26T23:21:18.800 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nfacct.ko 2024-04-26T23:21:18.800 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_osf.ko 2024-04-26T23:21:18.800 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_owner.ko 2024-04-26T23:21:18.800 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_physdev.ko 2024-04-26T23:21:18.800 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_pkttype.ko 2024-04-26T23:21:18.800 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_policy.ko 2024-04-26T23:21:18.800 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_quota.ko 2024-04-26T23:21:18.800 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_rateest.ko 2024-04-26T23:21:18.800 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_realm.ko 2024-04-26T23:21:18.800 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_recent.ko 2024-04-26T23:21:18.800 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_sctp.ko 2024-04-26T23:21:18.801 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_set.ko 2024-04-26T23:21:18.801 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_socket.ko 2024-04-26T23:21:18.801 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_state.ko 2024-04-26T23:21:18.801 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_statistic.ko 2024-04-26T23:21:18.801 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_string.ko 2024-04-26T23:21:18.801 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_tcpmss.ko 2024-04-26T23:21:18.801 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_time.ko 2024-04-26T23:21:18.801 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_u32.ko 2024-04-26T23:21:18.801 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink 2024-04-26T23:21:18.801 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink/netlink_diag.ko 2024-04-26T23:21:18.801 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh 2024-04-26T23:21:18.801 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh/nsh.ko 2024-04-26T23:21:18.801 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch 2024-04-26T23:21:18.801 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/openvswitch.ko 2024-04-26T23:21:18.801 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-geneve.ko 2024-04-26T23:21:18.802 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-gre.ko 2024-04-26T23:21:18.802 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-vxlan.ko 2024-04-26T23:21:18.802 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet 2024-04-26T23:21:18.802 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet/af_packet_diag.ko 2024-04-26T23:21:18.802 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds 2024-04-26T23:21:18.802 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds.ko 2024-04-26T23:21:18.802 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_rdma.ko 2024-04-26T23:21:18.802 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_tcp.ko 2024-04-26T23:21:18.802 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched 2024-04-26T23:21:18.802 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_csum.ko 2024-04-26T23:21:18.802 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_gact.ko 2024-04-26T23:21:18.802 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_mirred.ko 2024-04-26T23:21:18.802 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_nat.ko 2024-04-26T23:21:18.802 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_pedit.ko 2024-04-26T23:21:18.803 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_police.ko 2024-04-26T23:21:18.803 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_simple.ko 2024-04-26T23:21:18.803 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_skbedit.ko 2024-04-26T23:21:18.803 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_vlan.ko 2024-04-26T23:21:18.803 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_basic.ko 2024-04-26T23:21:18.803 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_bpf.ko 2024-04-26T23:21:18.803 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_flow.ko 2024-04-26T23:21:18.803 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_fw.ko 2024-04-26T23:21:18.803 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_route.ko 2024-04-26T23:21:18.803 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_u32.ko 2024-04-26T23:21:18.803 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_cmp.ko 2024-04-26T23:21:18.803 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_ipset.ko 2024-04-26T23:21:18.803 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_meta.ko 2024-04-26T23:21:18.803 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_nbyte.ko 2024-04-26T23:21:18.803 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_text.ko 2024-04-26T23:21:18.804 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_u32.ko 2024-04-26T23:21:18.804 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_choke.ko 2024-04-26T23:21:18.804 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_codel.ko 2024-04-26T23:21:18.804 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_drr.ko 2024-04-26T23:21:18.804 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_fq.ko 2024-04-26T23:21:18.804 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_gred.ko 2024-04-26T23:21:18.804 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hfsc.ko 2024-04-26T23:21:18.804 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hhf.ko 2024-04-26T23:21:18.804 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_htb.ko 2024-04-26T23:21:18.804 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_ingress.ko 2024-04-26T23:21:18.804 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio.ko 2024-04-26T23:21:18.804 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio_lib.ko 2024-04-26T23:21:18.804 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_multiq.ko 2024-04-26T23:21:18.804 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_netem.ko 2024-04-26T23:21:18.804 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_pie.ko 2024-04-26T23:21:18.804 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_plug.ko 2024-04-26T23:21:18.805 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_prio.ko 2024-04-26T23:21:18.805 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_qfq.ko 2024-04-26T23:21:18.805 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_red.ko 2024-04-26T23:21:18.805 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfb.ko 2024-04-26T23:21:18.805 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfq.ko 2024-04-26T23:21:18.805 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_tbf.ko 2024-04-26T23:21:18.805 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_teql.ko 2024-04-26T23:21:18.805 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp 2024-04-26T23:21:18.805 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp.ko 2024-04-26T23:21:18.805 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp_diag.ko 2024-04-26T23:21:18.805 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc 2024-04-26T23:21:18.805 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss 2024-04-26T23:21:18.805 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-04-26T23:21:18.805 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-04-26T23:21:18.805 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/sunrpc.ko 2024-04-26T23:21:18.806 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma 2024-04-26T23:21:18.806 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-04-26T23:21:18.806 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc 2024-04-26T23:21:18.806 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc.ko 2024-04-26T23:21:18.806 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc_diag.ko 2024-04-26T23:21:18.806 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix 2024-04-26T23:21:18.806 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix/unix_diag.ko 2024-04-26T23:21:18.806 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock 2024-04-26T23:21:18.806 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-04-26T23:21:18.806 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-04-26T23:21:18.806 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock.ko 2024-04-26T23:21:18.806 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_diag.ko 2024-04-26T23:21:18.806 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_loopback.ko 2024-04-26T23:21:18.807 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm 2024-04-26T23:21:18.807 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm/xfrm_ipcomp.ko 2024-04-26T23:21:18.807 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security 2024-04-26T23:21:18.807 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys 2024-04-26T23:21:18.807 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys 2024-04-26T23:21:18.807 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-04-26T23:21:18.807 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys 2024-04-26T23:21:18.807 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys/trusted.ko 2024-04-26T23:21:18.807 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias 2024-04-26T23:21:18.807 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias.bin 2024-04-26T23:21:18.807 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin 2024-04-26T23:21:18.807 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.alias.bin 2024-04-26T23:21:18.807 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.bin 2024-04-26T23:21:18.807 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.modinfo 2024-04-26T23:21:18.807 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep 2024-04-26T23:21:18.808 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep.bin 2024-04-26T23:21:18.808 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.devname 2024-04-26T23:21:18.808 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.order 2024-04-26T23:21:18.808 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.softdep 2024-04-26T23:21:18.808 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols 2024-04-26T23:21:18.808 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols.bin 2024-04-26T23:21:18.808 INFO:teuthology.orchestra.run.smithi072.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-26T23:21:18.812 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-rc3-g806ec36b2112 2024-04-26T23:21:18.812 DEBUG:teuthology.orchestra.run.smithi072:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-04-26T23:21:18.837 INFO:teuthology.orchestra.run.smithi072.stdout:postinstall scriptlet (using /bin/sh): 2024-04-26T23:21:18.837 INFO:teuthology.orchestra.run.smithi072.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-26T23:21:18.837 INFO:teuthology.orchestra.run.smithi072.stdout: /usr/bin/kernel-install add 6.9.0-rc3-g806ec36b2112 /lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-26T23:21:18.837 INFO:teuthology.orchestra.run.smithi072.stdout:fi 2024-04-26T23:21:18.837 INFO:teuthology.orchestra.run.smithi072.stdout:for file in vmlinuz System.map config; do 2024-04-26T23:21:18.837 INFO:teuthology.orchestra.run.smithi072.stdout: if ! cmp --silent "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112"; then 2024-04-26T23:21:18.837 INFO:teuthology.orchestra.run.smithi072.stdout: cp "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112" 2024-04-26T23:21:18.838 INFO:teuthology.orchestra.run.smithi072.stdout: fi 2024-04-26T23:21:18.838 INFO:teuthology.orchestra.run.smithi072.stdout:done 2024-04-26T23:21:18.838 INFO:teuthology.orchestra.run.smithi072.stdout:if [ -d "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" ] && \ 2024-04-26T23:21:18.838 INFO:teuthology.orchestra.run.smithi072.stdout: ! diff -rq "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" >/dev/null 2>&1; then 2024-04-26T23:21:18.838 INFO:teuthology.orchestra.run.smithi072.stdout: rm -rf "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-26T23:21:18.838 INFO:teuthology.orchestra.run.smithi072.stdout: cp -r "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-26T23:21:18.838 INFO:teuthology.orchestra.run.smithi072.stdout:fi 2024-04-26T23:21:18.838 INFO:teuthology.orchestra.run.smithi072.stdout:if [ ! -e "/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep" ]; then 2024-04-26T23:21:18.838 INFO:teuthology.orchestra.run.smithi072.stdout: /usr/sbin/depmod 6.9.0-rc3-g806ec36b2112 2024-04-26T23:21:18.838 INFO:teuthology.orchestra.run.smithi072.stdout:fi 2024-04-26T23:21:18.838 INFO:teuthology.orchestra.run.smithi072.stdout:preuninstall scriptlet (using /bin/sh): 2024-04-26T23:21:18.838 INFO:teuthology.orchestra.run.smithi072.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-26T23:21:18.838 INFO:teuthology.orchestra.run.smithi072.stdout:kernel-install remove 6.9.0-rc3-g806ec36b2112 2024-04-26T23:21:18.838 INFO:teuthology.orchestra.run.smithi072.stdout:fi 2024-04-26T23:21:18.839 DEBUG:teuthology.orchestra.run.smithi072:> sudo rpm -qi grub2-tools 2024-04-26T23:21:18.909 INFO:teuthology.orchestra.run.smithi072.stdout:Name : grub2-tools 2024-04-26T23:21:18.909 INFO:teuthology.orchestra.run.smithi072.stdout:Epoch : 1 2024-04-26T23:21:18.909 INFO:teuthology.orchestra.run.smithi072.stdout:Version : 2.06 2024-04-26T23:21:18.909 INFO:teuthology.orchestra.run.smithi072.stdout:Release : 68.el9 2024-04-26T23:21:18.910 INFO:teuthology.orchestra.run.smithi072.stdout:Architecture: x86_64 2024-04-26T23:21:18.910 INFO:teuthology.orchestra.run.smithi072.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-04-26T23:21:18.910 INFO:teuthology.orchestra.run.smithi072.stdout:Group : Unspecified 2024-04-26T23:21:18.910 INFO:teuthology.orchestra.run.smithi072.stdout:Size : 8264254 2024-04-26T23:21:18.910 INFO:teuthology.orchestra.run.smithi072.stdout:License : GPLv3+ 2024-04-26T23:21:18.911 INFO:teuthology.orchestra.run.smithi072.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-26T23:21:18.911 INFO:teuthology.orchestra.run.smithi072.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-26T23:21:18.911 INFO:teuthology.orchestra.run.smithi072.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-26T23:21:18.911 INFO:teuthology.orchestra.run.smithi072.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-26T23:21:18.911 INFO:teuthology.orchestra.run.smithi072.stdout:Packager : builder@centos.org 2024-04-26T23:21:18.911 INFO:teuthology.orchestra.run.smithi072.stdout:Vendor : CentOS 2024-04-26T23:21:18.911 INFO:teuthology.orchestra.run.smithi072.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-26T23:21:18.911 INFO:teuthology.orchestra.run.smithi072.stdout:Summary : Support tools for GRUB. 2024-04-26T23:21:18.911 INFO:teuthology.orchestra.run.smithi072.stdout:Description : 2024-04-26T23:21:18.911 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:21:18.911 INFO:teuthology.orchestra.run.smithi072.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-26T23:21:18.911 INFO:teuthology.orchestra.run.smithi072.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-26T23:21:18.912 INFO:teuthology.orchestra.run.smithi072.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-26T23:21:18.912 INFO:teuthology.orchestra.run.smithi072.stdout:hardware devices. 2024-04-26T23:21:18.912 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:21:18.912 INFO:teuthology.orchestra.run.smithi072.stdout:This subpackage provides tools for support of all platforms. 2024-04-26T23:21:18.913 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-26T23:21:18.913 INFO:teuthology.task.kernel:Updating grub on smithi072 to boot 6.9.0-rc3-g806ec36b2112 2024-04-26T23:21:18.913 DEBUG:teuthology.orchestra.run.smithi072:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-26T23:21:19.524 INFO:teuthology.orchestra.run.smithi072.stderr:Generating grub configuration file ... 2024-04-26T23:21:21.152 INFO:teuthology.orchestra.run.smithi072.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-26T23:21:21.176 INFO:teuthology.orchestra.run.smithi072.stderr:done 2024-04-26T23:21:21.178 DEBUG:teuthology.orchestra.run.smithi072:> mktemp 2024-04-26T23:21:21.192 INFO:teuthology.orchestra.run.smithi072.stdout:/tmp/tmp.cQMBoNehfm 2024-04-26T23:21:21.192 DEBUG:teuthology.orchestra.run.smithi072:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.cQMBoNehfm 2024-04-26T23:21:21.274 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 0666 /tmp/tmp.cQMBoNehfm 2024-04-26T23:21:21.408 DEBUG:teuthology.orchestra.remote:smithi072:/tmp/tmp.cQMBoNehfm is 6KB 2024-04-26T23:21:21.458 DEBUG:teuthology.orchestra.run.smithi072:> rm -fr /tmp/tmp.cQMBoNehfm 2024-04-26T23:21:21.473 DEBUG:teuthology.orchestra.run.smithi072:> sudo /bin/ls /boot/loader/entries || true 2024-04-26T23:21:21.534 INFO:teuthology.orchestra.run.smithi072.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-04-26T23:21:21.534 INFO:teuthology.orchestra.run.smithi072.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-04-26T23:21:21.534 INFO:teuthology.orchestra.run.smithi072.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-04-26T23:21:21.534 INFO:teuthology.orchestra.run.smithi072.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112.conf 2024-04-26T23:21:21.535 DEBUG:teuthology.orchestra.run.smithi072:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112 2024-04-26T23:21:21.685 DEBUG:teuthology.orchestra.run.smithi072:> sudo shutdown -r now 2024-04-26T23:21:51.688 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-04-26T23:21:51.689 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-04-26T23:21:51.689 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T23:22:10.078 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.72 2024-04-26T23:22:19.079 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-04-26T23:22:19.080 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T23:22:22.146 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.72 2024-04-26T23:22:34.147 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-04-26T23:22:34.147 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T23:23:34.150 DEBUG:teuthology.orchestra.remote:timed out 2024-04-26T23:23:49.151 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-04-26T23:23:49.152 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T23:23:49.493 DEBUG:teuthology.orchestra.run.smithi072:> true 2024-04-26T23:23:50.213 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-04-26T23:23:50.213 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.9.0-rc3-g806ec36b2112"... 2024-04-26T23:23:50.213 DEBUG:teuthology.orchestra.run.smithi072:> uname -r 2024-04-26T23:23:50.265 INFO:teuthology.orchestra.run.smithi072.stdout:6.9.0-rc3-g806ec36b2112 2024-04-26T23:23:50.265 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-rc3-g806ec36b2112 vs 6.9.0-rc3-g806ec36b2112 2024-04-26T23:23:50.266 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-26T23:23:50.266 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-26T23:23:51.267 DEBUG:teuthology.parallel:result is None 2024-04-26T23:23:51.267 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-26T23:23:51.275 INFO:teuthology.task.internal:Creating test directory... 2024-04-26T23:23:51.275 DEBUG:teuthology.orchestra.run.smithi072:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T23:23:51.283 DEBUG:teuthology.orchestra.run.smithi076:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T23:23:51.285 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T23:23:51.332 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-26T23:23:51.341 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-26T23:23:51.350 INFO:teuthology.task.internal:Creating archive directory... 2024-04-26T23:23:51.351 DEBUG:teuthology.orchestra.run.smithi072:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T23:23:51.356 DEBUG:teuthology.orchestra.run.smithi076:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T23:23:51.359 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T23:23:51.584 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-26T23:23:51.591 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-26T23:23:51.591 DEBUG:teuthology.orchestra.run.smithi076:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T23:23:51.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T23:23:51.607 DEBUG:teuthology.orchestra.run.smithi136:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T23:23:51.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T23:23:51.624 DEBUG:teuthology.orchestra.run.smithi072:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T23:23:51.630 DEBUG:teuthology.orchestra.run.smithi076:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T23:23:51.650 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T23:23:51.686 INFO:teuthology.orchestra.run.smithi076.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T23:23:51.702 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T23:23:51.703 INFO:teuthology.orchestra.run.smithi076.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T23:23:51.713 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T23:23:51.784 INFO:teuthology.orchestra.run.smithi072.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T23:23:51.842 INFO:teuthology.orchestra.run.smithi072.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T23:23:51.847 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-26T23:23:51.854 INFO:teuthology.task.internal:Configuring sudo... 2024-04-26T23:23:51.854 DEBUG:teuthology.orchestra.run.smithi072:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T23:23:51.860 DEBUG:teuthology.orchestra.run.smithi076:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T23:23:51.863 DEBUG:teuthology.orchestra.run.smithi136:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T23:23:51.943 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-26T23:23:51.952 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-26T23:23:51.952 DEBUG:teuthology.orchestra.run.smithi072:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T23:23:51.957 DEBUG:teuthology.orchestra.run.smithi076:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T23:23:51.959 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T23:23:52.004 DEBUG:teuthology.orchestra.run.smithi072:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T23:23:52.086 DEBUG:teuthology.orchestra.run.smithi072:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T23:23:52.216 DEBUG:teuthology.orchestra.run.smithi072:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T23:23:52.299 DEBUG:teuthology.orchestra.run.smithi072:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T23:23:52.387 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-26T23:23:52.387 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T23:23:52.468 DEBUG:teuthology.orchestra.run.smithi076:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T23:23:52.512 DEBUG:teuthology.orchestra.run.smithi076:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T23:23:52.566 DEBUG:teuthology.orchestra.run.smithi076:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T23:23:52.646 DEBUG:teuthology.orchestra.run.smithi076:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T23:23:52.692 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-04-26T23:23:52.692 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T23:23:52.757 DEBUG:teuthology.orchestra.run.smithi136:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T23:23:52.802 DEBUG:teuthology.orchestra.run.smithi136:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T23:23:52.864 DEBUG:teuthology.orchestra.run.smithi136:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T23:23:52.906 DEBUG:teuthology.orchestra.run.smithi136:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T23:23:52.949 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-26T23:23:52.950 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T23:23:53.015 DEBUG:teuthology.orchestra.run.smithi072:> sudo service rsyslog restart 2024-04-26T23:23:53.022 DEBUG:teuthology.orchestra.run.smithi076:> sudo service rsyslog restart 2024-04-26T23:23:53.025 DEBUG:teuthology.orchestra.run.smithi136:> sudo service rsyslog restart 2024-04-26T23:23:53.055 INFO:teuthology.orchestra.run.smithi076.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T23:23:53.089 INFO:teuthology.orchestra.run.smithi136.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T23:23:53.130 INFO:teuthology.orchestra.run.smithi072.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T23:23:53.631 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-26T23:23:53.643 INFO:teuthology.task.internal:Starting timer... 2024-04-26T23:23:53.644 INFO:teuthology.run_tasks:Running task pcp... 2024-04-26T23:23:53.654 INFO:teuthology.run_tasks:Running task selinux... 2024-04-26T23:23:53.664 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-26T23:23:53.664 DEBUG:teuthology.orchestra.run.smithi072:> sudo service auditd rotate 2024-04-26T23:23:53.760 INFO:teuthology.orchestra.run.smithi072.stdout:Rotating logs: 2024-04-26T23:23:53.764 DEBUG:teuthology.orchestra.run.smithi076:> sudo service auditd rotate 2024-04-26T23:23:53.829 INFO:teuthology.orchestra.run.smithi076.stdout:Rotating logs: 2024-04-26T23:23:53.832 DEBUG:teuthology.orchestra.run.smithi136:> sudo service auditd rotate 2024-04-26T23:23:53.911 INFO:teuthology.orchestra.run.smithi136.stdout:Rotating logs: 2024-04-26T23:23:53.912 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-26T23:23:53.913 DEBUG:teuthology.orchestra.run.smithi072:> /usr/sbin/getenforce 2024-04-26T23:23:53.981 INFO:teuthology.orchestra.run.smithi072.stdout:Permissive 2024-04-26T23:23:53.981 DEBUG:teuthology.orchestra.run.smithi076:> /usr/sbin/getenforce 2024-04-26T23:23:54.015 INFO:teuthology.orchestra.run.smithi076.stdout:Permissive 2024-04-26T23:23:54.015 DEBUG:teuthology.orchestra.run.smithi136:> /usr/sbin/getenforce 2024-04-26T23:23:54.052 INFO:teuthology.orchestra.run.smithi136.stdout:Permissive 2024-04-26T23:23:54.053 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi072.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi076.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi136.front.sepia.ceph.com': 'permissive'} 2024-04-26T23:23:54.053 DEBUG:teuthology.orchestra.run.smithi072:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T23:23:54.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T23:23:54.123 DEBUG:teuthology.orchestra.run.smithi076:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T23:23:54.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T23:23:54.152 DEBUG:teuthology.orchestra.run.smithi136:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T23:23:54.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T23:23:54.179 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-26T23:23:54.180 DEBUG:teuthology.orchestra.run.smithi072:> sudo /usr/sbin/setenforce permissive 2024-04-26T23:23:54.262 DEBUG:teuthology.orchestra.run.smithi076:> sudo /usr/sbin/setenforce permissive 2024-04-26T23:23:54.288 DEBUG:teuthology.orchestra.run.smithi136:> sudo /usr/sbin/setenforce permissive 2024-04-26T23:23:54.322 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-26T23:23:54.342 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-26T23:23:54.359 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-26T23:23:54.361 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi072.front.sepia.ceph.com,smithi076.front.sepia.ceph.com,smithi136.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-26T23:29:02.391 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi072.front.sepia.ceph.com'), Remote(name='ubuntu@smithi076.front.sepia.ceph.com'), Remote(name='ubuntu@smithi136.front.sepia.ceph.com')] 2024-04-26T23:29:02.392 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-04-26T23:29:02.393 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T23:29:02.505 DEBUG:teuthology.orchestra.run.smithi072:> true 2024-04-26T23:29:02.664 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-04-26T23:29:02.665 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-04-26T23:29:02.665 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T23:29:02.740 DEBUG:teuthology.orchestra.run.smithi076:> true 2024-04-26T23:29:02.817 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-04-26T23:29:02.818 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-04-26T23:29:02.819 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T23:29:02.900 DEBUG:teuthology.orchestra.run.smithi136:> true 2024-04-26T23:29:02.975 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-04-26T23:29:02.976 INFO:teuthology.run_tasks:Running task clock... 2024-04-26T23:29:02.991 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-26T23:29:02.992 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T23:29:02.992 DEBUG:teuthology.orchestra.run.smithi072:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T23:29:03.000 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T23:29:03.000 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T23:29:03.003 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T23:29:03.004 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T23:29:03.030 INFO:teuthology.orchestra.run.smithi076.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-26T23:29:03.045 INFO:teuthology.orchestra.run.smithi076.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-26T23:29:03.046 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-26T23:29:03.061 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-26T23:29:03.068 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: ntpd: command not found 2024-04-26T23:29:03.079 INFO:teuthology.orchestra.run.smithi076.stdout:506 Cannot talk to daemon 2024-04-26T23:29:03.084 INFO:teuthology.orchestra.run.smithi072.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-26T23:29:03.085 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: ntpd: command not found 2024-04-26T23:29:03.092 INFO:teuthology.orchestra.run.smithi076.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-26T23:29:03.095 INFO:teuthology.orchestra.run.smithi136.stdout:506 Cannot talk to daemon 2024-04-26T23:29:03.105 INFO:teuthology.orchestra.run.smithi076.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-26T23:29:03.109 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-26T23:29:03.123 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-26T23:29:03.125 INFO:teuthology.orchestra.run.smithi072.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-26T23:29:03.163 INFO:teuthology.orchestra.run.smithi076.stderr:bash: line 1: ntpq: command not found 2024-04-26T23:29:03.166 INFO:teuthology.orchestra.run.smithi076.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T23:29:03.166 INFO:teuthology.orchestra.run.smithi076.stdout:=============================================================================== 2024-04-26T23:29:03.166 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T23:29:03.167 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T23:29:03.167 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T23:29:03.167 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T23:29:03.177 INFO:teuthology.orchestra.run.smithi136.stderr:bash: line 1: ntpq: command not found 2024-04-26T23:29:03.180 INFO:teuthology.orchestra.run.smithi136.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T23:29:03.180 INFO:teuthology.orchestra.run.smithi136.stdout:=============================================================================== 2024-04-26T23:29:03.180 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T23:29:03.181 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T23:29:03.181 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T23:29:03.181 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T23:29:03.197 INFO:teuthology.orchestra.run.smithi072.stderr:sudo: ntpd: command not found 2024-04-26T23:29:03.228 INFO:teuthology.orchestra.run.smithi072.stdout:506 Cannot talk to daemon 2024-04-26T23:29:03.269 INFO:teuthology.orchestra.run.smithi072.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-26T23:29:03.308 INFO:teuthology.orchestra.run.smithi072.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-26T23:29:03.442 INFO:teuthology.orchestra.run.smithi072.stderr:bash: line 1: ntpq: command not found 2024-04-26T23:29:03.450 INFO:teuthology.orchestra.run.smithi072.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T23:29:03.450 INFO:teuthology.orchestra.run.smithi072.stdout:=============================================================================== 2024-04-26T23:29:03.450 INFO:teuthology.orchestra.run.smithi072.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T23:29:03.450 INFO:teuthology.orchestra.run.smithi072.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T23:29:03.451 INFO:teuthology.orchestra.run.smithi072.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T23:29:03.451 INFO:teuthology.orchestra.run.smithi072.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T23:29:03.452 INFO:teuthology.run_tasks:Running task install... 2024-04-26T23:29:03.461 DEBUG:teuthology.task.install:project ceph 2024-04-26T23:29:03.461 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387'}} 2024-04-26T23:29:03.461 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387'} 2024-04-26T23:29:03.462 INFO:teuthology.task.install:Using flavor: default 2024-04-26T23:29:03.469 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-26T23:29:03.469 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-26T23:29:03.469 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'tag': None, 'wait_for_package': False} 2024-04-26T23:29:03.470 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T23:29:03.471 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'tag': None, 'wait_for_package': False} 2024-04-26T23:29:03.471 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T23:29:03.472 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'tag': None, 'wait_for_package': False} 2024-04-26T23:29:03.473 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T23:29:03.687 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/ 2024-04-26T23:29:03.687 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3306.gbc612ffb 2024-04-26T23:29:03.698 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/ 2024-04-26T23:29:03.698 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3306.gbc612ffb 2024-04-26T23:29:03.705 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/ 2024-04-26T23:29:03.705 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3306.gbc612ffb 2024-04-26T23:29:03.810 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-26T23:29:03.810 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-26T23:29:03.811 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-26T23:29:03.836 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-26T23:29:03.836 DEBUG:teuthology.orchestra.run.smithi136:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bc612ffb9c1d5bc27536bc917a705262a36d4387/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-26T23:29:03.862 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-26T23:29:03.862 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-26T23:29:03.862 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-26T23:29:03.875 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-26T23:29:03.876 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-04-26T23:29:03.876 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-26T23:29:03.900 DEBUG:teuthology.orchestra.run.smithi136:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-26T23:29:03.902 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-26T23:29:03.902 DEBUG:teuthology.orchestra.run.smithi076:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bc612ffb9c1d5bc27536bc917a705262a36d4387/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-26T23:29:03.952 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-26T23:29:03.953 DEBUG:teuthology.orchestra.run.smithi072:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bc612ffb9c1d5bc27536bc917a705262a36d4387/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-26T23:29:03.971 DEBUG:teuthology.orchestra.run.smithi076:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-26T23:29:03.974 DEBUG:teuthology.orchestra.run.smithi136:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-26T23:29:04.037 INFO:teuthology.orchestra.run.smithi136.stdout:check_obsoletes = 1 2024-04-26T23:29:04.038 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum clean all 2024-04-26T23:29:04.046 DEBUG:teuthology.orchestra.run.smithi076:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-26T23:29:04.081 DEBUG:teuthology.orchestra.run.smithi072:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-26T23:29:04.110 INFO:teuthology.orchestra.run.smithi076.stdout:check_obsoletes = 1 2024-04-26T23:29:04.112 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum clean all 2024-04-26T23:29:04.188 DEBUG:teuthology.orchestra.run.smithi072:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-26T23:29:04.262 INFO:teuthology.orchestra.run.smithi136.stdout:48 files removed 2024-04-26T23:29:04.262 INFO:teuthology.orchestra.run.smithi072.stdout:check_obsoletes = 1 2024-04-26T23:29:04.266 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum clean all 2024-04-26T23:29:04.289 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-26T23:29:04.339 INFO:teuthology.orchestra.run.smithi076.stdout:48 files removed 2024-04-26T23:29:04.366 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-26T23:29:04.578 INFO:teuthology.orchestra.run.smithi072.stdout:48 files removed 2024-04-26T23:29:04.617 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-26T23:29:04.872 INFO:teuthology.orchestra.run.smithi136.stdout:ceph packages for x86_64 259 kB/s | 84 kB 00:00 2024-04-26T23:29:05.008 INFO:teuthology.orchestra.run.smithi076.stdout:ceph packages for x86_64 218 kB/s | 84 kB 00:00 2024-04-26T23:29:05.289 INFO:teuthology.orchestra.run.smithi136.stdout:ceph noarch packages 31 kB/s | 12 kB 00:00 2024-04-26T23:29:05.384 INFO:teuthology.orchestra.run.smithi076.stdout:ceph noarch packages 35 kB/s | 12 kB 00:00 2024-04-26T23:29:05.541 INFO:teuthology.orchestra.run.smithi072.stdout:ceph packages for x86_64 175 kB/s | 84 kB 00:00 2024-04-26T23:29:05.648 INFO:teuthology.orchestra.run.smithi136.stdout:ceph source packages 5.8 kB/s | 1.9 kB 00:00 2024-04-26T23:29:05.738 INFO:teuthology.orchestra.run.smithi076.stdout:ceph source packages 6.0 kB/s | 1.9 kB 00:00 2024-04-26T23:29:05.994 INFO:teuthology.orchestra.run.smithi072.stdout:ceph noarch packages 33 kB/s | 12 kB 00:00 2024-04-26T23:29:06.323 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-04-26T23:29:06.435 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-04-26T23:29:06.604 INFO:teuthology.orchestra.run.smithi072.stdout:ceph source packages 3.6 kB/s | 1.9 kB 00:00 2024-04-26T23:29:07.570 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - BaseOS 9.1 MB/s | 8.0 MB 00:00 2024-04-26T23:29:08.383 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:01 2024-04-26T23:29:08.812 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - AppStream 14 MB/s | 19 MB 00:01 2024-04-26T23:29:09.847 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - AppStream 18 MB/s | 19 MB 00:01 2024-04-26T23:29:19.662 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - CRB 868 kB/s | 6.2 MB 00:07 2024-04-26T23:29:20.833 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - CRB 888 kB/s | 6.2 MB 00:07 2024-04-26T23:29:21.205 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 16 kB 00:00 2024-04-26T23:29:22.502 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - Extras packages 53 kB/s | 16 kB 00:00 2024-04-26T23:29:23.132 INFO:teuthology.orchestra.run.smithi076.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-04-26T23:29:25.227 INFO:teuthology.orchestra.run.smithi072.stdout:Extra Packages for Enterprise Linux 8.2 MB/s | 21 MB 00:02 2024-04-26T23:29:28.207 INFO:teuthology.orchestra.run.smithi076.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-04-26T23:29:30.358 INFO:teuthology.orchestra.run.smithi076.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T23:29:30.359 INFO:teuthology.orchestra.run.smithi076.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T23:29:30.369 INFO:teuthology.orchestra.run.smithi076.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-26T23:29:30.370 INFO:teuthology.orchestra.run.smithi076.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-26T23:29:30.371 INFO:teuthology.orchestra.run.smithi076.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-26T23:29:30.372 INFO:teuthology.orchestra.run.smithi076.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-26T23:29:30.373 INFO:teuthology.orchestra.run.smithi076.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-26T23:29:30.375 INFO:teuthology.orchestra.run.smithi076.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-26T23:29:30.376 INFO:teuthology.orchestra.run.smithi076.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-26T23:29:30.439 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-04-26T23:29:30.458 INFO:teuthology.orchestra.run.smithi076.stdout:======================================================================================= 2024-04-26T23:29:30.458 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2024-04-26T23:29:30.458 INFO:teuthology.orchestra.run.smithi076.stdout:======================================================================================= 2024-04-26T23:29:30.458 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-04-26T23:29:30.458 INFO:teuthology.orchestra.run.smithi076.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-26T23:29:30.458 INFO:teuthology.orchestra.run.smithi076.stdout: ceph x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 6.6 k 2024-04-26T23:29:30.458 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-base x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-26T23:29:30.458 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 889 k 2024-04-26T23:29:30.458 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 145 k 2024-04-26T23:29:30.458 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 1.5 M 2024-04-26T23:29:30.459 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 142 k 2024-04-26T23:29:30.459 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 3.5 M 2024-04-26T23:29:30.459 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 7.4 M 2024-04-26T23:29:30.459 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-rook noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 49 k 2024-04-26T23:29:30.459 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-radosgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 11 M 2024-04-26T23:29:30.459 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-test x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 49 M 2024-04-26T23:29:30.459 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-volume noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 273 k 2024-04-26T23:29:30.459 INFO:teuthology.orchestra.run.smithi076.stdout: cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 762 k 2024-04-26T23:29:30.459 INFO:teuthology.orchestra.run.smithi076.stdout: cephfs-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 225 k 2024-04-26T23:29:30.459 INFO:teuthology.orchestra.run.smithi076.stdout: cephfs-shell x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 27 k 2024-04-26T23:29:30.459 INFO:teuthology.orchestra.run.smithi076.stdout: cephfs-top noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 18 k 2024-04-26T23:29:30.459 INFO:teuthology.orchestra.run.smithi076.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-26T23:29:30.459 INFO:teuthology.orchestra.run.smithi076.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-26T23:29:30.459 INFO:teuthology.orchestra.run.smithi076.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-26T23:29:30.459 INFO:teuthology.orchestra.run.smithi076.stdout: libcephfs-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 32 k 2024-04-26T23:29:30.460 INFO:teuthology.orchestra.run.smithi076.stdout: libcephfs2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 747 k 2024-04-26T23:29:30.460 INFO:teuthology.orchestra.run.smithi076.stdout: librados-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 127 k 2024-04-26T23:29:30.460 INFO:teuthology.orchestra.run.smithi076.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-26T23:29:30.460 INFO:teuthology.orchestra.run.smithi076.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-26T23:29:30.460 INFO:teuthology.orchestra.run.smithi076.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-26T23:29:30.460 INFO:teuthology.orchestra.run.smithi076.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-26T23:29:30.460 INFO:teuthology.orchestra.run.smithi076.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-26T23:29:30.460 INFO:teuthology.orchestra.run.smithi076.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-26T23:29:30.460 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cephfs x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 161 k 2024-04-26T23:29:30.460 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rados x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 321 k 2024-04-26T23:29:30.460 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 298 k 2024-04-26T23:29:30.460 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 99 k 2024-04-26T23:29:30.460 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 85 k 2024-04-26T23:29:30.461 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-26T23:29:30.461 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-nbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 171 k 2024-04-26T23:29:30.461 INFO:teuthology.orchestra.run.smithi076.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-26T23:29:30.461 INFO:teuthology.orchestra.run.smithi076.stdout:Upgrading: 2024-04-26T23:29:30.461 INFO:teuthology.orchestra.run.smithi076.stdout: librados2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.4 M 2024-04-26T23:29:30.461 INFO:teuthology.orchestra.run.smithi076.stdout: librbd1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-26T23:29:30.461 INFO:teuthology.orchestra.run.smithi076.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-26T23:29:30.461 INFO:teuthology.orchestra.run.smithi076.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-26T23:29:30.461 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2024-04-26T23:29:30.461 INFO:teuthology.orchestra.run.smithi076.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-26T23:29:30.461 INFO:teuthology.orchestra.run.smithi076.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-26T23:29:30.461 INFO:teuthology.orchestra.run.smithi076.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-26T23:29:30.461 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 21 M 2024-04-26T23:29:30.461 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 34 k 2024-04-26T23:29:30.462 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mds x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 2.4 M 2024-04-26T23:29:30.462 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 249 k 2024-04-26T23:29:30.462 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mon x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 4.7 M 2024-04-26T23:29:30.462 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-osd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 17 M 2024-04-26T23:29:30.462 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 16 k 2024-04-26T23:29:30.462 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-selinux x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 25 k 2024-04-26T23:29:30.462 INFO:teuthology.orchestra.run.smithi076.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-26T23:29:30.462 INFO:teuthology.orchestra.run.smithi076.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-26T23:29:30.463 INFO:teuthology.orchestra.run.smithi076.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-26T23:29:30.463 INFO:teuthology.orchestra.run.smithi076.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-26T23:29:30.463 INFO:teuthology.orchestra.run.smithi076.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-26T23:29:30.463 INFO:teuthology.orchestra.run.smithi076.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-26T23:29:30.463 INFO:teuthology.orchestra.run.smithi076.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-26T23:29:30.463 INFO:teuthology.orchestra.run.smithi076.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-26T23:29:30.463 INFO:teuthology.orchestra.run.smithi076.stdout: libcephsqlite x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 163 k 2024-04-26T23:29:30.463 INFO:teuthology.orchestra.run.smithi076.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-26T23:29:30.463 INFO:teuthology.orchestra.run.smithi076.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-26T23:29:30.463 INFO:teuthology.orchestra.run.smithi076.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-26T23:29:30.463 INFO:teuthology.orchestra.run.smithi076.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-26T23:29:30.463 INFO:teuthology.orchestra.run.smithi076.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-26T23:29:30.464 INFO:teuthology.orchestra.run.smithi076.stdout: libradosstriper1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 501 k 2024-04-26T23:29:30.464 INFO:teuthology.orchestra.run.smithi076.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-26T23:29:30.464 INFO:teuthology.orchestra.run.smithi076.stdout: librgw2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-26T23:29:30.464 INFO:teuthology.orchestra.run.smithi076.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-26T23:29:30.464 INFO:teuthology.orchestra.run.smithi076.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-26T23:29:30.464 INFO:teuthology.orchestra.run.smithi076.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-26T23:29:30.464 INFO:teuthology.orchestra.run.smithi076.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-26T23:29:30.464 INFO:teuthology.orchestra.run.smithi076.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-26T23:29:30.464 INFO:teuthology.orchestra.run.smithi076.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-26T23:29:30.464 INFO:teuthology.orchestra.run.smithi076.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-26T23:29:30.464 INFO:teuthology.orchestra.run.smithi076.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-26T23:29:30.464 INFO:teuthology.orchestra.run.smithi076.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-26T23:29:30.464 INFO:teuthology.orchestra.run.smithi076.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-26T23:29:30.464 INFO:teuthology.orchestra.run.smithi076.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-26T23:29:30.465 INFO:teuthology.orchestra.run.smithi076.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-26T23:29:30.465 INFO:teuthology.orchestra.run.smithi076.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-26T23:29:30.465 INFO:teuthology.orchestra.run.smithi076.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-26T23:29:30.465 INFO:teuthology.orchestra.run.smithi076.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-26T23:29:30.465 INFO:teuthology.orchestra.run.smithi076.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-26T23:29:30.465 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-26T23:29:30.465 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ceph-argparse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 45 k 2024-04-26T23:29:30.465 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 135 k 2024-04-26T23:29:30.465 INFO:teuthology.orchestra.run.smithi076.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-26T23:29:30.465 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-26T23:29:30.465 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-26T23:29:30.465 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-26T23:29:30.465 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-26T23:29:30.465 INFO:teuthology.orchestra.run.smithi076.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-26T23:29:30.466 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-26T23:29:30.466 INFO:teuthology.orchestra.run.smithi076.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-26T23:29:30.466 INFO:teuthology.orchestra.run.smithi076.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-26T23:29:30.466 INFO:teuthology.orchestra.run.smithi076.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-26T23:29:30.466 INFO:teuthology.orchestra.run.smithi076.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-26T23:29:30.466 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-26T23:29:30.466 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-26T23:29:30.466 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-26T23:29:30.466 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-26T23:29:30.466 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-26T23:29:30.466 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-26T23:29:30.466 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-26T23:29:30.467 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-26T23:29:30.467 INFO:teuthology.orchestra.run.smithi076.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-26T23:29:30.467 INFO:teuthology.orchestra.run.smithi076.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-26T23:29:30.467 INFO:teuthology.orchestra.run.smithi076.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-26T23:29:30.467 INFO:teuthology.orchestra.run.smithi076.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-26T23:29:30.467 INFO:teuthology.orchestra.run.smithi076.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-26T23:29:30.467 INFO:teuthology.orchestra.run.smithi076.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-26T23:29:30.467 INFO:teuthology.orchestra.run.smithi076.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-26T23:29:30.467 INFO:teuthology.orchestra.run.smithi076.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-26T23:29:30.467 INFO:teuthology.orchestra.run.smithi076.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-26T23:29:30.467 INFO:teuthology.orchestra.run.smithi076.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-26T23:29:30.467 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-26T23:29:30.467 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-26T23:29:30.468 INFO:teuthology.orchestra.run.smithi076.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-26T23:29:30.468 INFO:teuthology.orchestra.run.smithi076.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-26T23:29:30.468 INFO:teuthology.orchestra.run.smithi076.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-26T23:29:30.468 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-26T23:29:30.468 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-26T23:29:30.468 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-26T23:29:30.468 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-26T23:29:30.468 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-26T23:29:30.468 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-26T23:29:30.468 INFO:teuthology.orchestra.run.smithi076.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-26T23:29:30.468 INFO:teuthology.orchestra.run.smithi076.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-26T23:29:30.468 INFO:teuthology.orchestra.run.smithi076.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-26T23:29:30.468 INFO:teuthology.orchestra.run.smithi076.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-26T23:29:30.468 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-26T23:29:30.469 INFO:teuthology.orchestra.run.smithi076.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-26T23:29:30.469 INFO:teuthology.orchestra.run.smithi076.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-26T23:29:30.469 INFO:teuthology.orchestra.run.smithi076.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-26T23:29:30.469 INFO:teuthology.orchestra.run.smithi076.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-26T23:29:30.469 INFO:teuthology.orchestra.run.smithi076.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-26T23:29:30.470 INFO:teuthology.orchestra.run.smithi076.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-26T23:29:30.470 INFO:teuthology.orchestra.run.smithi076.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-26T23:29:30.470 INFO:teuthology.orchestra.run.smithi076.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-26T23:29:30.470 INFO:teuthology.orchestra.run.smithi076.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-26T23:29:30.470 INFO:teuthology.orchestra.run.smithi076.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-26T23:29:30.470 INFO:teuthology.orchestra.run.smithi076.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-26T23:29:30.470 INFO:teuthology.orchestra.run.smithi076.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-26T23:29:30.470 INFO:teuthology.orchestra.run.smithi076.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-26T23:29:30.470 INFO:teuthology.orchestra.run.smithi076.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-26T23:29:30.470 INFO:teuthology.orchestra.run.smithi076.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-26T23:29:30.470 INFO:teuthology.orchestra.run.smithi076.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-26T23:29:30.470 INFO:teuthology.orchestra.run.smithi076.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-26T23:29:30.470 INFO:teuthology.orchestra.run.smithi076.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-26T23:29:30.471 INFO:teuthology.orchestra.run.smithi076.stdout:Installing weak dependencies: 2024-04-26T23:29:30.471 INFO:teuthology.orchestra.run.smithi076.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-26T23:29:30.471 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-26T23:29:30.471 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-04-26T23:29:30.471 INFO:teuthology.orchestra.run.smithi076.stdout:======================================================================================= 2024-04-26T23:29:30.471 INFO:teuthology.orchestra.run.smithi076.stdout:Install 148 Packages 2024-04-26T23:29:30.471 INFO:teuthology.orchestra.run.smithi076.stdout:Upgrade 4 Packages 2024-04-26T23:29:30.471 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-26T23:29:30.472 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 222 M 2024-04-26T23:29:30.472 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-04-26T23:29:31.090 INFO:teuthology.orchestra.run.smithi072.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-04-26T23:29:31.825 INFO:teuthology.orchestra.run.smithi076.stdout:(1/152): ceph-19.0.0-3306.gbc612ffb.el9.x86_64. 20 kB/s | 6.6 kB 00:00 2024-04-26T23:29:32.051 INFO:teuthology.orchestra.run.smithi076.stdout:(2/152): ceph-fuse-19.0.0-3306.gbc612ffb.el9.x8 3.9 MB/s | 889 kB 00:00 2024-04-26T23:29:32.109 INFO:teuthology.orchestra.run.smithi076.stdout:(3/152): ceph-immutable-object-cache-19.0.0-330 2.4 MB/s | 145 kB 00:00 2024-04-26T23:29:32.418 INFO:teuthology.orchestra.run.smithi076.stdout:(4/152): ceph-mds-19.0.0-3306.gbc612ffb.el9.x86 7.7 MB/s | 2.4 MB 00:00 2024-04-26T23:29:32.660 INFO:teuthology.orchestra.run.smithi076.stdout:(5/152): ceph-mgr-19.0.0-3306.gbc612ffb.el9.x86 6.3 MB/s | 1.5 MB 00:00 2024-04-26T23:29:32.752 INFO:teuthology.orchestra.run.smithi076.stdout:(6/152): ceph-base-19.0.0-3306.gbc612ffb.el9.x8 4.3 MB/s | 5.4 MB 00:01 2024-04-26T23:29:33.260 INFO:teuthology.orchestra.run.smithi076.stdout:(7/152): ceph-mon-19.0.0-3306.gbc612ffb.el9.x86 7.8 MB/s | 4.7 MB 00:00 2024-04-26T23:29:33.636 INFO:teuthology.orchestra.run.smithi072.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T23:29:33.638 INFO:teuthology.orchestra.run.smithi072.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T23:29:33.659 INFO:teuthology.orchestra.run.smithi072.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-26T23:29:33.661 INFO:teuthology.orchestra.run.smithi072.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-26T23:29:33.661 INFO:teuthology.orchestra.run.smithi076.stdout:(8/152): ceph-common-19.0.0-3306.gbc612ffb.el9. 9.8 MB/s | 21 MB 00:02 2024-04-26T23:29:33.662 INFO:teuthology.orchestra.run.smithi072.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-26T23:29:33.663 INFO:teuthology.orchestra.run.smithi072.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-26T23:29:33.666 INFO:teuthology.orchestra.run.smithi072.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-26T23:29:33.669 INFO:teuthology.orchestra.run.smithi072.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-26T23:29:33.671 INFO:teuthology.orchestra.run.smithi072.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-26T23:29:33.712 INFO:teuthology.orchestra.run.smithi076.stdout:(9/152): ceph-selinux-19.0.0-3306.gbc612ffb.el9 504 kB/s | 25 kB 00:00 2024-04-26T23:29:33.742 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-04-26T23:29:33.767 INFO:teuthology.orchestra.run.smithi072.stdout:======================================================================================= 2024-04-26T23:29:33.767 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2024-04-26T23:29:33.767 INFO:teuthology.orchestra.run.smithi072.stdout:======================================================================================= 2024-04-26T23:29:33.767 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-04-26T23:29:33.767 INFO:teuthology.orchestra.run.smithi072.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-26T23:29:33.767 INFO:teuthology.orchestra.run.smithi072.stdout: ceph x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 6.6 k 2024-04-26T23:29:33.767 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-base x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-26T23:29:33.767 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 889 k 2024-04-26T23:29:33.767 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 145 k 2024-04-26T23:29:33.767 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 1.5 M 2024-04-26T23:29:33.767 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 142 k 2024-04-26T23:29:33.767 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 3.5 M 2024-04-26T23:29:33.768 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 7.4 M 2024-04-26T23:29:33.768 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-rook noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 49 k 2024-04-26T23:29:33.768 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-radosgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 11 M 2024-04-26T23:29:33.768 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-test x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 49 M 2024-04-26T23:29:33.768 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-volume noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 273 k 2024-04-26T23:29:33.768 INFO:teuthology.orchestra.run.smithi072.stdout: cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 762 k 2024-04-26T23:29:33.768 INFO:teuthology.orchestra.run.smithi072.stdout: cephfs-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 225 k 2024-04-26T23:29:33.768 INFO:teuthology.orchestra.run.smithi072.stdout: cephfs-shell x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 27 k 2024-04-26T23:29:33.768 INFO:teuthology.orchestra.run.smithi072.stdout: cephfs-top noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 18 k 2024-04-26T23:29:33.768 INFO:teuthology.orchestra.run.smithi072.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-26T23:29:33.768 INFO:teuthology.orchestra.run.smithi072.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-26T23:29:33.768 INFO:teuthology.orchestra.run.smithi072.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-26T23:29:33.768 INFO:teuthology.orchestra.run.smithi072.stdout: libcephfs-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 32 k 2024-04-26T23:29:33.768 INFO:teuthology.orchestra.run.smithi072.stdout: libcephfs2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 747 k 2024-04-26T23:29:33.768 INFO:teuthology.orchestra.run.smithi072.stdout: librados-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 127 k 2024-04-26T23:29:33.769 INFO:teuthology.orchestra.run.smithi072.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-26T23:29:33.769 INFO:teuthology.orchestra.run.smithi072.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-26T23:29:33.769 INFO:teuthology.orchestra.run.smithi072.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-26T23:29:33.769 INFO:teuthology.orchestra.run.smithi072.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-26T23:29:33.769 INFO:teuthology.orchestra.run.smithi072.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-26T23:29:33.769 INFO:teuthology.orchestra.run.smithi072.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-26T23:29:33.769 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cephfs x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 161 k 2024-04-26T23:29:33.769 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rados x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 321 k 2024-04-26T23:29:33.769 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 298 k 2024-04-26T23:29:33.769 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 99 k 2024-04-26T23:29:33.769 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 85 k 2024-04-26T23:29:33.769 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-26T23:29:33.769 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-nbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 171 k 2024-04-26T23:29:33.769 INFO:teuthology.orchestra.run.smithi072.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-26T23:29:33.769 INFO:teuthology.orchestra.run.smithi072.stdout:Upgrading: 2024-04-26T23:29:33.770 INFO:teuthology.orchestra.run.smithi072.stdout: librados2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.4 M 2024-04-26T23:29:33.770 INFO:teuthology.orchestra.run.smithi072.stdout: librbd1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-26T23:29:33.770 INFO:teuthology.orchestra.run.smithi072.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-26T23:29:33.770 INFO:teuthology.orchestra.run.smithi072.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-26T23:29:33.770 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2024-04-26T23:29:33.770 INFO:teuthology.orchestra.run.smithi072.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-26T23:29:33.770 INFO:teuthology.orchestra.run.smithi072.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-26T23:29:33.770 INFO:teuthology.orchestra.run.smithi072.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-26T23:29:33.770 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 21 M 2024-04-26T23:29:33.770 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 34 k 2024-04-26T23:29:33.770 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mds x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 2.4 M 2024-04-26T23:29:33.770 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 249 k 2024-04-26T23:29:33.770 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mon x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 4.7 M 2024-04-26T23:29:33.770 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-osd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 17 M 2024-04-26T23:29:33.770 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 16 k 2024-04-26T23:29:33.771 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-selinux x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 25 k 2024-04-26T23:29:33.771 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-26T23:29:33.771 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-26T23:29:33.771 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-26T23:29:33.771 INFO:teuthology.orchestra.run.smithi072.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-26T23:29:33.771 INFO:teuthology.orchestra.run.smithi072.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-26T23:29:33.771 INFO:teuthology.orchestra.run.smithi072.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-26T23:29:33.771 INFO:teuthology.orchestra.run.smithi072.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-26T23:29:33.771 INFO:teuthology.orchestra.run.smithi072.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-26T23:29:33.771 INFO:teuthology.orchestra.run.smithi072.stdout: libcephsqlite x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 163 k 2024-04-26T23:29:33.771 INFO:teuthology.orchestra.run.smithi072.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-26T23:29:33.771 INFO:teuthology.orchestra.run.smithi072.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-26T23:29:33.771 INFO:teuthology.orchestra.run.smithi072.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-26T23:29:33.771 INFO:teuthology.orchestra.run.smithi072.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-26T23:29:33.772 INFO:teuthology.orchestra.run.smithi072.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-26T23:29:33.772 INFO:teuthology.orchestra.run.smithi072.stdout: libradosstriper1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 501 k 2024-04-26T23:29:33.772 INFO:teuthology.orchestra.run.smithi072.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-26T23:29:33.772 INFO:teuthology.orchestra.run.smithi072.stdout: librgw2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-26T23:29:33.772 INFO:teuthology.orchestra.run.smithi072.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-26T23:29:33.772 INFO:teuthology.orchestra.run.smithi072.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-26T23:29:33.772 INFO:teuthology.orchestra.run.smithi072.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-26T23:29:33.772 INFO:teuthology.orchestra.run.smithi072.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-26T23:29:33.772 INFO:teuthology.orchestra.run.smithi072.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-26T23:29:33.772 INFO:teuthology.orchestra.run.smithi072.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-26T23:29:33.772 INFO:teuthology.orchestra.run.smithi072.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-26T23:29:33.772 INFO:teuthology.orchestra.run.smithi072.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-26T23:29:33.772 INFO:teuthology.orchestra.run.smithi072.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-26T23:29:33.772 INFO:teuthology.orchestra.run.smithi072.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-26T23:29:33.772 INFO:teuthology.orchestra.run.smithi072.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-26T23:29:33.772 INFO:teuthology.orchestra.run.smithi072.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-26T23:29:33.773 INFO:teuthology.orchestra.run.smithi072.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-26T23:29:33.773 INFO:teuthology.orchestra.run.smithi072.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-26T23:29:33.773 INFO:teuthology.orchestra.run.smithi072.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-26T23:29:33.773 INFO:teuthology.orchestra.run.smithi072.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-26T23:29:33.773 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-26T23:29:33.773 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ceph-argparse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 45 k 2024-04-26T23:29:33.773 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 135 k 2024-04-26T23:29:33.773 INFO:teuthology.orchestra.run.smithi072.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-26T23:29:33.773 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-26T23:29:33.773 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-26T23:29:33.773 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-26T23:29:33.773 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-26T23:29:33.773 INFO:teuthology.orchestra.run.smithi072.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-26T23:29:33.773 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-26T23:29:33.774 INFO:teuthology.orchestra.run.smithi072.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-26T23:29:33.774 INFO:teuthology.orchestra.run.smithi072.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-26T23:29:33.774 INFO:teuthology.orchestra.run.smithi072.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-26T23:29:33.774 INFO:teuthology.orchestra.run.smithi072.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-26T23:29:33.774 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-26T23:29:33.774 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-26T23:29:33.774 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-26T23:29:33.774 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-26T23:29:33.774 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-26T23:29:33.774 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-26T23:29:33.774 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-26T23:29:33.774 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-26T23:29:33.774 INFO:teuthology.orchestra.run.smithi072.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-26T23:29:33.774 INFO:teuthology.orchestra.run.smithi072.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-26T23:29:33.774 INFO:teuthology.orchestra.run.smithi072.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-26T23:29:33.775 INFO:teuthology.orchestra.run.smithi072.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-26T23:29:33.775 INFO:teuthology.orchestra.run.smithi072.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-26T23:29:33.775 INFO:teuthology.orchestra.run.smithi072.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-26T23:29:33.775 INFO:teuthology.orchestra.run.smithi072.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-26T23:29:33.776 INFO:teuthology.orchestra.run.smithi072.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-26T23:29:33.776 INFO:teuthology.orchestra.run.smithi072.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-26T23:29:33.776 INFO:teuthology.orchestra.run.smithi072.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-26T23:29:33.776 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-26T23:29:33.776 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-26T23:29:33.776 INFO:teuthology.orchestra.run.smithi072.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-26T23:29:33.776 INFO:teuthology.orchestra.run.smithi072.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-26T23:29:33.776 INFO:teuthology.orchestra.run.smithi072.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-26T23:29:33.777 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-26T23:29:33.777 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-26T23:29:33.777 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-26T23:29:33.777 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-26T23:29:33.777 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-26T23:29:33.777 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-26T23:29:33.777 INFO:teuthology.orchestra.run.smithi072.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-26T23:29:33.777 INFO:teuthology.orchestra.run.smithi072.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-26T23:29:33.777 INFO:teuthology.orchestra.run.smithi072.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-26T23:29:33.777 INFO:teuthology.orchestra.run.smithi072.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-26T23:29:33.777 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-26T23:29:33.778 INFO:teuthology.orchestra.run.smithi072.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-26T23:29:33.778 INFO:teuthology.orchestra.run.smithi072.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-26T23:29:33.778 INFO:teuthology.orchestra.run.smithi072.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-26T23:29:33.778 INFO:teuthology.orchestra.run.smithi072.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-26T23:29:33.778 INFO:teuthology.orchestra.run.smithi072.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-26T23:29:33.778 INFO:teuthology.orchestra.run.smithi072.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-26T23:29:33.778 INFO:teuthology.orchestra.run.smithi072.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-26T23:29:33.778 INFO:teuthology.orchestra.run.smithi072.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-26T23:29:33.778 INFO:teuthology.orchestra.run.smithi072.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-26T23:29:33.778 INFO:teuthology.orchestra.run.smithi072.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-26T23:29:33.778 INFO:teuthology.orchestra.run.smithi072.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-26T23:29:33.778 INFO:teuthology.orchestra.run.smithi072.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-26T23:29:33.779 INFO:teuthology.orchestra.run.smithi072.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-26T23:29:33.779 INFO:teuthology.orchestra.run.smithi072.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-26T23:29:33.779 INFO:teuthology.orchestra.run.smithi072.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-26T23:29:33.779 INFO:teuthology.orchestra.run.smithi072.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-26T23:29:33.779 INFO:teuthology.orchestra.run.smithi072.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-26T23:29:33.779 INFO:teuthology.orchestra.run.smithi072.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-26T23:29:33.779 INFO:teuthology.orchestra.run.smithi072.stdout:Installing weak dependencies: 2024-04-26T23:29:33.779 INFO:teuthology.orchestra.run.smithi072.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-26T23:29:33.779 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:29:33.779 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-04-26T23:29:33.779 INFO:teuthology.orchestra.run.smithi072.stdout:======================================================================================= 2024-04-26T23:29:33.779 INFO:teuthology.orchestra.run.smithi072.stdout:Install 148 Packages 2024-04-26T23:29:33.779 INFO:teuthology.orchestra.run.smithi072.stdout:Upgrade 4 Packages 2024-04-26T23:29:33.779 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:29:33.780 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 222 M 2024-04-26T23:29:33.780 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-04-26T23:29:34.170 INFO:teuthology.orchestra.run.smithi076.stdout:(10/152): ceph-radosgw-19.0.0-3306.gbc612ffb.el 12 MB/s | 11 MB 00:00 2024-04-26T23:29:34.246 INFO:teuthology.orchestra.run.smithi076.stdout:(11/152): cephfs-mirror-19.0.0-3306.gbc612ffb.e 2.9 MB/s | 225 kB 00:00 2024-04-26T23:29:34.304 INFO:teuthology.orchestra.run.smithi076.stdout:(12/152): cephfs-shell-19.0.0-3306.gbc612ffb.el 465 kB/s | 27 kB 00:00 2024-04-26T23:29:34.338 INFO:teuthology.orchestra.run.smithi076.stdout:(13/152): libcephfs-devel-19.0.0-3306.gbc612ffb 945 kB/s | 32 kB 00:00 2024-04-26T23:29:34.438 INFO:teuthology.orchestra.run.smithi076.stdout:(14/152): libcephfs2-19.0.0-3306.gbc612ffb.el9. 7.3 MB/s | 747 kB 00:00 2024-04-26T23:29:34.497 INFO:teuthology.orchestra.run.smithi076.stdout:(15/152): libcephsqlite-19.0.0-3306.gbc612ffb.e 2.7 MB/s | 163 kB 00:00 2024-04-26T23:29:34.555 INFO:teuthology.orchestra.run.smithi076.stdout:(16/152): librados-devel-19.0.0-3306.gbc612ffb. 2.1 MB/s | 127 kB 00:00 2024-04-26T23:29:34.731 INFO:teuthology.orchestra.run.smithi076.stdout:(17/152): ceph-osd-19.0.0-3306.gbc612ffb.el9.x8 8.5 MB/s | 17 MB 00:01 2024-04-26T23:29:34.756 INFO:teuthology.orchestra.run.smithi076.stdout:(18/152): libradosstriper1-19.0.0-3306.gbc612ff 2.4 MB/s | 501 kB 00:00 2024-04-26T23:29:34.830 INFO:teuthology.orchestra.run.smithi076.stdout:(19/152): python3-ceph-argparse-19.0.0-3306.gbc 617 kB/s | 45 kB 00:00 2024-04-26T23:29:34.899 INFO:teuthology.orchestra.run.smithi076.stdout:(20/152): python3-ceph-common-19.0.0-3306.gbc61 1.9 MB/s | 135 kB 00:00 2024-04-26T23:29:34.974 INFO:teuthology.orchestra.run.smithi076.stdout:(21/152): python3-cephfs-19.0.0-3306.gbc612ffb. 2.1 MB/s | 161 kB 00:00 2024-04-26T23:29:35.050 INFO:teuthology.orchestra.run.smithi076.stdout:(22/152): python3-rados-19.0.0-3306.gbc612ffb.e 4.2 MB/s | 321 kB 00:00 2024-04-26T23:29:35.125 INFO:teuthology.orchestra.run.smithi076.stdout:(23/152): python3-rbd-19.0.0-3306.gbc612ffb.el9 3.9 MB/s | 298 kB 00:00 2024-04-26T23:29:35.192 INFO:teuthology.orchestra.run.smithi076.stdout:(24/152): python3-rgw-19.0.0-3306.gbc612ffb.el9 1.5 MB/s | 99 kB 00:00 2024-04-26T23:29:35.194 INFO:teuthology.orchestra.run.smithi072.stdout:(1/152): ceph-19.0.0-3306.gbc612ffb.el9.x86_64. 16 kB/s | 6.6 kB 00:00 2024-04-26T23:29:35.267 INFO:teuthology.orchestra.run.smithi076.stdout:(25/152): librgw2-19.0.0-3306.gbc612ffb.el9.x86 10 MB/s | 5.4 MB 00:00 2024-04-26T23:29:35.301 INFO:teuthology.orchestra.run.smithi076.stdout:(26/152): rbd-fuse-19.0.0-3306.gbc612ffb.el9.x8 785 kB/s | 85 kB 00:00 2024-04-26T23:29:35.359 INFO:teuthology.orchestra.run.smithi076.stdout:(27/152): rbd-nbd-19.0.0-3306.gbc612ffb.el9.x86 2.9 MB/s | 171 kB 00:00 2024-04-26T23:29:35.435 INFO:teuthology.orchestra.run.smithi076.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3306.g 455 kB/s | 34 kB 00:00 2024-04-26T23:29:35.510 INFO:teuthology.orchestra.run.smithi076.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3306.gbc612ff 1.9 MB/s | 142 kB 00:00 2024-04-26T23:29:35.594 INFO:teuthology.orchestra.run.smithi076.stdout:(30/152): rbd-mirror-19.0.0-3306.gbc612ffb.el9. 9.5 MB/s | 3.1 MB 00:00 2024-04-26T23:29:35.945 INFO:teuthology.orchestra.run.smithi072.stdout:(2/152): ceph-fuse-19.0.0-3306.gbc612ffb.el9.x8 1.2 MB/s | 889 kB 00:00 2024-04-26T23:29:35.953 INFO:teuthology.orchestra.run.smithi076.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3306.gbc612 8.0 MB/s | 3.5 MB 00:00 2024-04-26T23:29:36.028 INFO:teuthology.orchestra.run.smithi076.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3306.gbc 3.3 MB/s | 249 kB 00:00 2024-04-26T23:29:36.095 INFO:teuthology.orchestra.run.smithi076.stdout:(33/152): ceph-mgr-rook-19.0.0-3306.gbc612ffb.e 737 kB/s | 49 kB 00:00 2024-04-26T23:29:36.104 INFO:teuthology.orchestra.run.smithi072.stdout:(3/152): ceph-immutable-object-cache-19.0.0-330 920 kB/s | 145 kB 00:00 2024-04-26T23:29:36.162 INFO:teuthology.orchestra.run.smithi076.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3306.gb 247 kB/s | 16 kB 00:00 2024-04-26T23:29:36.237 INFO:teuthology.orchestra.run.smithi076.stdout:(35/152): ceph-volume-19.0.0-3306.gbc612ffb.el9 3.6 MB/s | 273 kB 00:00 2024-04-26T23:29:36.387 INFO:teuthology.orchestra.run.smithi076.stdout:(36/152): cephadm-19.0.0-3306.gbc612ffb.el9.noa 5.0 MB/s | 762 kB 00:00 2024-04-26T23:29:36.463 INFO:teuthology.orchestra.run.smithi076.stdout:(37/152): cephfs-top-19.0.0-3306.gbc612ffb.el9. 243 kB/s | 18 kB 00:00 2024-04-26T23:29:36.563 INFO:teuthology.orchestra.run.smithi076.stdout:(38/152): ceph-mgr-diskprediction-local-19.0.0- 7.6 MB/s | 7.4 MB 00:00 2024-04-26T23:29:37.106 INFO:teuthology.orchestra.run.smithi076.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 125 kB/s | 80 kB 00:00 2024-04-26T23:29:37.321 INFO:teuthology.orchestra.run.smithi072.stdout:(4/152): ceph-mds-19.0.0-3306.gbc612ffb.el9.x86 1.9 MB/s | 2.4 MB 00:01 2024-04-26T23:29:37.964 INFO:teuthology.orchestra.run.smithi076.stdout:(40/152): ceph-test-19.0.0-3306.gbc612ffb.el9.x 12 MB/s | 49 MB 00:04 2024-04-26T23:29:38.013 INFO:teuthology.orchestra.run.smithi072.stdout:(5/152): ceph-mgr-19.0.0-3306.gbc612ffb.el9.x86 2.2 MB/s | 1.5 MB 00:00 2024-04-26T23:29:38.256 INFO:teuthology.orchestra.run.smithi076.stdout:(41/152): libgfortran-11.4.1-3.el9.x86_64.rpm 474 kB/s | 803 kB 00:01 2024-04-26T23:29:38.330 INFO:teuthology.orchestra.run.smithi072.stdout:(6/152): ceph-base-19.0.0-3306.gbc612ffb.el9.x8 1.5 MB/s | 5.4 MB 00:03 2024-04-26T23:29:38.364 INFO:teuthology.orchestra.run.smithi076.stdout:(42/152): libquadmath-11.4.1-3.el9.x86_64.rpm 155 kB/s | 195 kB 00:01 2024-04-26T23:29:38.758 INFO:teuthology.orchestra.run.smithi072.stdout:(7/152): ceph-common-19.0.0-3306.gbc612ffb.el9. 5.3 MB/s | 21 MB 00:03 2024-04-26T23:29:38.865 INFO:teuthology.orchestra.run.smithi076.stdout:(43/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 281 kB/s | 253 kB 00:00 2024-04-26T23:29:39.073 INFO:teuthology.orchestra.run.smithi076.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 150 kB/s | 106 kB 00:00 2024-04-26T23:29:39.533 INFO:teuthology.orchestra.run.smithi072.stdout:(8/152): ceph-radosgw-19.0.0-3306.gbc612ffb.el9 14 MB/s | 11 MB 00:00 2024-04-26T23:29:39.549 INFO:teuthology.orchestra.run.smithi076.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 198 kB/s | 135 kB 00:00 2024-04-26T23:29:39.600 INFO:teuthology.orchestra.run.smithi072.stdout:(9/152): ceph-selinux-19.0.0-3306.gbc612ffb.el9 382 kB/s | 25 kB 00:00 2024-04-26T23:29:39.816 INFO:teuthology.orchestra.run.smithi076.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 175 kB/s | 125 kB 00:00 2024-04-26T23:29:39.967 INFO:teuthology.orchestra.run.smithi072.stdout:(10/152): ceph-mon-19.0.0-3306.gbc612ffb.el9.x8 2.4 MB/s | 4.7 MB 00:01 2024-04-26T23:29:40.110 INFO:teuthology.orchestra.run.smithi072.stdout:(11/152): cephfs-mirror-19.0.0-3306.gbc612ffb.e 1.6 MB/s | 225 kB 00:00 2024-04-26T23:29:40.202 INFO:teuthology.orchestra.run.smithi072.stdout:(12/152): cephfs-shell-19.0.0-3306.gbc612ffb.el 297 kB/s | 27 kB 00:00 2024-04-26T23:29:40.327 INFO:teuthology.orchestra.run.smithi072.stdout:(13/152): libcephfs-devel-19.0.0-3306.gbc612ffb 255 kB/s | 32 kB 00:00 2024-04-26T23:29:40.333 INFO:teuthology.orchestra.run.smithi076.stdout:(47/152): python3-cryptography-36.0.1-4.el9.x86 604 kB/s | 1.2 MB 00:02 2024-04-26T23:29:40.358 INFO:teuthology.orchestra.run.smithi076.stdout:(48/152): python3-urllib3-1.26.5-5.el9.noarch.r 265 kB/s | 215 kB 00:00 2024-04-26T23:29:40.753 INFO:teuthology.orchestra.run.smithi072.stdout:(14/152): libcephfs2-19.0.0-3306.gbc612ffb.el9. 1.7 MB/s | 747 kB 00:00 2024-04-26T23:29:40.953 INFO:teuthology.orchestra.run.smithi072.stdout:(15/152): libcephsqlite-19.0.0-3306.gbc612ffb.e 820 kB/s | 163 kB 00:00 2024-04-26T23:29:41.058 INFO:teuthology.orchestra.run.smithi076.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 153 kB/s | 107 kB 00:00 2024-04-26T23:29:41.062 INFO:teuthology.orchestra.run.smithi072.stdout:(16/152): librados-devel-19.0.0-3306.gbc612ffb. 1.2 MB/s | 127 kB 00:00 2024-04-26T23:29:41.267 INFO:teuthology.orchestra.run.smithi076.stdout:(50/152): automake-1.16.2-8.el9.noarch.rpm 475 kB/s | 689 kB 00:01 2024-04-26T23:29:41.521 INFO:teuthology.orchestra.run.smithi072.stdout:(17/152): libradosstriper1-19.0.0-3306.gbc612ff 1.1 MB/s | 501 kB 00:00 2024-04-26T23:29:41.701 INFO:teuthology.orchestra.run.smithi076.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 33 kB/s | 21 kB 00:00 2024-04-26T23:29:42.076 INFO:teuthology.orchestra.run.smithi076.stdout:(52/152): bison-3.7.4-5.el9.x86_64.rpm 541 kB/s | 943 kB 00:01 2024-04-26T23:29:42.218 INFO:teuthology.orchestra.run.smithi076.stdout:(53/152): flex-2.6.4-9.el9.x86_64.rpm 330 kB/s | 313 kB 00:00 2024-04-26T23:29:42.252 INFO:teuthology.orchestra.run.smithi076.stdout:(54/152): flexiblas-3.0.4-8.el9.x86_64.rpm 59 kB/s | 33 kB 00:00 2024-04-26T23:29:42.613 INFO:teuthology.orchestra.run.smithi072.stdout:(18/152): ceph-test-19.0.0-3306.gbc612ffb.el9.x 16 MB/s | 49 MB 00:03 2024-04-26T23:29:42.682 INFO:teuthology.orchestra.run.smithi072.stdout:(19/152): python3-ceph-argparse-19.0.0-3306.gbc 687 kB/s | 45 kB 00:00 2024-04-26T23:29:42.710 INFO:teuthology.orchestra.run.smithi076.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 37 kB/s | 18 kB 00:00 2024-04-26T23:29:42.736 INFO:teuthology.orchestra.run.smithi076.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 23 kB/s | 11 kB 00:00 2024-04-26T23:29:42.750 INFO:teuthology.orchestra.run.smithi072.stdout:(20/152): python3-ceph-common-19.0.0-3306.gbc61 2.0 MB/s | 135 kB 00:00 2024-04-26T23:29:42.823 INFO:teuthology.orchestra.run.smithi072.stdout:(21/152): python3-cephfs-19.0.0-3306.gbc612ffb. 2.1 MB/s | 161 kB 00:00 2024-04-26T23:29:42.917 INFO:teuthology.orchestra.run.smithi072.stdout:(22/152): python3-rados-19.0.0-3306.gbc612ffb.e 3.4 MB/s | 321 kB 00:00 2024-04-26T23:29:43.001 INFO:teuthology.orchestra.run.smithi072.stdout:(23/152): python3-rbd-19.0.0-3306.gbc612ffb.el9 3.5 MB/s | 298 kB 00:00 2024-04-26T23:29:43.076 INFO:teuthology.orchestra.run.smithi072.stdout:(24/152): python3-rgw-19.0.0-3306.gbc612ffb.el9 1.3 MB/s | 99 kB 00:00 2024-04-26T23:29:43.144 INFO:teuthology.orchestra.run.smithi072.stdout:(25/152): rbd-fuse-19.0.0-3306.gbc612ffb.el9.x8 1.3 MB/s | 85 kB 00:00 2024-04-26T23:29:43.327 INFO:teuthology.orchestra.run.smithi076.stdout:(57/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 76 kB/s | 45 kB 00:00 2024-04-26T23:29:43.402 INFO:teuthology.orchestra.run.smithi072.stdout:(26/152): rbd-mirror-19.0.0-3306.gbc612ffb.el9. 12 MB/s | 3.1 MB 00:00 2024-04-26T23:29:43.486 INFO:teuthology.orchestra.run.smithi072.stdout:(27/152): rbd-nbd-19.0.0-3306.gbc612ffb.el9.x86 2.0 MB/s | 171 kB 00:00 2024-04-26T23:29:43.503 INFO:teuthology.orchestra.run.smithi076.stdout:(58/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 202 kB/s | 160 kB 00:00 2024-04-26T23:29:43.637 INFO:teuthology.orchestra.run.smithi072.stdout:(28/152): librgw2-19.0.0-3306.gbc612ffb.el9.x86 2.5 MB/s | 5.4 MB 00:02 2024-04-26T23:29:43.671 INFO:teuthology.orchestra.run.smithi072.stdout:(29/152): ceph-grafana-dashboards-19.0.0-3306.g 187 kB/s | 34 kB 00:00 2024-04-26T23:29:43.705 INFO:teuthology.orchestra.run.smithi072.stdout:(30/152): ceph-mgr-cephadm-19.0.0-3306.gbc612ff 2.1 MB/s | 142 kB 00:00 2024-04-26T23:29:43.972 INFO:teuthology.orchestra.run.smithi072.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3306.gbc612 12 MB/s | 3.5 MB 00:00 2024-04-26T23:29:44.047 INFO:teuthology.orchestra.run.smithi072.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3306.gbc 3.3 MB/s | 249 kB 00:00 2024-04-26T23:29:44.156 INFO:teuthology.orchestra.run.smithi072.stdout:(33/152): ceph-mgr-rook-19.0.0-3306.gbc612ffb.e 460 kB/s | 49 kB 00:00 2024-04-26T23:29:44.232 INFO:teuthology.orchestra.run.smithi072.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3306.gb 221 kB/s | 16 kB 00:00 2024-04-26T23:29:44.308 INFO:teuthology.orchestra.run.smithi072.stdout:(35/152): ceph-volume-19.0.0-3306.gbc612ffb.el9 3.6 MB/s | 273 kB 00:00 2024-04-26T23:29:44.425 INFO:teuthology.orchestra.run.smithi072.stdout:(36/152): cephadm-19.0.0-3306.gbc612ffb.el9.noa 6.4 MB/s | 762 kB 00:00 2024-04-26T23:29:44.493 INFO:teuthology.orchestra.run.smithi072.stdout:(37/152): cephfs-top-19.0.0-3306.gbc612ffb.el9. 270 kB/s | 18 kB 00:00 2024-04-26T23:29:44.869 INFO:teuthology.orchestra.run.smithi076.stdout:(59/152): libtool-2.4.6-45.el9.x86_64.rpm 426 kB/s | 582 kB 00:01 2024-04-26T23:29:44.895 INFO:teuthology.orchestra.run.smithi076.stdout:(60/152): librdkafka-1.6.1-102.el9.x86_64.rpm 423 kB/s | 662 kB 00:01 2024-04-26T23:29:45.194 INFO:teuthology.orchestra.run.smithi072.stdout:(38/152): ceph-osd-19.0.0-3306.gbc612ffb.el9.x8 2.5 MB/s | 17 MB 00:06 2024-04-26T23:29:45.412 INFO:teuthology.orchestra.run.smithi076.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 47 kB/s | 25 kB 00:00 2024-04-26T23:29:45.529 INFO:teuthology.orchestra.run.smithi076.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 75 kB/s | 47 kB 00:00 2024-04-26T23:29:45.828 INFO:teuthology.orchestra.run.smithi072.stdout:(39/152): ceph-mgr-diskprediction-local-19.0.0- 3.5 MB/s | 7.4 MB 00:02 2024-04-26T23:29:46.079 INFO:teuthology.orchestra.run.smithi076.stdout:(63/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 765 kB/s | 3.0 MB 00:04 2024-04-26T23:29:46.280 INFO:teuthology.orchestra.run.smithi076.stdout:(64/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 337 kB/s | 292 kB 00:00 2024-04-26T23:29:46.330 INFO:teuthology.orchestra.run.smithi076.stdout:(65/152): lua-5.4.4-4.el9.x86_64.rpm 235 kB/s | 188 kB 00:00 2024-04-26T23:29:46.622 INFO:teuthology.orchestra.run.smithi076.stdout:(66/152): openblas-0.3.21-2.el9.x86_64.rpm 65 kB/s | 35 kB 00:00 2024-04-26T23:29:48.922 INFO:teuthology.orchestra.run.smithi076.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 707 kB/s | 1.6 MB 00:02 2024-04-26T23:29:50.414 INFO:teuthology.orchestra.run.smithi076.stdout:(68/152): postgresql-contrib-13.14-1.el9.x86_64 611 kB/s | 911 kB 00:01 2024-04-26T23:29:51.097 INFO:teuthology.orchestra.run.smithi076.stdout:(69/152): postgresql-private-libs-13.14-1.el9.x 195 kB/s | 133 kB 00:00 2024-04-26T23:29:51.998 INFO:teuthology.orchestra.run.smithi076.stdout:(70/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 799 kB/s | 4.4 MB 00:05 2024-04-26T23:29:52.123 INFO:teuthology.orchestra.run.smithi076.stdout:(71/152): openblas-openmp-0.3.21-2.el9.x86_64.r 834 kB/s | 4.7 MB 00:05 2024-04-26T23:29:52.806 INFO:teuthology.orchestra.run.smithi076.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 128 kB/s | 87 kB 00:00 2024-04-26T23:29:53.807 INFO:teuthology.orchestra.run.smithi076.stdout:(73/152): protobuf-3.14.0-13.el9.x86_64.rpm 571 kB/s | 1.0 MB 00:01 2024-04-26T23:29:54.590 INFO:teuthology.orchestra.run.smithi076.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 312 kB/s | 244 kB 00:00 2024-04-26T23:29:55.390 INFO:teuthology.orchestra.run.smithi076.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 310 kB/s | 248 kB 00:00 2024-04-26T23:29:55.899 INFO:teuthology.orchestra.run.smithi076.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 50 kB/s | 26 kB 00:00 2024-04-26T23:29:56.374 INFO:teuthology.orchestra.run.smithi076.stdout:(77/152): python3-jsonpointer-2.0-4.el9.noarch. 40 kB/s | 19 kB 00:00 2024-04-26T23:29:57.141 INFO:teuthology.orchestra.run.smithi076.stdout:(78/152): python3-mako-1.1.4-6.el9.noarch.rpm 225 kB/s | 172 kB 00:00 2024-04-26T23:29:57.692 INFO:teuthology.orchestra.run.smithi076.stdout:(79/152): python3-markupsafe-1.1.1-12.el9.x86_6 63 kB/s | 35 kB 00:00 2024-04-26T23:29:58.258 INFO:teuthology.orchestra.run.smithi076.stdout:(80/152): postgresql-server-13.14-1.el9.x86_64. 826 kB/s | 5.8 MB 00:07 2024-04-26T23:29:59.067 INFO:teuthology.orchestra.run.smithi076.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 310 kB/s | 250 kB 00:00 2024-04-26T23:29:59.876 INFO:teuthology.orchestra.run.smithi076.stdout:(82/152): python3-babel-2.9.1-2.el9.noarch.rpm 864 kB/s | 6.0 MB 00:07 2024-04-26T23:29:59.901 INFO:teuthology.orchestra.run.smithi076.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 267 kB/s | 222 kB 00:00 2024-04-26T23:30:00.468 INFO:teuthology.orchestra.run.smithi076.stdout:(84/152): python3-prettytable-0.7.2-27.el9.noar 75 kB/s | 42 kB 00:00 2024-04-26T23:30:00.493 INFO:teuthology.orchestra.run.smithi076.stdout:(85/152): python3-packaging-20.9-5.el9.noarch.r 125 kB/s | 77 kB 00:00 2024-04-26T23:30:01.210 INFO:teuthology.orchestra.run.smithi076.stdout:(86/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 221 kB/s | 159 kB 00:00 2024-04-26T23:30:01.336 INFO:teuthology.orchestra.run.smithi076.stdout:(87/152): python3-protobuf-3.14.0-13.el9.noarch 311 kB/s | 269 kB 00:00 2024-04-26T23:30:01.911 INFO:teuthology.orchestra.run.smithi076.stdout:(88/152): python3-pytz-2021.1-5.el9.noarch.rpm 89 kB/s | 51 kB 00:00 2024-04-26T23:30:02.086 INFO:teuthology.orchestra.run.smithi076.stdout:(89/152): python3-pyasn1-modules-0.4.8-6.el9.no 318 kB/s | 279 kB 00:00 2024-04-26T23:30:02.503 INFO:teuthology.orchestra.run.smithi076.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 91 kB/s | 54 kB 00:00 2024-04-26T23:30:03.062 INFO:teuthology.orchestra.run.smithi076.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 75 kB/s | 42 kB 00:00 2024-04-26T23:30:03.621 INFO:teuthology.orchestra.run.smithi076.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 79 kB/s | 44 kB 00:00 2024-04-26T23:30:04.154 INFO:teuthology.orchestra.run.smithi076.stdout:(93/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 808 kB/s | 5.1 MB 00:06 2024-04-26T23:30:04.438 INFO:teuthology.orchestra.run.smithi076.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 270 kB/s | 221 kB 00:00 2024-04-26T23:30:04.688 INFO:teuthology.orchestra.run.smithi076.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 67 kB/s | 36 kB 00:00 2024-04-26T23:30:05.005 INFO:teuthology.orchestra.run.smithi076.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 84 kB/s | 47 kB 00:00 2024-04-26T23:30:05.597 INFO:teuthology.orchestra.run.smithi076.stdout:(97/152): uuid-1.6.2-55.el9.x86_64.rpm 98 kB/s | 58 kB 00:00 2024-04-26T23:30:05.622 INFO:teuthology.orchestra.run.smithi076.stdout:(98/152): socat-1.7.4.1-5.el9.x86_64.rpm 326 kB/s | 305 kB 00:00 2024-04-26T23:30:06.106 INFO:teuthology.orchestra.run.smithi076.stdout:(99/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 4.0 MB/s | 16 MB 00:04 2024-04-26T23:30:06.265 INFO:teuthology.orchestra.run.smithi076.stdout:(100/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 140 kB/s | 94 kB 00:00 2024-04-26T23:30:06.348 INFO:teuthology.orchestra.run.smithi076.stdout:(101/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 88 kB/s | 64 kB 00:00 2024-04-26T23:30:06.432 INFO:teuthology.orchestra.run.smithi076.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 133 kB/s | 22 kB 00:00 2024-04-26T23:30:06.657 INFO:teuthology.orchestra.run.smithi076.stdout:(103/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 81 kB/s | 45 kB 00:00 2024-04-26T23:30:06.682 INFO:teuthology.orchestra.run.smithi076.stdout:(104/152): protobuf-compiler-3.14.0-13.el9.x86_ 2.5 MB/s | 863 kB 00:00 2024-04-26T23:30:06.916 INFO:teuthology.orchestra.run.smithi076.stdout:(105/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.1 MB/s | 548 kB 00:00 2024-04-26T23:30:06.933 INFO:teuthology.orchestra.run.smithi076.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 78 kB/s | 19 kB 00:00 2024-04-26T23:30:07.025 INFO:teuthology.orchestra.run.smithi076.stdout:(107/152): libarrow-doc-9.0.0-11.el9.noarch.rpm 284 kB/s | 26 kB 00:00 2024-04-26T23:30:07.075 INFO:teuthology.orchestra.run.smithi076.stdout:(108/152): gperftools-libs-2.9.1-2.el9.x86_64.r 740 kB/s | 309 kB 00:00 2024-04-26T23:30:07.151 INFO:teuthology.orchestra.run.smithi076.stdout:(109/152): libunwind-1.6.2-1.el9.x86_64.rpm 896 kB/s | 67 kB 00:00 2024-04-26T23:30:07.176 INFO:teuthology.orchestra.run.smithi076.stdout:(110/152): liboath-2.6.7-2.el9.x86_64.rpm 322 kB/s | 49 kB 00:00 2024-04-26T23:30:07.244 INFO:teuthology.orchestra.run.smithi076.stdout:(111/152): libarrow-9.0.0-11.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-04-26T23:30:07.269 INFO:teuthology.orchestra.run.smithi076.stdout:(112/152): luarocks-3.9.2-1.el9.noarch.rpm 1.3 MB/s | 151 kB 00:00 2024-04-26T23:30:07.344 INFO:teuthology.orchestra.run.smithi076.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 5.3 MB/s | 548 kB 00:00 2024-04-26T23:30:07.369 INFO:teuthology.orchestra.run.smithi076.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 433 kB/s | 43 kB 00:00 2024-04-26T23:30:07.462 INFO:teuthology.orchestra.run.smithi076.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 275 kB/s | 32 kB 00:00 2024-04-26T23:30:07.486 INFO:teuthology.orchestra.run.smithi076.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 128 kB/s | 15 kB 00:00 2024-04-26T23:30:07.537 INFO:teuthology.orchestra.run.smithi076.stdout:(117/152): parquet-libs-9.0.0-11.el9.x86_64.rpm 2.3 MB/s | 839 kB 00:00 2024-04-26T23:30:07.596 INFO:teuthology.orchestra.run.smithi076.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 3.2 MB/s | 358 kB 00:00 2024-04-26T23:30:07.629 INFO:teuthology.orchestra.run.smithi076.stdout:(119/152): python3-cheroot-8.6.0-4.el9.noarch.r 1.0 MB/s | 172 kB 00:00 2024-04-26T23:30:07.664 INFO:teuthology.orchestra.run.smithi076.stdout:(120/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.4 MB/s | 303 kB 00:00 2024-04-26T23:30:07.722 INFO:teuthology.orchestra.run.smithi076.stdout:(121/152): python3-colorama-0.4.4-12.el9.noarch 281 kB/s | 35 kB 00:00 2024-04-26T23:30:07.747 INFO:teuthology.orchestra.run.smithi076.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 1.7 MB/s | 206 kB 00:00 2024-04-26T23:30:07.831 INFO:teuthology.orchestra.run.smithi076.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 1.3 MB/s | 144 kB 00:00 2024-04-26T23:30:07.856 INFO:teuthology.orchestra.run.smithi076.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 98 kB/s | 11 kB 00:00 2024-04-26T23:30:07.915 INFO:teuthology.orchestra.run.smithi076.stdout:(125/152): python3-grpcio-1.46.7-10.el9.x86_64. 8.1 MB/s | 2.0 MB 00:00 2024-04-26T23:30:07.940 INFO:teuthology.orchestra.run.smithi076.stdout:(126/152): python3-jaraco-classes-3.2.1-5.el9.n 162 kB/s | 18 kB 00:00 2024-04-26T23:30:07.965 INFO:teuthology.orchestra.run.smithi076.stdout:(127/152): python3-jaraco-collections-3.0.0-8.e 212 kB/s | 23 kB 00:00 2024-04-26T23:30:07.991 INFO:teuthology.orchestra.run.smithi076.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 257 kB/s | 19 kB 00:00 2024-04-26T23:30:08.024 INFO:teuthology.orchestra.run.smithi076.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 234 kB/s | 20 kB 00:00 2024-04-26T23:30:08.074 INFO:teuthology.orchestra.run.smithi076.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 555 kB/s | 46 kB 00:00 2024-04-26T23:30:08.100 INFO:teuthology.orchestra.run.smithi076.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 1.0 MB/s | 79 kB 00:00 2024-04-26T23:30:08.161 INFO:teuthology.orchestra.run.smithi076.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 689 kB/s | 58 kB 00:00 2024-04-26T23:30:08.194 INFO:teuthology.orchestra.run.smithi076.stdout:(133/152): python3-kubernetes-26.1.0-2.el9.noar 4.5 MB/s | 1.0 MB 00:00 2024-04-26T23:30:08.224 INFO:teuthology.orchestra.run.smithi076.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-04-26T23:30:08.237 INFO:teuthology.orchestra.run.smithi076.stdout:(135/152): python3-portend-3.1.0-2.el9.noarch.r 217 kB/s | 16 kB 00:00 2024-04-26T23:30:08.272 INFO:teuthology.orchestra.run.smithi076.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-04-26T23:30:08.310 INFO:teuthology.orchestra.run.smithi076.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 408 kB/s | 31 kB 00:00 2024-04-26T23:30:08.335 INFO:teuthology.orchestra.run.smithi076.stdout:(138/152): python3-pyperclip-1.8.2-2.el9.noarch 240 kB/s | 28 kB 00:00 2024-04-26T23:30:08.361 INFO:teuthology.orchestra.run.smithi076.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 2.0 MB/s | 188 kB 00:00 2024-04-26T23:30:08.402 INFO:teuthology.orchestra.run.smithi076.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 641 kB/s | 59 kB 00:00 2024-04-26T23:30:08.428 INFO:teuthology.orchestra.run.smithi076.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 387 kB/s | 36 kB 00:00 2024-04-26T23:30:08.478 INFO:teuthology.orchestra.run.smithi076.stdout:(142/152): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-04-26T23:30:08.496 INFO:teuthology.orchestra.run.smithi076.stdout:(143/152): python3-typing-extensions-4.4.0-2.el 379 kB/s | 51 kB 00:00 2024-04-26T23:30:08.554 INFO:teuthology.orchestra.run.smithi076.stdout:(144/152): python3-werkzeug-2.0.3-3.el9.1.noarc 5.5 MB/s | 427 kB 00:00 2024-04-26T23:30:08.572 INFO:teuthology.orchestra.run.smithi076.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 265 kB/s | 20 kB 00:00 2024-04-26T23:30:08.597 INFO:teuthology.orchestra.run.smithi076.stdout:(146/152): python3-websocket-client-1.2.3-2.el9 533 kB/s | 90 kB 00:00 2024-04-26T23:30:08.689 INFO:teuthology.orchestra.run.smithi076.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 1.4 MB/s | 192 kB 00:00 2024-04-26T23:30:08.755 INFO:teuthology.orchestra.run.smithi076.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 8.6 MB/s | 1.6 MB 00:00 2024-04-26T23:30:08.948 INFO:teuthology.orchestra.run.smithi076.stdout:(149/152): librados2-19.0.0-3306.gbc612ffb.el9. 9.7 MB/s | 3.4 MB 00:00 2024-04-26T23:30:09.139 INFO:teuthology.orchestra.run.smithi076.stdout:(150/152): librbd1-19.0.0-3306.gbc612ffb.el9.x8 7.0 MB/s | 3.1 MB 00:00 2024-04-26T23:30:10.781 INFO:teuthology.orchestra.run.smithi076.stdout:(151/152): openssl-3.2.1-1.el9.x86_64.rpm 682 kB/s | 1.3 MB 00:02 2024-04-26T23:30:12.156 INFO:teuthology.orchestra.run.smithi076.stdout:(152/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 771 kB/s | 2.4 MB 00:03 2024-04-26T23:30:12.164 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-04-26T23:30:12.164 INFO:teuthology.orchestra.run.smithi076.stdout:Total 5.3 MB/s | 222 MB 00:41 2024-04-26T23:30:13.205 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-04-26T23:30:13.351 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-04-26T23:30:13.351 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-04-26T23:30:15.262 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-04-26T23:30:15.263 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-04-26T23:30:15.478 INFO:teuthology.orchestra.run.smithi072.stdout:(40/152): fuse-2.9.9-15.el9.x86_64.rpm 2.6 kB/s | 80 kB 00:30 2024-04-26T23:30:15.512 INFO:teuthology.orchestra.run.smithi072.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 6.6 kB/s | 195 kB 00:29 2024-04-26T23:30:15.588 INFO:teuthology.orchestra.run.smithi072.stdout:(42/152): libgfortran-11.4.1-3.el9.x86_64.rpm 26 kB/s | 803 kB 00:30 2024-04-26T23:30:15.630 INFO:teuthology.orchestra.run.smithi072.stdout:(43/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.6 MB/s | 253 kB 00:00 2024-04-26T23:30:15.713 INFO:teuthology.orchestra.run.smithi072.stdout:(44/152): python3-cryptography-36.0.1-4.el9.x86 6.1 MB/s | 1.2 MB 00:00 2024-04-26T23:30:15.739 INFO:teuthology.orchestra.run.smithi072.stdout:(45/152): python3-ply-3.11-14.el9.noarch.rpm 709 kB/s | 106 kB 00:00 2024-04-26T23:30:15.774 INFO:teuthology.orchestra.run.smithi072.stdout:(46/152): python3-pycparser-2.20-6.el9.noarch.r 942 kB/s | 135 kB 00:00 2024-04-26T23:30:15.825 INFO:teuthology.orchestra.run.smithi072.stdout:(47/152): python3-requests-2.25.1-8.el9.noarch. 1.1 MB/s | 125 kB 00:00 2024-04-26T23:30:15.859 INFO:teuthology.orchestra.run.smithi072.stdout:(48/152): python3-urllib3-1.26.5-5.el9.noarch.r 1.8 MB/s | 215 kB 00:00 2024-04-26T23:30:15.881 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - CRB 99 kB/s | 6.2 MB 01:03 2024-04-26T23:30:16.684 INFO:teuthology.orchestra.run.smithi072.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 130 kB/s | 107 kB 00:00 2024-04-26T23:30:17.260 INFO:teuthology.orchestra.run.smithi072.stdout:(50/152): automake-1.16.2-8.el9.noarch.rpm 464 kB/s | 689 kB 00:01 2024-04-26T23:30:17.310 INFO:teuthology.orchestra.run.smithi072.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 34 kB/s | 21 kB 00:00 2024-04-26T23:30:17.472 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 16 kB 00:00 2024-04-26T23:30:17.686 INFO:teuthology.orchestra.run.smithi072.stdout:(52/152): bison-3.7.4-5.el9.x86_64.rpm 507 kB/s | 943 kB 00:01 2024-04-26T23:30:17.994 INFO:teuthology.orchestra.run.smithi072.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 48 kB/s | 33 kB 00:00 2024-04-26T23:30:18.345 INFO:teuthology.orchestra.run.smithi072.stdout:(54/152): flex-2.6.4-9.el9.x86_64.rpm 289 kB/s | 313 kB 00:01 2024-04-26T23:30:18.637 INFO:teuthology.orchestra.run.smithi072.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 28 kB/s | 18 kB 00:00 2024-04-26T23:30:18.640 INFO:teuthology.orchestra.run.smithi136.stdout:Extra Packages for Enterprise Linux 20 MB/s | 21 MB 00:01 2024-04-26T23:30:18.921 INFO:teuthology.orchestra.run.smithi072.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 19 kB/s | 11 kB 00:00 2024-04-26T23:30:19.422 INFO:teuthology.orchestra.run.smithi072.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 204 kB/s | 160 kB 00:00 2024-04-26T23:30:19.531 INFO:teuthology.orchestra.run.smithi072.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 74 kB/s | 45 kB 00:00 2024-04-26T23:30:20.706 INFO:teuthology.orchestra.run.smithi072.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 516 kB/s | 662 kB 00:01 2024-04-26T23:30:20.781 INFO:teuthology.orchestra.run.smithi072.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 466 kB/s | 582 kB 00:01 2024-04-26T23:30:21.248 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-04-26T23:30:21.335 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-26T23:30:21.367 INFO:teuthology.orchestra.run.smithi072.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 39 kB/s | 25 kB 00:00 2024-04-26T23:30:21.373 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-26T23:30:21.436 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-26T23:30:21.469 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-26T23:30:21.495 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-26T23:30:21.500 INFO:teuthology.orchestra.run.smithi072.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 66 kB/s | 47 kB 00:00 2024-04-26T23:30:21.538 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-26T23:30:21.818 INFO:teuthology.orchestra.run.smithi072.stdout:(63/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 742 kB/s | 3.0 MB 00:04 2024-04-26T23:30:21.867 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-26T23:30:21.997 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T23:30:22.118 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T23:30:22.148 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T23:30:22.239 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T23:30:22.269 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-26T23:30:22.417 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-26T23:30:22.467 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-26T23:30:22.468 INFO:teuthology.orchestra.run.smithi072.stdout:(64/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 266 kB/s | 292 kB 00:01 2024-04-26T23:30:22.495 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T23:30:22.502 INFO:teuthology.orchestra.run.smithi072.stdout:(65/152): lua-5.4.4-4.el9.x86_64.rpm 188 kB/s | 188 kB 00:01 2024-04-26T23:30:22.528 INFO:teuthology.orchestra.run.smithi072.stdout:(66/152): openblas-0.3.21-2.el9.x86_64.rpm 55 kB/s | 35 kB 00:00 2024-04-26T23:30:22.598 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T23:30:22.621 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-26T23:30:22.684 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-26T23:30:22.705 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T23:30:22.726 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T23:30:22.931 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T23:30:22.964 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T23:30:23.071 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-26T23:30:23.158 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 18/156 2024-04-26T23:30:23.196 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-26T23:30:23.240 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-26T23:30:23.303 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-26T23:30:23.339 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-26T23:30:23.382 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-26T23:30:23.423 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-26T23:30:23.451 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-26T23:30:23.520 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-26T23:30:23.573 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-26T23:30:23.604 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-26T23:30:23.640 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-26T23:30:23.716 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-26T23:30:23.747 INFO:teuthology.orchestra.run.smithi136.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-04-26T23:30:23.760 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 31/156 2024-04-26T23:30:23.797 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 32/156 2024-04-26T23:30:23.824 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 33/156 2024-04-26T23:30:23.895 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-26T23:30:23.938 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-26T23:30:24.012 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-26T23:30:24.145 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-26T23:30:24.199 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-26T23:30:24.715 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-26T23:30:24.811 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-26T23:30:24.849 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-26T23:30:24.892 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-26T23:30:24.938 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 43/156 2024-04-26T23:30:24.981 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-26T23:30:25.003 INFO:teuthology.orchestra.run.smithi072.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 657 kB/s | 1.6 MB 00:02 2024-04-26T23:30:25.150 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-26T23:30:25.208 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-26T23:30:25.264 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-26T23:30:25.336 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-26T23:30:25.376 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-26T23:30:25.413 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-26T23:30:25.460 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-26T23:30:25.496 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-26T23:30:25.539 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-26T23:30:25.581 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-26T23:30:25.716 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-26T23:30:25.762 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-26T23:30:25.792 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-26T23:30:25.834 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-26T23:30:25.841 INFO:teuthology.orchestra.run.smithi136.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T23:30:25.842 INFO:teuthology.orchestra.run.smithi136.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T23:30:25.853 INFO:teuthology.orchestra.run.smithi136.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-26T23:30:25.853 INFO:teuthology.orchestra.run.smithi136.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-26T23:30:25.854 INFO:teuthology.orchestra.run.smithi136.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-26T23:30:25.855 INFO:teuthology.orchestra.run.smithi136.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-26T23:30:25.856 INFO:teuthology.orchestra.run.smithi136.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-26T23:30:25.859 INFO:teuthology.orchestra.run.smithi136.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-26T23:30:25.860 INFO:teuthology.orchestra.run.smithi136.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-26T23:30:25.920 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-04-26T23:30:25.929 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-26T23:30:25.936 INFO:teuthology.orchestra.run.smithi136.stdout:======================================================================================= 2024-04-26T23:30:25.936 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-04-26T23:30:25.936 INFO:teuthology.orchestra.run.smithi136.stdout:======================================================================================= 2024-04-26T23:30:25.936 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-04-26T23:30:25.936 INFO:teuthology.orchestra.run.smithi136.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-26T23:30:25.936 INFO:teuthology.orchestra.run.smithi136.stdout: ceph x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 6.6 k 2024-04-26T23:30:25.936 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-base x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-26T23:30:25.937 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 889 k 2024-04-26T23:30:25.937 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 145 k 2024-04-26T23:30:25.937 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 1.5 M 2024-04-26T23:30:25.937 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 142 k 2024-04-26T23:30:25.937 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 3.5 M 2024-04-26T23:30:25.937 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 7.4 M 2024-04-26T23:30:25.937 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-rook noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 49 k 2024-04-26T23:30:25.937 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-radosgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 11 M 2024-04-26T23:30:25.937 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-test x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 49 M 2024-04-26T23:30:25.937 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-volume noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 273 k 2024-04-26T23:30:25.938 INFO:teuthology.orchestra.run.smithi136.stdout: cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 762 k 2024-04-26T23:30:25.938 INFO:teuthology.orchestra.run.smithi136.stdout: cephfs-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 225 k 2024-04-26T23:30:25.938 INFO:teuthology.orchestra.run.smithi136.stdout: cephfs-shell x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 27 k 2024-04-26T23:30:25.938 INFO:teuthology.orchestra.run.smithi136.stdout: cephfs-top noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 18 k 2024-04-26T23:30:25.938 INFO:teuthology.orchestra.run.smithi136.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-26T23:30:25.938 INFO:teuthology.orchestra.run.smithi136.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-26T23:30:25.938 INFO:teuthology.orchestra.run.smithi136.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-26T23:30:25.938 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 32 k 2024-04-26T23:30:25.938 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 747 k 2024-04-26T23:30:25.938 INFO:teuthology.orchestra.run.smithi136.stdout: librados-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 127 k 2024-04-26T23:30:25.938 INFO:teuthology.orchestra.run.smithi136.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-26T23:30:25.938 INFO:teuthology.orchestra.run.smithi136.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-26T23:30:25.938 INFO:teuthology.orchestra.run.smithi136.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-26T23:30:25.939 INFO:teuthology.orchestra.run.smithi136.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-26T23:30:25.939 INFO:teuthology.orchestra.run.smithi136.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-26T23:30:25.939 INFO:teuthology.orchestra.run.smithi136.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-26T23:30:25.939 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cephfs x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 161 k 2024-04-26T23:30:25.939 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rados x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 321 k 2024-04-26T23:30:25.939 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 298 k 2024-04-26T23:30:25.939 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 99 k 2024-04-26T23:30:25.939 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 85 k 2024-04-26T23:30:25.940 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-26T23:30:25.940 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-nbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 171 k 2024-04-26T23:30:25.940 INFO:teuthology.orchestra.run.smithi136.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-26T23:30:25.940 INFO:teuthology.orchestra.run.smithi136.stdout:Upgrading: 2024-04-26T23:30:25.940 INFO:teuthology.orchestra.run.smithi136.stdout: librados2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.4 M 2024-04-26T23:30:25.940 INFO:teuthology.orchestra.run.smithi136.stdout: librbd1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-26T23:30:25.940 INFO:teuthology.orchestra.run.smithi136.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-26T23:30:25.940 INFO:teuthology.orchestra.run.smithi136.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-26T23:30:25.940 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-04-26T23:30:25.941 INFO:teuthology.orchestra.run.smithi136.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-26T23:30:25.941 INFO:teuthology.orchestra.run.smithi136.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-26T23:30:25.941 INFO:teuthology.orchestra.run.smithi136.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-26T23:30:25.941 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 21 M 2024-04-26T23:30:25.941 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 34 k 2024-04-26T23:30:25.941 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mds x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 2.4 M 2024-04-26T23:30:25.941 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 249 k 2024-04-26T23:30:25.941 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mon x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 4.7 M 2024-04-26T23:30:25.941 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-osd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 17 M 2024-04-26T23:30:25.942 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 16 k 2024-04-26T23:30:25.942 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-selinux x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 25 k 2024-04-26T23:30:25.942 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-26T23:30:25.942 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-26T23:30:25.942 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-26T23:30:25.942 INFO:teuthology.orchestra.run.smithi136.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-26T23:30:25.942 INFO:teuthology.orchestra.run.smithi136.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-26T23:30:25.942 INFO:teuthology.orchestra.run.smithi136.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-26T23:30:25.942 INFO:teuthology.orchestra.run.smithi136.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-26T23:30:25.942 INFO:teuthology.orchestra.run.smithi136.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-26T23:30:25.942 INFO:teuthology.orchestra.run.smithi136.stdout: libcephsqlite x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 163 k 2024-04-26T23:30:25.943 INFO:teuthology.orchestra.run.smithi136.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-26T23:30:25.943 INFO:teuthology.orchestra.run.smithi136.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-26T23:30:25.943 INFO:teuthology.orchestra.run.smithi136.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-26T23:30:25.943 INFO:teuthology.orchestra.run.smithi136.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-26T23:30:25.943 INFO:teuthology.orchestra.run.smithi136.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-26T23:30:25.944 INFO:teuthology.orchestra.run.smithi136.stdout: libradosstriper1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 501 k 2024-04-26T23:30:25.944 INFO:teuthology.orchestra.run.smithi136.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-26T23:30:25.944 INFO:teuthology.orchestra.run.smithi136.stdout: librgw2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-26T23:30:25.944 INFO:teuthology.orchestra.run.smithi136.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-26T23:30:25.944 INFO:teuthology.orchestra.run.smithi136.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-26T23:30:25.944 INFO:teuthology.orchestra.run.smithi136.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-26T23:30:25.944 INFO:teuthology.orchestra.run.smithi136.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-26T23:30:25.944 INFO:teuthology.orchestra.run.smithi136.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-26T23:30:25.944 INFO:teuthology.orchestra.run.smithi136.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-26T23:30:25.944 INFO:teuthology.orchestra.run.smithi136.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-26T23:30:25.944 INFO:teuthology.orchestra.run.smithi136.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-26T23:30:25.944 INFO:teuthology.orchestra.run.smithi136.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-26T23:30:25.944 INFO:teuthology.orchestra.run.smithi136.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-26T23:30:25.944 INFO:teuthology.orchestra.run.smithi136.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-26T23:30:25.945 INFO:teuthology.orchestra.run.smithi136.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-26T23:30:25.945 INFO:teuthology.orchestra.run.smithi136.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-26T23:30:25.945 INFO:teuthology.orchestra.run.smithi136.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-26T23:30:25.945 INFO:teuthology.orchestra.run.smithi136.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-26T23:30:25.945 INFO:teuthology.orchestra.run.smithi136.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-26T23:30:25.945 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-26T23:30:25.945 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-argparse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 45 k 2024-04-26T23:30:25.945 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 135 k 2024-04-26T23:30:25.945 INFO:teuthology.orchestra.run.smithi136.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-26T23:30:25.945 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-26T23:30:25.945 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-26T23:30:25.945 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-26T23:30:25.945 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-26T23:30:25.945 INFO:teuthology.orchestra.run.smithi136.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-26T23:30:25.946 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-26T23:30:25.946 INFO:teuthology.orchestra.run.smithi136.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-26T23:30:25.946 INFO:teuthology.orchestra.run.smithi136.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-26T23:30:25.946 INFO:teuthology.orchestra.run.smithi136.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-26T23:30:25.946 INFO:teuthology.orchestra.run.smithi136.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-26T23:30:25.946 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-26T23:30:25.946 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-26T23:30:25.946 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-26T23:30:25.946 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-26T23:30:25.946 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-26T23:30:25.946 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-26T23:30:25.946 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-26T23:30:25.946 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-26T23:30:25.946 INFO:teuthology.orchestra.run.smithi136.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-26T23:30:25.947 INFO:teuthology.orchestra.run.smithi136.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-26T23:30:25.947 INFO:teuthology.orchestra.run.smithi136.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-26T23:30:25.947 INFO:teuthology.orchestra.run.smithi136.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-26T23:30:25.947 INFO:teuthology.orchestra.run.smithi136.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-26T23:30:25.947 INFO:teuthology.orchestra.run.smithi136.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-26T23:30:25.947 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-26T23:30:25.947 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-26T23:30:25.947 INFO:teuthology.orchestra.run.smithi136.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-26T23:30:25.947 INFO:teuthology.orchestra.run.smithi136.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-26T23:30:25.947 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-26T23:30:25.947 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-26T23:30:25.947 INFO:teuthology.orchestra.run.smithi136.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-26T23:30:25.947 INFO:teuthology.orchestra.run.smithi136.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-26T23:30:25.947 INFO:teuthology.orchestra.run.smithi136.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-26T23:30:25.948 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-26T23:30:25.948 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-26T23:30:25.948 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-26T23:30:25.948 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-26T23:30:25.948 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-26T23:30:25.948 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-26T23:30:25.948 INFO:teuthology.orchestra.run.smithi136.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-26T23:30:25.948 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-26T23:30:25.948 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-26T23:30:25.948 INFO:teuthology.orchestra.run.smithi136.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-26T23:30:25.948 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-26T23:30:25.948 INFO:teuthology.orchestra.run.smithi136.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-26T23:30:25.948 INFO:teuthology.orchestra.run.smithi136.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-26T23:30:25.948 INFO:teuthology.orchestra.run.smithi136.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-26T23:30:25.948 INFO:teuthology.orchestra.run.smithi136.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-26T23:30:25.949 INFO:teuthology.orchestra.run.smithi136.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-26T23:30:25.949 INFO:teuthology.orchestra.run.smithi136.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-26T23:30:25.949 INFO:teuthology.orchestra.run.smithi136.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-26T23:30:25.949 INFO:teuthology.orchestra.run.smithi136.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-26T23:30:25.949 INFO:teuthology.orchestra.run.smithi136.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-26T23:30:25.949 INFO:teuthology.orchestra.run.smithi136.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-26T23:30:25.949 INFO:teuthology.orchestra.run.smithi136.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-26T23:30:25.949 INFO:teuthology.orchestra.run.smithi136.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-26T23:30:25.949 INFO:teuthology.orchestra.run.smithi136.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-26T23:30:25.949 INFO:teuthology.orchestra.run.smithi136.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-26T23:30:25.949 INFO:teuthology.orchestra.run.smithi136.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-26T23:30:25.949 INFO:teuthology.orchestra.run.smithi136.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-26T23:30:25.949 INFO:teuthology.orchestra.run.smithi136.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-26T23:30:25.949 INFO:teuthology.orchestra.run.smithi136.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-26T23:30:25.950 INFO:teuthology.orchestra.run.smithi136.stdout:Installing weak dependencies: 2024-04-26T23:30:25.950 INFO:teuthology.orchestra.run.smithi136.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-26T23:30:25.950 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-26T23:30:25.950 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-04-26T23:30:25.950 INFO:teuthology.orchestra.run.smithi136.stdout:======================================================================================= 2024-04-26T23:30:25.950 INFO:teuthology.orchestra.run.smithi136.stdout:Install 148 Packages 2024-04-26T23:30:25.950 INFO:teuthology.orchestra.run.smithi136.stdout:Upgrade 4 Packages 2024-04-26T23:30:25.950 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-26T23:30:25.951 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 222 M 2024-04-26T23:30:25.951 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-04-26T23:30:25.959 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-26T23:30:25.999 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-26T23:30:26.025 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-26T23:30:26.050 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 63/156 2024-04-26T23:30:26.125 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-26T23:30:26.253 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-26T23:30:26.292 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-26T23:30:26.320 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-26T23:30:26.354 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-26T23:30:26.393 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-26T23:30:26.425 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-26T23:30:26.455 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-26T23:30:26.489 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-26T23:30:26.523 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-26T23:30:26.684 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-26T23:30:26.703 INFO:teuthology.orchestra.run.smithi072.stdout:(68/152): postgresql-contrib-13.14-1.el9.x86_64 536 kB/s | 911 kB 00:01 2024-04-26T23:30:26.891 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-26T23:30:26.942 INFO:teuthology.orchestra.run.smithi136.stdout:(1/152): ceph-19.0.0-3306.gbc612ffb.el9.x86_64. 37 kB/s | 6.6 kB 00:00 2024-04-26T23:30:26.996 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-26T23:30:27.047 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-26T23:30:27.078 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-26T23:30:27.125 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-26T23:30:27.200 INFO:teuthology.orchestra.run.smithi136.stdout:(2/152): ceph-fuse-19.0.0-3306.gbc612ffb.el9.x8 3.4 MB/s | 889 kB 00:00 2024-04-26T23:30:27.223 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-26T23:30:27.250 INFO:teuthology.orchestra.run.smithi136.stdout:(3/152): ceph-immutable-object-cache-19.0.0-330 2.8 MB/s | 145 kB 00:00 2024-04-26T23:30:27.280 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-26T23:30:27.437 INFO:teuthology.orchestra.run.smithi072.stdout:(69/152): postgresql-private-libs-13.14-1.el9.x 182 kB/s | 133 kB 00:00 2024-04-26T23:30:27.809 INFO:teuthology.orchestra.run.smithi136.stdout:(4/152): ceph-mds-19.0.0-3306.gbc612ffb.el9.x86 4.2 MB/s | 2.4 MB 00:00 2024-04-26T23:30:27.997 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-26T23:30:28.052 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-26T23:30:28.090 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-26T23:30:28.185 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-26T23:30:28.192 INFO:teuthology.orchestra.run.smithi136.stdout:(5/152): ceph-mgr-19.0.0-3306.gbc612ffb.el9.x86 3.9 MB/s | 1.5 MB 00:00 2024-04-26T23:30:28.244 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-26T23:30:28.332 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-26T23:30:28.364 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-26T23:30:28.540 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-26T23:30:28.570 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-26T23:30:28.647 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-26T23:30:28.738 INFO:teuthology.orchestra.run.smithi072.stdout:(70/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 727 kB/s | 4.4 MB 00:06 2024-04-26T23:30:29.055 INFO:teuthology.orchestra.run.smithi072.stdout:(71/152): openblas-openmp-0.3.21-2.el9.x86_64.r 737 kB/s | 4.7 MB 00:06 2024-04-26T23:30:29.282 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-26T23:30:29.394 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-26T23:30:29.592 INFO:teuthology.orchestra.run.smithi136.stdout:(6/152): ceph-base-19.0.0-3306.gbc612ffb.el9.x8 1.9 MB/s | 5.4 MB 00:02 2024-04-26T23:30:29.651 INFO:teuthology.orchestra.run.smithi136.stdout:(7/152): ceph-mon-19.0.0-3306.gbc612ffb.el9.x86 3.2 MB/s | 4.7 MB 00:01 2024-04-26T23:30:29.789 INFO:teuthology.orchestra.run.smithi072.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 132 kB/s | 87 kB 00:00 2024-04-26T23:30:30.259 INFO:teuthology.orchestra.run.smithi136.stdout:(8/152): ceph-common-19.0.0-3306.gbc612ffb.el9. 6.0 MB/s | 21 MB 00:03 2024-04-26T23:30:30.318 INFO:teuthology.orchestra.run.smithi136.stdout:(9/152): ceph-selinux-19.0.0-3306.gbc612ffb.el9 434 kB/s | 25 kB 00:00 2024-04-26T23:30:30.616 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-26T23:30:30.702 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-26T23:30:30.731 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-26T23:30:30.731 INFO:teuthology.orchestra.run.smithi072.stdout:(73/152): protobuf-3.14.0-13.el9.x86_64.rpm 519 kB/s | 1.0 MB 00:01 2024-04-26T23:30:30.764 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-26T23:30:31.076 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-26T23:30:31.118 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-26T23:30:31.209 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-26T23:30:31.240 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 100/156 2024-04-26T23:30:31.276 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-26T23:30:31.501 INFO:teuthology.orchestra.run.smithi136.stdout:(10/152): ceph-radosgw-19.0.0-3306.gbc612ffb.el 5.7 MB/s | 11 MB 00:01 2024-04-26T23:30:31.602 INFO:teuthology.orchestra.run.smithi136.stdout:(11/152): cephfs-mirror-19.0.0-3306.gbc612ffb.e 2.2 MB/s | 225 kB 00:00 2024-04-26T23:30:31.615 INFO:teuthology.orchestra.run.smithi072.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 277 kB/s | 244 kB 00:00 2024-04-26T23:30:31.694 INFO:teuthology.orchestra.run.smithi136.stdout:(12/152): cephfs-shell-19.0.0-3306.gbc612ffb.el 295 kB/s | 27 kB 00:00 2024-04-26T23:30:31.755 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 102/156 2024-04-26T23:30:31.780 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-26T23:30:31.786 INFO:teuthology.orchestra.run.smithi136.stdout:(13/152): libcephfs-devel-19.0.0-3306.gbc612ffb 346 kB/s | 32 kB 00:00 2024-04-26T23:30:31.852 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-26T23:30:31.882 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 104/156 2024-04-26T23:30:31.970 INFO:teuthology.orchestra.run.smithi136.stdout:(14/152): libcephfs2-19.0.0-3306.gbc612ffb.el9. 4.0 MB/s | 747 kB 00:00 2024-04-26T23:30:32.045 INFO:teuthology.orchestra.run.smithi136.stdout:(15/152): libcephsqlite-19.0.0-3306.gbc612ffb.e 2.1 MB/s | 163 kB 00:00 2024-04-26T23:30:32.129 INFO:teuthology.orchestra.run.smithi136.stdout:(16/152): librados-devel-19.0.0-3306.gbc612ffb. 1.5 MB/s | 127 kB 00:00 2024-04-26T23:30:32.279 INFO:teuthology.orchestra.run.smithi136.stdout:(17/152): libradosstriper1-19.0.0-3306.gbc612ff 3.3 MB/s | 501 kB 00:00 2024-04-26T23:30:32.590 INFO:teuthology.orchestra.run.smithi072.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 255 kB/s | 248 kB 00:00 2024-04-26T23:30:33.279 INFO:teuthology.orchestra.run.smithi136.stdout:(18/152): librgw2-19.0.0-3306.gbc612ffb.el9.x86 5.4 MB/s | 5.4 MB 00:00 2024-04-26T23:30:33.379 INFO:teuthology.orchestra.run.smithi136.stdout:(19/152): python3-ceph-argparse-19.0.0-3306.gbc 451 kB/s | 45 kB 00:00 2024-04-26T23:30:33.471 INFO:teuthology.orchestra.run.smithi136.stdout:(20/152): python3-ceph-common-19.0.0-3306.gbc61 1.4 MB/s | 135 kB 00:00 2024-04-26T23:30:33.572 INFO:teuthology.orchestra.run.smithi136.stdout:(21/152): python3-cephfs-19.0.0-3306.gbc612ffb. 1.6 MB/s | 161 kB 00:00 2024-04-26T23:30:33.664 INFO:teuthology.orchestra.run.smithi136.stdout:(22/152): python3-rados-19.0.0-3306.gbc612ffb.e 3.4 MB/s | 321 kB 00:00 2024-04-26T23:30:33.755 INFO:teuthology.orchestra.run.smithi136.stdout:(23/152): python3-rbd-19.0.0-3306.gbc612ffb.el9 3.2 MB/s | 298 kB 00:00 2024-04-26T23:30:33.847 INFO:teuthology.orchestra.run.smithi136.stdout:(24/152): python3-rgw-19.0.0-3306.gbc612ffb.el9 1.1 MB/s | 99 kB 00:00 2024-04-26T23:30:33.940 INFO:teuthology.orchestra.run.smithi136.stdout:(25/152): rbd-fuse-19.0.0-3306.gbc612ffb.el9.x8 930 kB/s | 85 kB 00:00 2024-04-26T23:30:34.236 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T23:30:34.273 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T23:30:34.640 INFO:teuthology.orchestra.run.smithi136.stdout:(26/152): rbd-mirror-19.0.0-3306.gbc612ffb.el9. 4.4 MB/s | 3.1 MB 00:00 2024-04-26T23:30:34.698 INFO:teuthology.orchestra.run.smithi136.stdout:(27/152): rbd-nbd-19.0.0-3306.gbc612ffb.el9.x86 2.9 MB/s | 171 kB 00:00 2024-04-26T23:30:34.757 INFO:teuthology.orchestra.run.smithi136.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3306.g 585 kB/s | 34 kB 00:00 2024-04-26T23:30:34.775 INFO:teuthology.orchestra.run.smithi072.stdout:(76/152): postgresql-server-13.14-1.el9.x86_64. 807 kB/s | 5.8 MB 00:07 2024-04-26T23:30:34.814 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T23:30:34.824 INFO:teuthology.orchestra.run.smithi136.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3306.gbc612ff 2.1 MB/s | 142 kB 00:00 2024-04-26T23:30:34.852 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-26T23:30:34.944 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-26T23:30:34.944 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-26T23:30:34.944 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-26T23:30:34.944 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-26T23:30:34.978 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-26T23:30:35.292 INFO:teuthology.orchestra.run.smithi072.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 9.5 kB/s | 26 kB 00:02 2024-04-26T23:30:35.326 INFO:teuthology.orchestra.run.smithi072.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 35 kB/s | 19 kB 00:00 2024-04-26T23:30:35.574 INFO:teuthology.orchestra.run.smithi136.stdout:(30/152): ceph-mgr-dashboard-19.0.0-3306.gbc612 4.7 MB/s | 3.5 MB 00:00 2024-04-26T23:30:35.943 INFO:teuthology.orchestra.run.smithi072.stdout:(79/152): python3-markupsafe-1.1.1-12.el9.x86_6 56 kB/s | 35 kB 00:00 2024-04-26T23:30:36.094 INFO:teuthology.orchestra.run.smithi072.stdout:(80/152): python3-mako-1.1.4-6.el9.noarch.rpm 215 kB/s | 172 kB 00:00 2024-04-26T23:30:37.053 INFO:teuthology.orchestra.run.smithi072.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 262 kB/s | 250 kB 00:00 2024-04-26T23:30:37.191 INFO:teuthology.orchestra.run.smithi136.stdout:(31/152): ceph-mgr-diskprediction-local-19.0.0- 4.6 MB/s | 7.4 MB 00:01 2024-04-26T23:30:37.287 INFO:teuthology.orchestra.run.smithi072.stdout:(82/152): python3-babel-2.9.1-2.el9.noarch.rpm 815 kB/s | 6.0 MB 00:07 2024-04-26T23:30:37.308 INFO:teuthology.orchestra.run.smithi136.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3306.gbc 2.1 MB/s | 249 kB 00:00 2024-04-26T23:30:37.358 INFO:teuthology.orchestra.run.smithi136.stdout:(33/152): ceph-mgr-rook-19.0.0-3306.gbc612ffb.e 992 kB/s | 49 kB 00:00 2024-04-26T23:30:37.408 INFO:teuthology.orchestra.run.smithi136.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3306.gb 329 kB/s | 16 kB 00:00 2024-04-26T23:30:37.533 INFO:teuthology.orchestra.run.smithi136.stdout:(35/152): ceph-volume-19.0.0-3306.gbc612ffb.el9 2.1 MB/s | 273 kB 00:00 2024-04-26T23:30:37.726 INFO:teuthology.orchestra.run.smithi136.stdout:(36/152): cephadm-19.0.0-3306.gbc612ffb.el9.noa 3.9 MB/s | 762 kB 00:00 2024-04-26T23:30:37.801 INFO:teuthology.orchestra.run.smithi136.stdout:(37/152): cephfs-top-19.0.0-3306.gbc612ffb.el9. 243 kB/s | 18 kB 00:00 2024-04-26T23:30:37.951 INFO:teuthology.orchestra.run.smithi136.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 536 kB/s | 80 kB 00:00 2024-04-26T23:30:38.060 INFO:teuthology.orchestra.run.smithi136.stdout:(39/152): libgfortran-11.4.1-3.el9.x86_64.rpm 7.3 MB/s | 803 kB 00:00 2024-04-26T23:30:38.062 INFO:teuthology.orchestra.run.smithi072.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 220 kB/s | 222 kB 00:01 2024-04-26T23:30:38.096 INFO:teuthology.orchestra.run.smithi072.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 96 kB/s | 77 kB 00:00 2024-04-26T23:30:38.110 INFO:teuthology.orchestra.run.smithi136.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 3.8 MB/s | 195 kB 00:00 2024-04-26T23:30:38.161 INFO:teuthology.orchestra.run.smithi136.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.0 MB/s | 253 kB 00:00 2024-04-26T23:30:38.244 INFO:teuthology.orchestra.run.smithi136.stdout:(42/152): python3-cryptography-36.0.1-4.el9.x86 15 MB/s | 1.2 MB 00:00 2024-04-26T23:30:38.294 INFO:teuthology.orchestra.run.smithi136.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-04-26T23:30:38.345 INFO:teuthology.orchestra.run.smithi136.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-04-26T23:30:38.395 INFO:teuthology.orchestra.run.smithi136.stdout:(45/152): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-04-26T23:30:38.437 INFO:teuthology.orchestra.run.smithi136.stdout:(46/152): python3-urllib3-1.26.5-5.el9.noarch.r 5.1 MB/s | 215 kB 00:00 2024-04-26T23:30:38.529 INFO:teuthology.orchestra.run.smithi136.stdout:(47/152): automake-1.16.2-8.el9.noarch.rpm 7.4 MB/s | 689 kB 00:00 2024-04-26T23:30:38.629 INFO:teuthology.orchestra.run.smithi136.stdout:(48/152): bison-3.7.4-5.el9.x86_64.rpm 9.2 MB/s | 943 kB 00:00 2024-04-26T23:30:38.696 INFO:teuthology.orchestra.run.smithi136.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 107 kB 00:00 2024-04-26T23:30:38.721 INFO:teuthology.orchestra.run.smithi072.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 64 kB/s | 42 kB 00:00 2024-04-26T23:30:38.746 INFO:teuthology.orchestra.run.smithi136.stdout:(50/152): elfutils-libelf-devel-0.190-2.el9.x86 431 kB/s | 21 kB 00:00 2024-04-26T23:30:38.822 INFO:teuthology.orchestra.run.smithi136.stdout:(51/152): flex-2.6.4-9.el9.x86_64.rpm 4.1 MB/s | 313 kB 00:00 2024-04-26T23:30:38.872 INFO:teuthology.orchestra.run.smithi136.stdout:(52/152): flexiblas-3.0.4-8.el9.x86_64.rpm 653 kB/s | 33 kB 00:00 2024-04-26T23:30:39.031 INFO:teuthology.orchestra.run.smithi136.stdout:(53/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 19 MB/s | 3.0 MB 00:00 2024-04-26T23:30:39.081 INFO:teuthology.orchestra.run.smithi136.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 366 kB/s | 18 kB 00:00 2024-04-26T23:30:39.123 INFO:teuthology.orchestra.run.smithi136.stdout:(55/152): libaio-devel-0.3.111-13.el9.x86_64.rp 266 kB/s | 11 kB 00:00 2024-04-26T23:30:39.190 INFO:teuthology.orchestra.run.smithi136.stdout:(56/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.3 MB/s | 160 kB 00:00 2024-04-26T23:30:39.240 INFO:teuthology.orchestra.run.smithi136.stdout:(57/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 908 kB/s | 45 kB 00:00 2024-04-26T23:30:39.255 INFO:teuthology.orchestra.run.smithi072.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 232 kB/s | 269 kB 00:01 2024-04-26T23:30:39.340 INFO:teuthology.orchestra.run.smithi136.stdout:(58/152): librdkafka-1.6.1-102.el9.x86_64.rpm 6.5 MB/s | 662 kB 00:00 2024-04-26T23:30:39.516 INFO:teuthology.orchestra.run.smithi136.stdout:(59/152): ceph-osd-19.0.0-3306.gbc612ffb.el9.x8 1.7 MB/s | 17 MB 00:09 2024-04-26T23:30:39.558 INFO:teuthology.orchestra.run.smithi136.stdout:(60/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 617 kB/s | 25 kB 00:00 2024-04-26T23:30:39.600 INFO:teuthology.orchestra.run.smithi136.stdout:(61/152): libtool-2.4.6-45.el9.x86_64.rpm 2.2 MB/s | 582 kB 00:00 2024-04-26T23:30:39.616 INFO:teuthology.orchestra.run.smithi136.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 810 kB/s | 47 kB 00:00 2024-04-26T23:30:39.648 INFO:teuthology.orchestra.run.smithi072.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 172 kB/s | 159 kB 00:00 2024-04-26T23:30:39.683 INFO:teuthology.orchestra.run.smithi136.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.4 MB/s | 292 kB 00:00 2024-04-26T23:30:39.717 INFO:teuthology.orchestra.run.smithi136.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 1.8 MB/s | 188 kB 00:00 2024-04-26T23:30:39.751 INFO:teuthology.orchestra.run.smithi136.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 524 kB/s | 35 kB 00:00 2024-04-26T23:30:39.935 INFO:teuthology.orchestra.run.smithi136.stdout:(66/152): openblas-openmp-0.3.21-2.el9.x86_64.r 22 MB/s | 4.7 MB 00:00 2024-04-26T23:30:40.010 INFO:teuthology.orchestra.run.smithi136.stdout:(67/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 17 MB/s | 4.4 MB 00:00 2024-04-26T23:30:40.118 INFO:teuthology.orchestra.run.smithi136.stdout:(68/152): postgresql-contrib-13.14-1.el9.x86_64 8.2 MB/s | 911 kB 00:00 2024-04-26T23:30:40.156 INFO:teuthology.orchestra.run.smithi072.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 310 kB/s | 279 kB 00:00 2024-04-26T23:30:40.160 INFO:teuthology.orchestra.run.smithi136.stdout:(69/152): postgresql-13.14-1.el9.x86_64.rpm 7.0 MB/s | 1.6 MB 00:00 2024-04-26T23:30:40.219 INFO:teuthology.orchestra.run.smithi136.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 1.3 MB/s | 133 kB 00:00 2024-04-26T23:30:40.274 INFO:teuthology.orchestra.run.smithi072.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 82 kB/s | 51 kB 00:00 2024-04-26T23:30:40.328 INFO:teuthology.orchestra.run.smithi136.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 9.3 MB/s | 1.0 MB 00:00 2024-04-26T23:30:40.436 INFO:teuthology.orchestra.run.smithi136.stdout:(72/152): postgresql-server-13.14-1.el9.x86_64. 21 MB/s | 5.8 MB 00:00 2024-04-26T23:30:40.462 INFO:teuthology.orchestra.run.smithi136.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 655 kB/s | 87 kB 00:00 2024-04-26T23:30:40.851 INFO:teuthology.orchestra.run.smithi072.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 77 kB/s | 54 kB 00:00 2024-04-26T23:30:40.870 INFO:teuthology.orchestra.run.smithi136.stdout:(74/152): ceph-test-19.0.0-3306.gbc612ffb.el9.x 4.7 MB/s | 49 MB 00:10 2024-04-26T23:30:40.937 INFO:teuthology.orchestra.run.smithi136.stdout:(75/152): python3-devel-3.9.18-3.el9.x86_64.rpm 514 kB/s | 244 kB 00:00 2024-04-26T23:30:40.962 INFO:teuthology.orchestra.run.smithi136.stdout:(76/152): python3-jinja2-2.11.3-5.el9.noarch.rp 2.6 MB/s | 248 kB 00:00 2024-04-26T23:30:41.004 INFO:teuthology.orchestra.run.smithi136.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 383 kB/s | 26 kB 00:00 2024-04-26T23:30:41.030 INFO:teuthology.orchestra.run.smithi136.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 287 kB/s | 19 kB 00:00 2024-04-26T23:30:41.080 INFO:teuthology.orchestra.run.smithi136.stdout:(79/152): python3-markupsafe-1.1.1-12.el9.x86_6 695 kB/s | 35 kB 00:00 2024-04-26T23:30:41.105 INFO:teuthology.orchestra.run.smithi136.stdout:(80/152): python3-mako-1.1.4-6.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-04-26T23:30:41.189 INFO:teuthology.orchestra.run.smithi136.stdout:(81/152): python3-babel-2.9.1-2.el9.noarch.rpm 7.9 MB/s | 6.0 MB 00:00 2024-04-26T23:30:41.256 INFO:teuthology.orchestra.run.smithi136.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.6 MB/s | 250 kB 00:00 2024-04-26T23:30:41.290 INFO:teuthology.orchestra.run.smithi136.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 2.2 MB/s | 222 kB 00:00 2024-04-26T23:30:41.348 INFO:teuthology.orchestra.run.smithi136.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 841 kB/s | 77 kB 00:00 2024-04-26T23:30:41.374 INFO:teuthology.orchestra.run.smithi136.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 506 kB/s | 42 kB 00:00 2024-04-26T23:30:41.457 INFO:teuthology.orchestra.run.smithi136.stdout:(86/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 14 MB/s | 5.1 MB 00:00 2024-04-26T23:30:41.459 INFO:teuthology.orchestra.run.smithi072.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 69 kB/s | 42 kB 00:00 2024-04-26T23:30:41.507 INFO:teuthology.orchestra.run.smithi136.stdout:(87/152): python3-protobuf-3.14.0-13.el9.noarch 1.7 MB/s | 269 kB 00:00 2024-04-26T23:30:41.533 INFO:teuthology.orchestra.run.smithi136.stdout:(88/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.0 MB/s | 159 kB 00:00 2024-04-26T23:30:41.558 INFO:teuthology.orchestra.run.smithi136.stdout:(89/152): python3-pyasn1-modules-0.4.8-6.el9.no 2.7 MB/s | 279 kB 00:00 2024-04-26T23:30:41.583 INFO:teuthology.orchestra.run.smithi136.stdout:(90/152): python3-pytz-2021.1-5.el9.noarch.rpm 675 kB/s | 51 kB 00:00 2024-04-26T23:30:41.617 INFO:teuthology.orchestra.run.smithi136.stdout:(91/152): python3-requests-oauthlib-1.3.0-12.el 637 kB/s | 54 kB 00:00 2024-04-26T23:30:41.659 INFO:teuthology.orchestra.run.smithi136.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 552 kB/s | 42 kB 00:00 2024-04-26T23:30:41.684 INFO:teuthology.orchestra.run.smithi136.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 662 kB/s | 44 kB 00:00 2024-04-26T23:30:41.735 INFO:teuthology.orchestra.run.smithi136.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 717 kB/s | 36 kB 00:00 2024-04-26T23:30:41.760 INFO:teuthology.orchestra.run.smithi136.stdout:(95/152): qatlib-24.02.0-1.el9.x86_64.rpm 2.1 MB/s | 221 kB 00:00 2024-04-26T23:30:41.802 INFO:teuthology.orchestra.run.smithi136.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 707 kB/s | 47 kB 00:00 2024-04-26T23:30:41.852 INFO:teuthology.orchestra.run.smithi136.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 3.2 MB/s | 305 kB 00:00 2024-04-26T23:30:41.869 INFO:teuthology.orchestra.run.smithi136.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 863 kB/s | 58 kB 00:00 2024-04-26T23:30:41.936 INFO:teuthology.orchestra.run.smithi136.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.1 MB/s | 94 kB 00:00 2024-04-26T23:30:41.961 INFO:teuthology.orchestra.run.smithi136.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 692 kB/s | 64 kB 00:00 2024-04-26T23:30:42.012 INFO:teuthology.orchestra.run.smithi136.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 596 kB/s | 45 kB 00:00 2024-04-26T23:30:42.085 INFO:teuthology.orchestra.run.smithi072.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 71 kB/s | 44 kB 00:00 2024-04-26T23:30:42.188 INFO:teuthology.orchestra.run.smithi136.stdout:(102/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 26 MB/s | 16 MB 00:00 2024-04-26T23:30:42.554 INFO:teuthology.orchestra.run.smithi136.stdout:(103/152): lua-devel-5.4.4-4.el9.x86_64.rpm 38 kB/s | 22 kB 00:00 2024-04-26T23:30:42.871 INFO:teuthology.orchestra.run.smithi136.stdout:(104/152): abseil-cpp-20211102.0-3.el9.x86_64.r 802 kB/s | 548 kB 00:00 2024-04-26T23:30:42.877 INFO:teuthology.orchestra.run.smithi072.stdout:(93/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 753 kB/s | 5.1 MB 00:06 2024-04-26T23:30:42.928 INFO:teuthology.orchestra.run.smithi072.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 262 kB/s | 221 kB 00:00 2024-04-26T23:30:42.996 INFO:teuthology.orchestra.run.smithi136.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 155 kB/s | 19 kB 00:00 2024-04-26T23:30:43.296 INFO:teuthology.orchestra.run.smithi136.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 416 kB/s | 309 kB 00:00 2024-04-26T23:30:43.446 INFO:teuthology.orchestra.run.smithi136.stdout:(107/152): libarrow-doc-9.0.0-11.el9.noarch.rpm 173 kB/s | 26 kB 00:00 2024-04-26T23:30:43.478 INFO:teuthology.orchestra.run.smithi072.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 60 kB/s | 36 kB 00:00 2024-04-26T23:30:43.554 INFO:teuthology.orchestra.run.smithi072.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 76 kB/s | 47 kB 00:00 2024-04-26T23:30:43.555 INFO:teuthology.orchestra.run.smithi136.stdout:(108/152): libarrow-9.0.0-11.el9.x86_64.rpm 7.9 MB/s | 4.4 MB 00:00 2024-04-26T23:30:43.580 INFO:teuthology.orchestra.run.smithi136.stdout:(109/152): protobuf-compiler-3.14.0-13.el9.x86_ 550 kB/s | 863 kB 00:01 2024-04-26T23:30:43.605 INFO:teuthology.orchestra.run.smithi136.stdout:(110/152): liboath-2.6.7-2.el9.x86_64.rpm 306 kB/s | 49 kB 00:00 2024-04-26T23:30:43.689 INFO:teuthology.orchestra.run.smithi136.stdout:(111/152): libunwind-1.6.2-1.el9.x86_64.rpm 503 kB/s | 67 kB 00:00 2024-04-26T23:30:43.866 INFO:teuthology.orchestra.run.smithi136.stdout:(112/152): python3-asyncssh-2.13.2-5.el9.noarch 3.6 MB/s | 548 kB 00:00 2024-04-26T23:30:44.007 INFO:teuthology.orchestra.run.smithi136.stdout:(113/152): parquet-libs-9.0.0-11.el9.x86_64.rpm 2.0 MB/s | 839 kB 00:00 2024-04-26T23:30:44.032 INFO:teuthology.orchestra.run.smithi136.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 260 kB/s | 43 kB 00:00 2024-04-26T23:30:44.148 INFO:teuthology.orchestra.run.smithi136.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 227 kB/s | 32 kB 00:00 2024-04-26T23:30:44.174 INFO:teuthology.orchestra.run.smithi136.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 105 kB/s | 15 kB 00:00 2024-04-26T23:30:44.257 INFO:teuthology.orchestra.run.smithi136.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 1.5 MB/s | 172 kB 00:00 2024-04-26T23:30:44.266 INFO:teuthology.orchestra.run.smithi072.stdout:(97/152): uuid-1.6.2-55.el9.x86_64.rpm 83 kB/s | 58 kB 00:00 2024-04-26T23:30:44.284 INFO:teuthology.orchestra.run.smithi136.stdout:(118/152): luarocks-3.9.2-1.el9.noarch.rpm 215 kB/s | 151 kB 00:00 2024-04-26T23:30:44.317 INFO:teuthology.orchestra.run.smithi136.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 2.5 MB/s | 358 kB 00:00 2024-04-26T23:30:44.384 INFO:teuthology.orchestra.run.smithi136.stdout:(120/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.4 MB/s | 303 kB 00:00 2024-04-26T23:30:44.410 INFO:teuthology.orchestra.run.smithi136.stdout:(121/152): python3-colorama-0.4.4-12.el9.noarch 280 kB/s | 35 kB 00:00 2024-04-26T23:30:44.476 INFO:teuthology.orchestra.run.smithi136.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 1.3 MB/s | 206 kB 00:00 2024-04-26T23:30:44.548 INFO:teuthology.orchestra.run.smithi072.stdout:(98/152): socat-1.7.4.1-5.el9.x86_64.rpm 286 kB/s | 305 kB 00:01 2024-04-26T23:30:44.584 INFO:teuthology.orchestra.run.smithi136.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 825 kB/s | 144 kB 00:00 2024-04-26T23:30:44.635 INFO:teuthology.orchestra.run.smithi136.stdout:(124/152): python3-grpcio-1.46.7-10.el9.x86_64. 8.1 MB/s | 2.0 MB 00:00 2024-04-26T23:30:44.660 INFO:teuthology.orchestra.run.smithi136.stdout:(125/152): python3-jaraco-8.2.1-3.el9.noarch.rp 58 kB/s | 11 kB 00:00 2024-04-26T23:30:44.693 INFO:teuthology.orchestra.run.smithi136.stdout:(126/152): python3-jaraco-classes-3.2.1-5.el9.n 162 kB/s | 18 kB 00:00 2024-04-26T23:30:44.769 INFO:teuthology.orchestra.run.smithi136.stdout:(127/152): python3-jaraco-collections-3.0.0-8.e 173 kB/s | 23 kB 00:00 2024-04-26T23:30:44.795 INFO:teuthology.orchestra.run.smithi136.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 196 kB/s | 20 kB 00:00 2024-04-26T23:30:44.819 INFO:teuthology.orchestra.run.smithi136.stdout:(129/152): python3-jaraco-functools-3.5.0-2.el9 122 kB/s | 19 kB 00:00 2024-04-26T23:30:44.912 INFO:teuthology.orchestra.run.smithi136.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 396 kB/s | 46 kB 00:00 2024-04-26T23:30:44.970 INFO:teuthology.orchestra.run.smithi136.stdout:(131/152): python3-kubernetes-26.1.0-2.el9.noar 5.1 MB/s | 1.0 MB 00:00 2024-04-26T23:30:44.987 INFO:teuthology.orchestra.run.smithi136.stdout:(132/152): python3-more-itertools-8.12.0-2.el9. 471 kB/s | 79 kB 00:00 2024-04-26T23:30:44.988 INFO:teuthology.orchestra.run.smithi072.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 128 kB/s | 94 kB 00:00 2024-04-26T23:30:45.029 INFO:teuthology.orchestra.run.smithi136.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 493 kB/s | 58 kB 00:00 2024-04-26T23:30:45.104 INFO:teuthology.orchestra.run.smithi136.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-04-26T23:30:45.129 INFO:teuthology.orchestra.run.smithi136.stdout:(135/152): python3-portend-3.1.0-2.el9.noarch.r 115 kB/s | 16 kB 00:00 2024-04-26T23:30:45.154 INFO:teuthology.orchestra.run.smithi136.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 716 kB/s | 90 kB 00:00 2024-04-26T23:30:45.205 INFO:teuthology.orchestra.run.smithi136.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 281 kB/s | 28 kB 00:00 2024-04-26T23:30:45.222 INFO:teuthology.orchestra.run.smithi072.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 94 kB/s | 64 kB 00:00 2024-04-26T23:30:45.230 INFO:teuthology.orchestra.run.smithi136.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 306 kB/s | 31 kB 00:00 2024-04-26T23:30:45.305 INFO:teuthology.orchestra.run.smithi136.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 591 kB/s | 59 kB 00:00 2024-04-26T23:30:45.331 INFO:teuthology.orchestra.run.smithi136.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 356 kB/s | 36 kB 00:00 2024-04-26T23:30:45.356 INFO:teuthology.orchestra.run.smithi136.stdout:(141/152): python3-routes-2.5.1-5.el9.noarch.rp 938 kB/s | 188 kB 00:00 2024-04-26T23:30:45.390 INFO:teuthology.orchestra.run.smithi072.stdout:(101/152): lua-devel-5.4.4-4.el9.x86_64.rpm 238 kB/s | 22 kB 00:00 2024-04-26T23:30:45.398 INFO:teuthology.orchestra.run.smithi136.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 553 kB/s | 51 kB 00:00 2024-04-26T23:30:45.457 INFO:teuthology.orchestra.run.smithi136.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-04-26T23:30:45.482 INFO:teuthology.orchestra.run.smithi136.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 714 kB/s | 90 kB 00:00 2024-04-26T23:30:45.524 INFO:teuthology.orchestra.run.smithi136.stdout:(145/152): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-04-26T23:30:45.540 INFO:teuthology.orchestra.run.smithi072.stdout:(102/152): protobuf-compiler-3.14.0-13.el9.x86_ 5.7 MB/s | 863 kB 00:00 2024-04-26T23:30:45.574 INFO:teuthology.orchestra.run.smithi136.stdout:(146/152): python3-zc-lockfile-2.0-10.el9.noarc 170 kB/s | 20 kB 00:00 2024-04-26T23:30:45.633 INFO:teuthology.orchestra.run.smithi136.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-04-26T23:30:45.691 INFO:teuthology.orchestra.run.smithi072.stdout:(103/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 64 kB/s | 45 kB 00:00 2024-04-26T23:30:45.691 INFO:teuthology.orchestra.run.smithi136.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 9.5 MB/s | 1.6 MB 00:00 2024-04-26T23:30:45.884 INFO:teuthology.orchestra.run.smithi136.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 7.0 MB/s | 1.3 MB 00:00 2024-04-26T23:30:46.034 INFO:teuthology.orchestra.run.smithi136.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 16 MB/s | 2.4 MB 00:00 2024-04-26T23:30:46.087 INFO:teuthology.orchestra.run.smithi136.stdout:(151/152): librados2-19.0.0-3306.gbc612ffb.el9. 6.7 MB/s | 3.4 MB 00:00 2024-04-26T23:30:46.150 INFO:teuthology.orchestra.run.smithi072.stdout:(104/152): abseil-cpp-20211102.0-3.el9.x86_64.r 901 kB/s | 548 kB 00:00 2024-04-26T23:30:46.244 INFO:teuthology.orchestra.run.smithi072.stdout:(105/152): gperftools-libs-2.9.1-2.el9.x86_64.r 560 kB/s | 309 kB 00:00 2024-04-26T23:30:46.270 INFO:teuthology.orchestra.run.smithi072.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 163 kB/s | 19 kB 00:00 2024-04-26T23:30:46.370 INFO:teuthology.orchestra.run.smithi072.stdout:(107/152): libarrow-doc-9.0.0-11.el9.noarch.rpm 261 kB/s | 26 kB 00:00 2024-04-26T23:30:46.488 INFO:teuthology.orchestra.run.smithi072.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 418 kB/s | 49 kB 00:00 2024-04-26T23:30:46.588 INFO:teuthology.orchestra.run.smithi072.stdout:(109/152): libunwind-1.6.2-1.el9.x86_64.rpm 678 kB/s | 67 kB 00:00 2024-04-26T23:30:46.653 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-26T23:30:46.653 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /sys 2024-04-26T23:30:46.653 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /proc 2024-04-26T23:30:46.653 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /mnt 2024-04-26T23:30:46.653 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /var/tmp 2024-04-26T23:30:46.653 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /home 2024-04-26T23:30:46.653 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /root 2024-04-26T23:30:46.653 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /tmp 2024-04-26T23:30:46.653 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-26T23:30:46.701 INFO:teuthology.orchestra.run.smithi136.stdout:(152/152): librbd1-19.0.0-3306.gbc612ffb.el9.x8 2.9 MB/s | 3.1 MB 00:01 2024-04-26T23:30:46.707 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-04-26T23:30:46.707 INFO:teuthology.orchestra.run.smithi136.stdout:Total 11 MB/s | 222 MB 00:20 2024-04-26T23:30:46.739 INFO:teuthology.orchestra.run.smithi072.stdout:(110/152): libarrow-9.0.0-11.el9.x86_64.rpm 8.9 MB/s | 4.4 MB 00:00 2024-04-26T23:30:46.764 INFO:teuthology.orchestra.run.smithi072.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 865 kB/s | 151 kB 00:00 2024-04-26T23:30:46.823 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-26T23:30:46.838 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-26T23:30:46.838 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:30:46.838 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-26T23:30:46.838 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-26T23:30:46.890 INFO:teuthology.orchestra.run.smithi072.stdout:(112/152): parquet-libs-9.0.0-11.el9.x86_64.rpm 5.5 MB/s | 839 kB 00:00 2024-04-26T23:30:46.932 INFO:teuthology.orchestra.run.smithi072.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 3.2 MB/s | 548 kB 00:00 2024-04-26T23:30:46.999 INFO:teuthology.orchestra.run.smithi072.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 401 kB/s | 43 kB 00:00 2024-04-26T23:30:47.033 INFO:teuthology.orchestra.run.smithi072.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 321 kB/s | 32 kB 00:00 2024-04-26T23:30:47.101 INFO:teuthology.orchestra.run.smithi072.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 149 kB/s | 15 kB 00:00 2024-04-26T23:30:47.151 INFO:teuthology.orchestra.run.smithi072.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 1.4 MB/s | 172 kB 00:00 2024-04-26T23:30:47.220 INFO:teuthology.orchestra.run.smithi072.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 3.0 MB/s | 358 kB 00:00 2024-04-26T23:30:47.287 INFO:teuthology.orchestra.run.smithi072.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.2 MB/s | 303 kB 00:00 2024-04-26T23:30:47.321 INFO:teuthology.orchestra.run.smithi072.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 353 kB/s | 35 kB 00:00 2024-04-26T23:30:47.405 INFO:teuthology.orchestra.run.smithi072.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 1.7 MB/s | 206 kB 00:00 2024-04-26T23:30:47.514 INFO:teuthology.orchestra.run.smithi072.stdout:(122/152): python3-grpcio-1.46.7-10.el9.x86_64. 11 MB/s | 2.0 MB 00:00 2024-04-26T23:30:47.539 INFO:teuthology.orchestra.run.smithi072.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 1.1 MB/s | 144 kB 00:00 2024-04-26T23:30:47.607 INFO:teuthology.orchestra.run.smithi072.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 117 kB/s | 11 kB 00:00 2024-04-26T23:30:47.707 INFO:teuthology.orchestra.run.smithi072.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 106 kB/s | 18 kB 00:00 2024-04-26T23:30:47.732 INFO:teuthology.orchestra.run.smithi072.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 185 kB/s | 23 kB 00:00 2024-04-26T23:30:47.796 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-04-26T23:30:47.866 INFO:teuthology.orchestra.run.smithi072.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 122 kB/s | 19 kB 00:00 2024-04-26T23:30:47.892 INFO:teuthology.orchestra.run.smithi072.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 124 kB/s | 20 kB 00:00 2024-04-26T23:30:47.940 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-04-26T23:30:47.940 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-04-26T23:30:48.318 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-26T23:30:48.343 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-26T23:30:48.343 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:30:48.343 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-26T23:30:48.343 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T23:30:48.343 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T23:30:48.343 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-26T23:30:48.569 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-26T23:30:48.590 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-26T23:30:48.590 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:30:48.590 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-26T23:30:48.591 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T23:30:48.593 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T23:30:48.594 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-26T23:30:49.028 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-26T23:30:49.049 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-26T23:30:49.050 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:30:49.050 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-26T23:30:49.050 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T23:30:49.050 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T23:30:49.050 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-26T23:30:49.134 INFO:teuthology.orchestra.run.smithi072.stdout:(129/152): python3-logutils-0.3.5-21.el9.noarch 37 kB/s | 46 kB 00:01 2024-04-26T23:30:49.159 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-26T23:30:49.215 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-26T23:30:49.268 INFO:teuthology.orchestra.run.smithi072.stdout:(130/152): python3-kubernetes-26.1.0-2.el9.noar 760 kB/s | 1.0 MB 00:01 2024-04-26T23:30:49.271 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-26T23:30:49.302 INFO:teuthology.orchestra.run.smithi072.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 473 kB/s | 79 kB 00:00 2024-04-26T23:30:49.402 INFO:teuthology.orchestra.run.smithi072.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 433 kB/s | 58 kB 00:00 2024-04-26T23:30:49.438 INFO:teuthology.orchestra.run.smithi072.stdout:(133/152): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-04-26T23:30:49.457 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-26T23:30:49.513 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-26T23:30:49.538 INFO:teuthology.orchestra.run.smithi072.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 121 kB/s | 16 kB 00:00 2024-04-26T23:30:49.572 INFO:teuthology.orchestra.run.smithi072.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 674 kB/s | 90 kB 00:00 2024-04-26T23:30:49.591 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-26T23:30:49.679 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T23:30:49.681 INFO:teuthology.orchestra.run.smithi072.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 199 kB/s | 28 kB 00:00 2024-04-26T23:30:49.706 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-04-26T23:30:49.706 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-04-26T23:30:49.707 INFO:teuthology.orchestra.run.smithi072.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 230 kB/s | 31 kB 00:00 2024-04-26T23:30:49.807 INFO:teuthology.orchestra.run.smithi072.stdout:(138/152): python3-routes-2.5.1-5.el9.noarch.rp 1.5 MB/s | 188 kB 00:00 2024-04-26T23:30:49.819 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T23:30:49.832 INFO:teuthology.orchestra.run.smithi072.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 472 kB/s | 59 kB 00:00 2024-04-26T23:30:49.872 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-26T23:30:49.900 INFO:teuthology.orchestra.run.smithi072.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 390 kB/s | 36 kB 00:00 2024-04-26T23:30:49.904 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-26T23:30:49.925 INFO:teuthology.orchestra.run.smithi072.stdout:(141/152): python3-typing-extensions-4.4.0-2.el 556 kB/s | 51 kB 00:00 2024-04-26T23:30:49.932 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-26T23:30:50.026 INFO:teuthology.orchestra.run.smithi072.stdout:(142/152): python3-webob-1.8.7-6.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-04-26T23:30:50.060 INFO:teuthology.orchestra.run.smithi072.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 671 kB/s | 90 kB 00:00 2024-04-26T23:30:50.210 INFO:teuthology.orchestra.run.smithi072.stdout:(144/152): python3-werkzeug-2.0.3-3.el9.1.noarc 2.3 MB/s | 427 kB 00:00 2024-04-26T23:30:50.244 INFO:teuthology.orchestra.run.smithi072.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 109 kB/s | 20 kB 00:00 2024-04-26T23:30:50.326 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-26T23:30:50.356 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-26T23:30:50.361 INFO:teuthology.orchestra.run.smithi072.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-04-26T23:30:50.388 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 124/156 2024-04-26T23:30:50.454 INFO:teuthology.orchestra.run.smithi072.stdout:(147/152): thrift-0.15.0-2.el9.x86_64.rpm 7.6 MB/s | 1.6 MB 00:00 2024-04-26T23:30:50.458 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 125/156 2024-04-26T23:30:50.488 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-26T23:30:51.039 INFO:teuthology.orchestra.run.smithi072.stdout:(148/152): librbd1-19.0.0-3306.gbc612ffb.el9.x8 6.1 MB/s | 3.1 MB 00:00 2024-04-26T23:30:51.190 INFO:teuthology.orchestra.run.smithi072.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 9.1 MB/s | 1.3 MB 00:00 2024-04-26T23:30:51.374 INFO:teuthology.orchestra.run.smithi072.stdout:(150/152): librados2-19.0.0-3306.gbc612ffb.el9. 3.4 MB/s | 3.4 MB 00:01 2024-04-26T23:30:51.466 INFO:teuthology.orchestra.run.smithi072.stdout:(151/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 8.7 MB/s | 2.4 MB 00:00 2024-04-26T23:30:51.535 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-26T23:30:51.573 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-26T23:30:52.544 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-26T23:30:52.574 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-26T23:30:52.676 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-26T23:30:52.837 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 129/156 2024-04-26T23:30:52.878 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-26T23:30:52.901 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-26T23:30:52.902 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:30:52.902 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-26T23:30:52.902 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T23:30:52.902 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T23:30:52.902 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-26T23:30:52.952 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-26T23:30:52.962 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-26T23:30:53.061 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 132/156 2024-04-26T23:30:53.117 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 133/156 2024-04-26T23:30:54.050 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-26T23:30:54.080 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-26T23:30:54.102 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-26T23:30:54.102 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:30:54.102 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-26T23:30:54.102 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T23:30:54.102 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T23:30:54.102 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-26T23:30:54.152 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-26T23:30:54.169 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-26T23:30:54.169 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:30:54.169 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-26T23:30:54.169 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-26T23:30:54.215 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-26T23:30:54.237 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-26T23:30:54.237 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:30:54.237 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-26T23:30:54.237 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T23:30:54.237 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T23:30:54.237 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-26T23:30:54.541 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-26T23:30:54.562 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-26T23:30:54.563 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:30:54.563 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-26T23:30:54.563 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T23:30:54.563 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T23:30:54.563 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-26T23:30:55.682 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-04-26T23:30:55.762 INFO:teuthology.orchestra.run.smithi136.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-26T23:30:55.800 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-26T23:30:55.853 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-26T23:30:55.888 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-26T23:30:55.913 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-26T23:30:55.956 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-26T23:30:56.276 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-26T23:30:56.381 INFO:teuthology.orchestra.run.smithi136.stdout: Upgrading : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T23:30:56.486 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T23:30:56.516 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T23:30:56.607 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T23:30:56.645 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-26T23:30:56.800 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-26T23:30:56.850 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-26T23:30:56.879 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T23:30:56.983 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T23:30:57.013 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-26T23:30:57.075 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-26T23:30:57.097 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T23:30:57.118 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T23:30:57.322 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T23:30:57.355 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T23:30:57.463 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-26T23:30:57.566 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 18/156 2024-04-26T23:30:57.605 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-26T23:30:57.639 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-26T23:30:57.704 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-26T23:30:57.747 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-26T23:30:57.790 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-26T23:30:57.823 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-26T23:30:57.851 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-26T23:30:57.919 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-26T23:30:57.973 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-26T23:30:58.004 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-26T23:30:58.039 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-26T23:30:58.123 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-26T23:30:58.159 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 31/156 2024-04-26T23:30:58.201 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 32/156 2024-04-26T23:30:58.231 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 33/156 2024-04-26T23:30:58.299 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-26T23:30:58.345 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-26T23:30:58.419 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-26T23:30:58.551 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-26T23:30:58.606 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-26T23:30:59.124 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-26T23:30:59.196 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 139/156 2024-04-26T23:30:59.202 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-26T23:30:59.231 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-26T23:30:59.238 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 140/156 2024-04-26T23:30:59.266 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-26T23:30:59.279 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 141/156 2024-04-26T23:30:59.304 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 43/156 2024-04-26T23:30:59.310 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-26T23:30:59.338 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-26T23:30:59.345 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-26T23:30:59.482 INFO:teuthology.orchestra.run.smithi072.stdout:(152/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 866 kB/s | 16 MB 00:19 2024-04-26T23:30:59.489 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-04-26T23:30:59.489 INFO:teuthology.orchestra.run.smithi072.stdout:Total 2.6 MB/s | 222 MB 01:25 2024-04-26T23:30:59.495 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-26T23:30:59.522 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 144/156 2024-04-26T23:30:59.540 INFO:teuthology.orchestra.run.smithi136.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-26T23:30:59.570 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-26T23:30:59.594 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-26T23:30:59.600 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 146/156 2024-04-26T23:30:59.660 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-26T23:30:59.691 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-26T23:30:59.727 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-26T23:30:59.773 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-26T23:30:59.810 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-26T23:30:59.844 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-26T23:30:59.887 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-26T23:31:00.022 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-26T23:31:00.068 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-26T23:31:00.098 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-26T23:31:00.140 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-26T23:31:00.181 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T23:31:00.222 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T23:31:00.232 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-26T23:31:00.241 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T23:31:00.265 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-26T23:31:00.305 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-26T23:31:00.330 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-26T23:31:00.355 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 63/156 2024-04-26T23:31:00.422 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-26T23:31:00.549 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-26T23:31:00.588 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-26T23:31:00.616 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-26T23:31:00.650 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-26T23:31:00.689 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-26T23:31:00.721 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-26T23:31:00.751 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-26T23:31:00.785 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-26T23:31:00.819 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-26T23:31:00.860 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-26T23:31:00.894 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-26T23:31:00.981 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-26T23:31:01.033 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-26T23:31:01.065 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-26T23:31:01.081 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 148/156 2024-04-26T23:31:01.111 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-26T23:31:01.168 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-26T23:31:01.209 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-26T23:31:01.210 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-26T23:31:01.258 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-26T23:31:01.290 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-26T23:31:01.322 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-26T23:31:01.322 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T23:31:01.405 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T23:31:01.455 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T23:31:01.455 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T23:31:01.546 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T23:31:01.612 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T23:31:01.977 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-26T23:31:02.031 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-26T23:31:02.064 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-26T23:31:02.162 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-26T23:31:02.222 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-26T23:31:02.309 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-26T23:31:02.341 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-04-26T23:31:02.342 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-26T23:31:02.518 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-26T23:31:02.547 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-26T23:31:02.580 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-04-26T23:31:02.581 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-04-26T23:31:02.625 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-26T23:31:03.264 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-26T23:31:03.371 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-26T23:31:04.588 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-26T23:31:04.670 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-26T23:31:04.697 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-26T23:31:04.731 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-26T23:31:05.043 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-26T23:31:05.085 INFO:teuthology.orchestra.run.smithi136.stdout: Upgrading : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-26T23:31:05.175 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-26T23:31:05.207 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 100/156 2024-04-26T23:31:05.243 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-26T23:31:05.372 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-04-26T23:31:05.373 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-04-26T23:31:05.721 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 102/156 2024-04-26T23:31:05.771 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-26T23:31:05.843 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-26T23:31:05.873 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 104/156 2024-04-26T23:31:07.467 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T23:31:07.467 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 1/156 2024-04-26T23:31:07.467 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2/156 2024-04-26T23:31:07.467 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 3/156 2024-04-26T23:31:07.467 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 4/156 2024-04-26T23:31:07.467 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 5/156 2024-04-26T23:31:07.467 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 6/156 2024-04-26T23:31:07.467 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 7/156 2024-04-26T23:31:07.467 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T23:31:07.467 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T23:31:07.467 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-26T23:31:07.467 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 11/156 2024-04-26T23:31:07.467 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 12/156 2024-04-26T23:31:07.468 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T23:31:07.468 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 14/156 2024-04-26T23:31:07.468 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 15/156 2024-04-26T23:31:07.468 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 16/156 2024-04-26T23:31:07.468 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 17/156 2024-04-26T23:31:07.468 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 18/156 2024-04-26T23:31:07.468 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 19/156 2024-04-26T23:31:07.468 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 20/156 2024-04-26T23:31:07.468 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 21/156 2024-04-26T23:31:07.468 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 22/156 2024-04-26T23:31:07.468 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 23/156 2024-04-26T23:31:07.468 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 24/156 2024-04-26T23:31:07.468 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 25/156 2024-04-26T23:31:07.469 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 26/156 2024-04-26T23:31:07.469 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 27/156 2024-04-26T23:31:07.469 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 28/156 2024-04-26T23:31:07.469 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 29/156 2024-04-26T23:31:07.469 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 30/156 2024-04-26T23:31:07.480 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 31/156 2024-04-26T23:31:07.480 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 32/156 2024-04-26T23:31:07.480 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 33/156 2024-04-26T23:31:07.481 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 34/156 2024-04-26T23:31:07.481 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 35/156 2024-04-26T23:31:07.481 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 36/156 2024-04-26T23:31:07.481 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 37/156 2024-04-26T23:31:07.481 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 38/156 2024-04-26T23:31:07.481 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 39/156 2024-04-26T23:31:07.481 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-26T23:31:07.481 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-26T23:31:07.481 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-26T23:31:07.481 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-26T23:31:07.481 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-26T23:31:07.481 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-26T23:31:07.481 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-26T23:31:07.481 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-26T23:31:07.481 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-26T23:31:07.482 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-26T23:31:07.482 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-26T23:31:07.482 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-26T23:31:07.482 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-26T23:31:07.482 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-26T23:31:07.482 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-26T23:31:07.482 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-26T23:31:07.482 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-26T23:31:07.482 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-26T23:31:07.482 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-26T23:31:07.482 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-26T23:31:07.482 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-26T23:31:07.482 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-26T23:31:07.482 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-26T23:31:07.483 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-26T23:31:07.483 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-26T23:31:07.483 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-26T23:31:07.483 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-26T23:31:07.483 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-26T23:31:07.483 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-26T23:31:07.483 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-26T23:31:07.483 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-26T23:31:07.483 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-26T23:31:07.483 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-26T23:31:07.483 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-26T23:31:07.483 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-26T23:31:07.483 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-26T23:31:07.484 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-26T23:31:07.484 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-26T23:31:07.484 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-26T23:31:07.484 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-26T23:31:07.484 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-26T23:31:07.484 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-26T23:31:07.484 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-26T23:31:07.484 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-26T23:31:07.484 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-26T23:31:07.484 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-26T23:31:07.484 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-26T23:31:07.484 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-26T23:31:07.484 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-26T23:31:07.484 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-26T23:31:07.485 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-26T23:31:07.485 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-26T23:31:07.485 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-26T23:31:07.485 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-26T23:31:07.485 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-26T23:31:07.485 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-26T23:31:07.485 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-26T23:31:07.485 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-26T23:31:07.485 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-26T23:31:07.485 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-26T23:31:07.485 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-26T23:31:07.485 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-26T23:31:07.485 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-26T23:31:07.485 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-26T23:31:07.485 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-26T23:31:07.486 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-26T23:31:07.486 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-26T23:31:07.486 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-26T23:31:07.486 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/156 2024-04-26T23:31:07.486 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/156 2024-04-26T23:31:07.486 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-26T23:31:07.486 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-26T23:31:07.486 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-26T23:31:07.486 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/156 2024-04-26T23:31:07.486 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-26T23:31:07.486 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-26T23:31:07.486 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-26T23:31:07.486 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-26T23:31:07.486 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T23:31:07.486 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T23:31:07.487 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-26T23:31:07.487 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-26T23:31:07.487 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-26T23:31:07.487 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-26T23:31:07.487 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-26T23:31:07.487 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-26T23:31:07.487 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-26T23:31:07.487 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-26T23:31:07.487 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-26T23:31:07.487 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-26T23:31:07.487 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-26T23:31:07.487 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-26T23:31:07.487 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-26T23:31:07.487 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-26T23:31:07.487 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-26T23:31:07.488 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-26T23:31:07.488 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-26T23:31:07.488 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-26T23:31:07.488 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-26T23:31:07.488 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-26T23:31:07.488 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-26T23:31:07.488 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-26T23:31:07.488 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-26T23:31:07.488 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-26T23:31:07.488 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-26T23:31:07.488 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-26T23:31:07.488 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-26T23:31:07.488 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-26T23:31:07.488 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-26T23:31:07.488 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 149/156 2024-04-26T23:31:07.489 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-26T23:31:07.489 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 151/156 2024-04-26T23:31:07.489 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-26T23:31:07.489 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-26T23:31:07.489 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T23:31:07.489 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-26T23:31:08.132 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T23:31:08.170 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T23:31:08.711 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T23:31:08.757 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-26T23:31:08.859 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-26T23:31:08.859 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-26T23:31:08.859 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-26T23:31:08.859 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-26T23:31:08.883 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-26T23:31:09.622 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T23:31:09.622 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-26T23:31:09.622 INFO:teuthology.orchestra.run.smithi076.stdout:Upgraded: 2024-04-26T23:31:09.622 INFO:teuthology.orchestra.run.smithi076.stdout: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:09.622 INFO:teuthology.orchestra.run.smithi076.stdout: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:09.622 INFO:teuthology.orchestra.run.smithi076.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-26T23:31:09.622 INFO:teuthology.orchestra.run.smithi076.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-26T23:31:09.622 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-04-26T23:31:09.622 INFO:teuthology.orchestra.run.smithi076.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-26T23:31:09.623 INFO:teuthology.orchestra.run.smithi076.stdout: automake-1.16.2-8.el9.noarch 2024-04-26T23:31:09.623 INFO:teuthology.orchestra.run.smithi076.stdout: bison-3.7.4-5.el9.x86_64 2024-04-26T23:31:09.623 INFO:teuthology.orchestra.run.smithi076.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-26T23:31:09.623 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:09.623 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:09.623 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:09.623 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:09.623 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:31:09.623 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-immutable-object-cache-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:09.623 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:09.623 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:09.623 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:31:09.624 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:31:09.624 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:31:09.624 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:31:09.624 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:31:09.624 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:09.624 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:09.624 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:31:09.624 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:09.624 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:09.624 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:09.624 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:31:09.624 INFO:teuthology.orchestra.run.smithi076.stdout: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:31:09.624 INFO:teuthology.orchestra.run.smithi076.stdout: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:09.624 INFO:teuthology.orchestra.run.smithi076.stdout: cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:09.625 INFO:teuthology.orchestra.run.smithi076.stdout: cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:31:09.625 INFO:teuthology.orchestra.run.smithi076.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-26T23:31:09.625 INFO:teuthology.orchestra.run.smithi076.stdout: flex-2.6.4-9.el9.x86_64 2024-04-26T23:31:09.625 INFO:teuthology.orchestra.run.smithi076.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-26T23:31:09.625 INFO:teuthology.orchestra.run.smithi076.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-26T23:31:09.625 INFO:teuthology.orchestra.run.smithi076.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-26T23:31:09.625 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-26T23:31:09.625 INFO:teuthology.orchestra.run.smithi076.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-26T23:31:09.625 INFO:teuthology.orchestra.run.smithi076.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-26T23:31:09.625 INFO:teuthology.orchestra.run.smithi076.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-26T23:31:09.625 INFO:teuthology.orchestra.run.smithi076.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-26T23:31:09.625 INFO:teuthology.orchestra.run.smithi076.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-26T23:31:09.625 INFO:teuthology.orchestra.run.smithi076.stdout: libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:09.625 INFO:teuthology.orchestra.run.smithi076.stdout: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:09.626 INFO:teuthology.orchestra.run.smithi076.stdout: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:09.626 INFO:teuthology.orchestra.run.smithi076.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-26T23:31:09.626 INFO:teuthology.orchestra.run.smithi076.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-26T23:31:09.626 INFO:teuthology.orchestra.run.smithi076.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-26T23:31:09.626 INFO:teuthology.orchestra.run.smithi076.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-26T23:31:09.626 INFO:teuthology.orchestra.run.smithi076.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-26T23:31:09.626 INFO:teuthology.orchestra.run.smithi076.stdout: librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:09.626 INFO:teuthology.orchestra.run.smithi076.stdout: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:09.626 INFO:teuthology.orchestra.run.smithi076.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-26T23:31:09.626 INFO:teuthology.orchestra.run.smithi076.stdout: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:09.626 INFO:teuthology.orchestra.run.smithi076.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-26T23:31:09.626 INFO:teuthology.orchestra.run.smithi076.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-26T23:31:09.626 INFO:teuthology.orchestra.run.smithi076.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-26T23:31:09.626 INFO:teuthology.orchestra.run.smithi076.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-26T23:31:09.627 INFO:teuthology.orchestra.run.smithi076.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-26T23:31:09.627 INFO:teuthology.orchestra.run.smithi076.stdout: lua-5.4.4-4.el9.x86_64 2024-04-26T23:31:09.627 INFO:teuthology.orchestra.run.smithi076.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-26T23:31:09.627 INFO:teuthology.orchestra.run.smithi076.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-26T23:31:09.627 INFO:teuthology.orchestra.run.smithi076.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-26T23:31:09.627 INFO:teuthology.orchestra.run.smithi076.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-26T23:31:09.627 INFO:teuthology.orchestra.run.smithi076.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-26T23:31:09.627 INFO:teuthology.orchestra.run.smithi076.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-26T23:31:09.627 INFO:teuthology.orchestra.run.smithi076.stdout: postgresql-13.14-1.el9.x86_64 2024-04-26T23:31:09.627 INFO:teuthology.orchestra.run.smithi076.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-26T23:31:09.627 INFO:teuthology.orchestra.run.smithi076.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-26T23:31:09.627 INFO:teuthology.orchestra.run.smithi076.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-26T23:31:09.627 INFO:teuthology.orchestra.run.smithi076.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-26T23:31:09.627 INFO:teuthology.orchestra.run.smithi076.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-26T23:31:09.627 INFO:teuthology.orchestra.run.smithi076.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-26T23:31:09.628 INFO:teuthology.orchestra.run.smithi076.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-26T23:31:09.628 INFO:teuthology.orchestra.run.smithi076.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-26T23:31:09.628 INFO:teuthology.orchestra.run.smithi076.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-26T23:31:09.628 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-26T23:31:09.628 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:09.628 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:09.628 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:09.628 INFO:teuthology.orchestra.run.smithi076.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-26T23:31:09.628 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-26T23:31:09.628 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-26T23:31:09.628 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-26T23:31:09.628 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-26T23:31:09.628 INFO:teuthology.orchestra.run.smithi076.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-26T23:31:09.628 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-26T23:31:09.629 INFO:teuthology.orchestra.run.smithi076.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-26T23:31:09.629 INFO:teuthology.orchestra.run.smithi076.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-26T23:31:09.629 INFO:teuthology.orchestra.run.smithi076.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-26T23:31:09.629 INFO:teuthology.orchestra.run.smithi076.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-26T23:31:09.629 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-26T23:31:09.629 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-26T23:31:09.629 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-26T23:31:09.629 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-26T23:31:09.629 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-26T23:31:09.629 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-26T23:31:09.629 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-26T23:31:09.629 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-26T23:31:09.629 INFO:teuthology.orchestra.run.smithi076.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-26T23:31:09.629 INFO:teuthology.orchestra.run.smithi076.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-26T23:31:09.629 INFO:teuthology.orchestra.run.smithi076.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-26T23:31:09.630 INFO:teuthology.orchestra.run.smithi076.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-26T23:31:09.630 INFO:teuthology.orchestra.run.smithi076.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-26T23:31:09.630 INFO:teuthology.orchestra.run.smithi076.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-26T23:31:09.630 INFO:teuthology.orchestra.run.smithi076.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-26T23:31:09.630 INFO:teuthology.orchestra.run.smithi076.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-26T23:31:09.630 INFO:teuthology.orchestra.run.smithi076.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-26T23:31:09.630 INFO:teuthology.orchestra.run.smithi076.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-26T23:31:09.630 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-26T23:31:09.630 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ply-3.11-14.el9.noarch 2024-04-26T23:31:09.630 INFO:teuthology.orchestra.run.smithi076.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-26T23:31:09.630 INFO:teuthology.orchestra.run.smithi076.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-26T23:31:09.630 INFO:teuthology.orchestra.run.smithi076.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-26T23:31:09.630 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-26T23:31:09.630 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-26T23:31:09.631 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-26T23:31:09.631 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-26T23:31:09.632 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-26T23:31:09.632 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-26T23:31:09.632 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:09.632 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:09.632 INFO:teuthology.orchestra.run.smithi076.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-26T23:31:09.633 INFO:teuthology.orchestra.run.smithi076.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-26T23:31:09.633 INFO:teuthology.orchestra.run.smithi076.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-26T23:31:09.633 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:09.633 INFO:teuthology.orchestra.run.smithi076.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-26T23:31:09.633 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-26T23:31:09.633 INFO:teuthology.orchestra.run.smithi076.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-26T23:31:09.634 INFO:teuthology.orchestra.run.smithi076.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-26T23:31:09.634 INFO:teuthology.orchestra.run.smithi076.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-26T23:31:09.634 INFO:teuthology.orchestra.run.smithi076.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-26T23:31:09.634 INFO:teuthology.orchestra.run.smithi076.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-26T23:31:09.634 INFO:teuthology.orchestra.run.smithi076.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-26T23:31:09.635 INFO:teuthology.orchestra.run.smithi076.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-26T23:31:09.635 INFO:teuthology.orchestra.run.smithi076.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-26T23:31:09.635 INFO:teuthology.orchestra.run.smithi076.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-26T23:31:09.635 INFO:teuthology.orchestra.run.smithi076.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-26T23:31:09.635 INFO:teuthology.orchestra.run.smithi076.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-26T23:31:09.635 INFO:teuthology.orchestra.run.smithi076.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-26T23:31:09.636 INFO:teuthology.orchestra.run.smithi076.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-26T23:31:09.636 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:09.636 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:09.636 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:09.636 INFO:teuthology.orchestra.run.smithi076.stdout: re2-1:20211101-3.el9.x86_64 2024-04-26T23:31:09.637 INFO:teuthology.orchestra.run.smithi076.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-26T23:31:09.637 INFO:teuthology.orchestra.run.smithi076.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-26T23:31:09.637 INFO:teuthology.orchestra.run.smithi076.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-26T23:31:09.637 INFO:teuthology.orchestra.run.smithi076.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-26T23:31:09.637 INFO:teuthology.orchestra.run.smithi076.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-26T23:31:09.638 INFO:teuthology.orchestra.run.smithi076.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-26T23:31:09.638 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-04-26T23:31:09.638 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-04-26T23:31:10.045 DEBUG:teuthology.parallel:result is None 2024-04-26T23:31:11.757 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-04-26T23:31:11.933 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-26T23:31:12.021 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-26T23:31:12.155 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-26T23:31:12.220 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-26T23:31:12.263 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-26T23:31:12.391 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-26T23:31:12.818 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-26T23:31:12.996 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T23:31:13.180 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T23:31:13.208 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T23:31:13.335 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T23:31:13.374 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-26T23:31:13.762 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-26T23:31:13.826 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-26T23:31:13.859 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T23:31:14.019 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T23:31:14.053 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-26T23:31:14.156 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-26T23:31:14.188 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T23:31:14.252 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T23:31:14.593 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T23:31:14.661 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T23:31:14.857 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-26T23:31:14.982 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 18/156 2024-04-26T23:31:15.028 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-26T23:31:15.077 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-26T23:31:15.181 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-26T23:31:15.243 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-26T23:31:15.308 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-26T23:31:15.357 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-26T23:31:15.398 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-26T23:31:15.522 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-26T23:31:15.588 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-26T23:31:15.629 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-26T23:31:15.696 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-26T23:31:15.811 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-26T23:31:15.850 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 31/156 2024-04-26T23:31:15.901 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 32/156 2024-04-26T23:31:15.932 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 33/156 2024-04-26T23:31:16.091 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-26T23:31:16.156 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-26T23:31:16.263 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-26T23:31:16.440 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-26T23:31:16.510 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-26T23:31:17.192 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-26T23:31:17.303 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-26T23:31:17.340 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-26T23:31:17.387 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-26T23:31:17.428 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 43/156 2024-04-26T23:31:17.497 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-26T23:31:17.770 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-26T23:31:17.859 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-26T23:31:17.942 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-26T23:31:18.051 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-26T23:31:18.086 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-26T23:31:18.134 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-26T23:31:18.203 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-26T23:31:18.259 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-26T23:31:18.323 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-26T23:31:18.390 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-26T23:31:18.595 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-26T23:31:18.654 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-26T23:31:18.701 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-26T23:31:18.758 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-26T23:31:18.898 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-26T23:31:18.951 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-26T23:31:19.012 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-26T23:31:19.054 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-26T23:31:19.087 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 63/156 2024-04-26T23:31:19.272 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-26T23:31:19.488 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-26T23:31:19.533 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-26T23:31:19.572 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-26T23:31:19.618 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-26T23:31:19.678 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-26T23:31:19.740 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-26T23:31:19.791 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-26T23:31:19.845 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-26T23:31:19.897 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-26T23:31:19.972 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-26T23:31:20.021 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-26T23:31:20.164 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-26T23:31:20.227 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-26T23:31:20.281 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-26T23:31:20.353 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-26T23:31:20.508 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-26T23:31:20.579 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-26T23:31:20.740 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-26T23:31:20.740 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /sys 2024-04-26T23:31:20.740 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /proc 2024-04-26T23:31:20.740 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /mnt 2024-04-26T23:31:20.740 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /var/tmp 2024-04-26T23:31:20.740 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /home 2024-04-26T23:31:20.740 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /root 2024-04-26T23:31:20.740 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /tmp 2024-04-26T23:31:20.741 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-26T23:31:20.931 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-26T23:31:20.950 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-26T23:31:20.951 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:31:20.951 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-26T23:31:20.951 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-26T23:31:21.536 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-26T23:31:21.601 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-26T23:31:21.635 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-26T23:31:21.773 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-26T23:31:21.857 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-26T23:31:21.962 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-26T23:31:22.007 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-26T23:31:22.243 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-26T23:31:22.272 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-26T23:31:22.420 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-26T23:31:22.434 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-26T23:31:22.460 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-26T23:31:22.460 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:31:22.460 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-26T23:31:22.460 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T23:31:22.460 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T23:31:22.460 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-26T23:31:22.685 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-26T23:31:22.707 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-26T23:31:22.707 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:31:22.707 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-26T23:31:22.708 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T23:31:22.708 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T23:31:22.708 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-26T23:31:23.152 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-26T23:31:23.175 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-26T23:31:23.175 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:31:23.175 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-26T23:31:23.175 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T23:31:23.175 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T23:31:23.175 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-26T23:31:23.259 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-26T23:31:23.283 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-26T23:31:23.339 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-26T23:31:23.394 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-26T23:31:23.412 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-26T23:31:23.596 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-26T23:31:23.645 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-26T23:31:23.723 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-26T23:31:23.811 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T23:31:23.942 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T23:31:23.995 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-26T23:31:24.027 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-26T23:31:24.055 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-26T23:31:24.517 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-26T23:31:24.546 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-26T23:31:24.570 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 124/156 2024-04-26T23:31:24.631 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 125/156 2024-04-26T23:31:24.660 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-26T23:31:25.052 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-26T23:31:25.204 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-26T23:31:25.263 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-26T23:31:25.320 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-26T23:31:25.700 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-26T23:31:25.729 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-26T23:31:25.755 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-26T23:31:25.810 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-26T23:31:25.936 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-26T23:31:26.003 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 100/156 2024-04-26T23:31:26.064 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-26T23:31:26.613 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 102/156 2024-04-26T23:31:26.651 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-26T23:31:26.695 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-26T23:31:26.721 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-26T23:31:26.743 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-26T23:31:26.812 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 104/156 2024-04-26T23:31:26.821 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-26T23:31:27.010 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 129/156 2024-04-26T23:31:27.040 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-26T23:31:27.063 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-26T23:31:27.063 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:31:27.063 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-26T23:31:27.063 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T23:31:27.063 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T23:31:27.063 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-26T23:31:27.107 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-26T23:31:27.118 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-26T23:31:27.216 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 132/156 2024-04-26T23:31:27.264 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 133/156 2024-04-26T23:31:28.191 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-26T23:31:28.218 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-26T23:31:28.241 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-26T23:31:28.241 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:31:28.241 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-26T23:31:28.241 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T23:31:28.241 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T23:31:28.241 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-26T23:31:28.286 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-26T23:31:28.308 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-26T23:31:28.308 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:31:28.308 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-26T23:31:28.308 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-26T23:31:28.353 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-26T23:31:28.375 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-26T23:31:28.375 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:31:28.375 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-26T23:31:28.375 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T23:31:28.375 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T23:31:28.375 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-26T23:31:28.678 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-26T23:31:28.701 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-26T23:31:28.701 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:31:28.701 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-26T23:31:28.701 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T23:31:28.701 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T23:31:28.701 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-26T23:31:29.492 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T23:31:29.541 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T23:31:30.180 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T23:31:30.231 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-26T23:31:30.390 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-26T23:31:30.390 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-26T23:31:30.390 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-26T23:31:30.390 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:31:30.435 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-26T23:31:33.324 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 139/156 2024-04-26T23:31:33.365 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 140/156 2024-04-26T23:31:33.406 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 141/156 2024-04-26T23:31:33.438 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-26T23:31:33.465 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-26T23:31:33.646 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 144/156 2024-04-26T23:31:33.695 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-26T23:31:33.725 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 146/156 2024-04-26T23:31:34.382 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T23:31:34.426 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T23:31:34.443 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T23:31:35.270 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 148/156 2024-04-26T23:31:35.363 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-26T23:31:35.404 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-26T23:31:35.485 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-26T23:31:35.526 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-26T23:31:35.526 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T23:31:35.618 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T23:31:35.651 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T23:31:35.651 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T23:31:35.738 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T23:31:35.807 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T23:31:41.569 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T23:31:41.570 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 1/156 2024-04-26T23:31:41.570 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2/156 2024-04-26T23:31:41.570 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 3/156 2024-04-26T23:31:41.570 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 4/156 2024-04-26T23:31:41.570 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 5/156 2024-04-26T23:31:41.570 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 6/156 2024-04-26T23:31:41.570 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 7/156 2024-04-26T23:31:41.570 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T23:31:41.570 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T23:31:41.570 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-26T23:31:41.570 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 11/156 2024-04-26T23:31:41.570 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 12/156 2024-04-26T23:31:41.571 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T23:31:41.571 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 14/156 2024-04-26T23:31:41.571 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 15/156 2024-04-26T23:31:41.571 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 16/156 2024-04-26T23:31:41.571 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 17/156 2024-04-26T23:31:41.571 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 18/156 2024-04-26T23:31:41.571 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 19/156 2024-04-26T23:31:41.571 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 20/156 2024-04-26T23:31:41.571 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 21/156 2024-04-26T23:31:41.571 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 22/156 2024-04-26T23:31:41.571 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 23/156 2024-04-26T23:31:41.571 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 24/156 2024-04-26T23:31:41.572 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 25/156 2024-04-26T23:31:41.572 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 26/156 2024-04-26T23:31:41.572 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 27/156 2024-04-26T23:31:41.572 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 28/156 2024-04-26T23:31:41.572 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 29/156 2024-04-26T23:31:41.572 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 30/156 2024-04-26T23:31:41.572 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 31/156 2024-04-26T23:31:41.572 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 32/156 2024-04-26T23:31:41.572 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 33/156 2024-04-26T23:31:41.572 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 34/156 2024-04-26T23:31:41.572 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 35/156 2024-04-26T23:31:41.572 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 36/156 2024-04-26T23:31:41.573 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 37/156 2024-04-26T23:31:41.573 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 38/156 2024-04-26T23:31:41.573 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 39/156 2024-04-26T23:31:41.573 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-26T23:31:41.573 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-26T23:31:41.573 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-26T23:31:41.573 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-26T23:31:41.573 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-26T23:31:41.573 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-26T23:31:41.573 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-26T23:31:41.573 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-26T23:31:41.573 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-26T23:31:41.573 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-26T23:31:41.574 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-26T23:31:41.574 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-26T23:31:41.574 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-26T23:31:41.574 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-26T23:31:41.574 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-26T23:31:41.574 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-26T23:31:41.574 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-26T23:31:41.574 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-26T23:31:41.574 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-26T23:31:41.574 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-26T23:31:41.574 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-26T23:31:41.574 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-26T23:31:41.574 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-26T23:31:41.575 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-26T23:31:41.575 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-26T23:31:41.575 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-26T23:31:41.575 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-26T23:31:41.575 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-26T23:31:41.575 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-26T23:31:41.575 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-26T23:31:41.575 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-26T23:31:41.576 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-26T23:31:41.576 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-26T23:31:41.576 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-26T23:31:41.576 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-26T23:31:41.576 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-26T23:31:41.576 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-26T23:31:41.576 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-26T23:31:41.576 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-26T23:31:41.576 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-26T23:31:41.576 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-26T23:31:41.576 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-26T23:31:41.576 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-26T23:31:41.576 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-26T23:31:41.576 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-26T23:31:41.577 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-26T23:31:41.577 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-26T23:31:41.577 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-26T23:31:41.577 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-26T23:31:41.577 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-26T23:31:41.577 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-26T23:31:41.577 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-26T23:31:41.577 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-26T23:31:41.577 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-26T23:31:41.577 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-26T23:31:41.578 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-26T23:31:41.578 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-26T23:31:41.578 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-26T23:31:41.578 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-26T23:31:41.578 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-26T23:31:41.578 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-26T23:31:41.578 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-26T23:31:41.578 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-26T23:31:41.578 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-26T23:31:41.578 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-26T23:31:41.578 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-26T23:31:41.578 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-26T23:31:41.578 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-26T23:31:41.578 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/156 2024-04-26T23:31:41.579 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/156 2024-04-26T23:31:41.579 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-26T23:31:41.579 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-26T23:31:41.579 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-26T23:31:41.579 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/156 2024-04-26T23:31:41.579 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-26T23:31:41.579 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-26T23:31:41.579 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-26T23:31:41.579 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-26T23:31:41.579 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T23:31:41.579 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T23:31:41.580 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-26T23:31:41.580 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-26T23:31:41.580 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-26T23:31:41.580 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-26T23:31:41.580 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-26T23:31:41.580 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-26T23:31:41.580 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-26T23:31:41.580 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-26T23:31:41.580 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-26T23:31:41.580 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-26T23:31:41.580 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-26T23:31:41.580 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-26T23:31:41.580 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-26T23:31:41.580 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-26T23:31:41.581 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-26T23:31:41.581 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-26T23:31:41.581 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-26T23:31:41.581 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-26T23:31:41.581 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-26T23:31:41.581 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-26T23:31:41.581 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-26T23:31:41.581 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-26T23:31:41.581 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-26T23:31:41.581 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-26T23:31:41.581 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-26T23:31:41.581 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-26T23:31:41.581 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-26T23:31:41.582 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-26T23:31:41.582 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-26T23:31:41.582 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 149/156 2024-04-26T23:31:41.582 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-26T23:31:41.582 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 151/156 2024-04-26T23:31:41.582 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-26T23:31:41.582 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-26T23:31:41.582 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T23:31:41.582 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-26T23:31:43.642 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T23:31:43.642 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-26T23:31:43.642 INFO:teuthology.orchestra.run.smithi136.stdout:Upgraded: 2024-04-26T23:31:43.642 INFO:teuthology.orchestra.run.smithi136.stdout: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:43.642 INFO:teuthology.orchestra.run.smithi136.stdout: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:43.642 INFO:teuthology.orchestra.run.smithi136.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-26T23:31:43.643 INFO:teuthology.orchestra.run.smithi136.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-26T23:31:43.643 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-04-26T23:31:43.643 INFO:teuthology.orchestra.run.smithi136.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-26T23:31:43.643 INFO:teuthology.orchestra.run.smithi136.stdout: automake-1.16.2-8.el9.noarch 2024-04-26T23:31:43.643 INFO:teuthology.orchestra.run.smithi136.stdout: bison-3.7.4-5.el9.x86_64 2024-04-26T23:31:43.643 INFO:teuthology.orchestra.run.smithi136.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-26T23:31:43.643 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:43.643 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:43.643 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:43.643 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:43.643 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:31:43.643 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-immutable-object-cache-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:43.643 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:43.643 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:43.643 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:31:43.644 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:31:43.644 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:31:43.644 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:31:43.644 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:31:43.644 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:43.644 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:43.644 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:31:43.644 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:43.644 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:43.644 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:43.644 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:31:43.644 INFO:teuthology.orchestra.run.smithi136.stdout: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:31:43.644 INFO:teuthology.orchestra.run.smithi136.stdout: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:43.644 INFO:teuthology.orchestra.run.smithi136.stdout: cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:43.645 INFO:teuthology.orchestra.run.smithi136.stdout: cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:31:43.645 INFO:teuthology.orchestra.run.smithi136.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-26T23:31:43.645 INFO:teuthology.orchestra.run.smithi136.stdout: flex-2.6.4-9.el9.x86_64 2024-04-26T23:31:43.645 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-26T23:31:43.645 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-26T23:31:43.645 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-26T23:31:43.645 INFO:teuthology.orchestra.run.smithi136.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-26T23:31:43.645 INFO:teuthology.orchestra.run.smithi136.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-26T23:31:43.645 INFO:teuthology.orchestra.run.smithi136.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-26T23:31:43.645 INFO:teuthology.orchestra.run.smithi136.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-26T23:31:43.645 INFO:teuthology.orchestra.run.smithi136.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-26T23:31:43.645 INFO:teuthology.orchestra.run.smithi136.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-26T23:31:43.645 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:43.646 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:43.646 INFO:teuthology.orchestra.run.smithi136.stdout: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:43.646 INFO:teuthology.orchestra.run.smithi136.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-26T23:31:43.646 INFO:teuthology.orchestra.run.smithi136.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-26T23:31:43.646 INFO:teuthology.orchestra.run.smithi136.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-26T23:31:43.646 INFO:teuthology.orchestra.run.smithi136.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-26T23:31:43.650 INFO:teuthology.orchestra.run.smithi136.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-26T23:31:43.650 INFO:teuthology.orchestra.run.smithi136.stdout: librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:43.650 INFO:teuthology.orchestra.run.smithi136.stdout: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:43.650 INFO:teuthology.orchestra.run.smithi136.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-26T23:31:43.650 INFO:teuthology.orchestra.run.smithi136.stdout: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:43.650 INFO:teuthology.orchestra.run.smithi136.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-26T23:31:43.650 INFO:teuthology.orchestra.run.smithi136.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-26T23:31:43.651 INFO:teuthology.orchestra.run.smithi136.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-26T23:31:43.651 INFO:teuthology.orchestra.run.smithi136.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-26T23:31:43.651 INFO:teuthology.orchestra.run.smithi136.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-26T23:31:43.651 INFO:teuthology.orchestra.run.smithi136.stdout: lua-5.4.4-4.el9.x86_64 2024-04-26T23:31:43.651 INFO:teuthology.orchestra.run.smithi136.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-26T23:31:43.651 INFO:teuthology.orchestra.run.smithi136.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-26T23:31:43.651 INFO:teuthology.orchestra.run.smithi136.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-26T23:31:43.651 INFO:teuthology.orchestra.run.smithi136.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-26T23:31:43.651 INFO:teuthology.orchestra.run.smithi136.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-26T23:31:43.651 INFO:teuthology.orchestra.run.smithi136.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-26T23:31:43.652 INFO:teuthology.orchestra.run.smithi136.stdout: postgresql-13.14-1.el9.x86_64 2024-04-26T23:31:43.652 INFO:teuthology.orchestra.run.smithi136.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-26T23:31:43.652 INFO:teuthology.orchestra.run.smithi136.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-26T23:31:43.652 INFO:teuthology.orchestra.run.smithi136.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-26T23:31:43.652 INFO:teuthology.orchestra.run.smithi136.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-26T23:31:43.652 INFO:teuthology.orchestra.run.smithi136.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-26T23:31:43.652 INFO:teuthology.orchestra.run.smithi136.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-26T23:31:43.652 INFO:teuthology.orchestra.run.smithi136.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-26T23:31:43.652 INFO:teuthology.orchestra.run.smithi136.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-26T23:31:43.652 INFO:teuthology.orchestra.run.smithi136.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-26T23:31:43.652 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-26T23:31:43.652 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:43.652 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:43.652 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:43.653 INFO:teuthology.orchestra.run.smithi136.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-26T23:31:43.653 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-26T23:31:43.653 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-26T23:31:43.653 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-26T23:31:43.653 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-26T23:31:43.653 INFO:teuthology.orchestra.run.smithi136.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-26T23:31:43.653 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-26T23:31:43.653 INFO:teuthology.orchestra.run.smithi136.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-26T23:31:43.653 INFO:teuthology.orchestra.run.smithi136.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-26T23:31:43.653 INFO:teuthology.orchestra.run.smithi136.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-26T23:31:43.653 INFO:teuthology.orchestra.run.smithi136.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-26T23:31:43.653 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-26T23:31:43.653 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-26T23:31:43.653 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-26T23:31:43.653 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-26T23:31:43.654 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-26T23:31:43.654 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-26T23:31:43.654 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-26T23:31:43.654 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-26T23:31:43.654 INFO:teuthology.orchestra.run.smithi136.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-26T23:31:43.654 INFO:teuthology.orchestra.run.smithi136.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-26T23:31:43.654 INFO:teuthology.orchestra.run.smithi136.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-26T23:31:43.654 INFO:teuthology.orchestra.run.smithi136.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-26T23:31:43.654 INFO:teuthology.orchestra.run.smithi136.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-26T23:31:43.654 INFO:teuthology.orchestra.run.smithi136.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-26T23:31:43.654 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-26T23:31:43.654 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-26T23:31:43.654 INFO:teuthology.orchestra.run.smithi136.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-26T23:31:43.654 INFO:teuthology.orchestra.run.smithi136.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-26T23:31:43.655 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-26T23:31:43.655 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ply-3.11-14.el9.noarch 2024-04-26T23:31:43.655 INFO:teuthology.orchestra.run.smithi136.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-26T23:31:43.655 INFO:teuthology.orchestra.run.smithi136.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-26T23:31:43.655 INFO:teuthology.orchestra.run.smithi136.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-26T23:31:43.655 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-26T23:31:43.655 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-26T23:31:43.655 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-26T23:31:43.655 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-26T23:31:43.655 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-26T23:31:43.655 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-26T23:31:43.655 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:43.655 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:43.655 INFO:teuthology.orchestra.run.smithi136.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-26T23:31:43.655 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-26T23:31:43.656 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-26T23:31:43.656 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:43.656 INFO:teuthology.orchestra.run.smithi136.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-26T23:31:43.656 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-26T23:31:43.656 INFO:teuthology.orchestra.run.smithi136.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-26T23:31:43.656 INFO:teuthology.orchestra.run.smithi136.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-26T23:31:43.656 INFO:teuthology.orchestra.run.smithi136.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-26T23:31:43.656 INFO:teuthology.orchestra.run.smithi136.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-26T23:31:43.656 INFO:teuthology.orchestra.run.smithi136.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-26T23:31:43.656 INFO:teuthology.orchestra.run.smithi136.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-26T23:31:43.656 INFO:teuthology.orchestra.run.smithi136.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-26T23:31:43.656 INFO:teuthology.orchestra.run.smithi136.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-26T23:31:43.656 INFO:teuthology.orchestra.run.smithi136.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-26T23:31:43.656 INFO:teuthology.orchestra.run.smithi136.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-26T23:31:43.656 INFO:teuthology.orchestra.run.smithi136.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-26T23:31:43.656 INFO:teuthology.orchestra.run.smithi136.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-26T23:31:43.657 INFO:teuthology.orchestra.run.smithi136.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-26T23:31:43.657 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:43.657 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:43.657 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:31:43.657 INFO:teuthology.orchestra.run.smithi136.stdout: re2-1:20211101-3.el9.x86_64 2024-04-26T23:31:43.657 INFO:teuthology.orchestra.run.smithi136.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-26T23:31:43.657 INFO:teuthology.orchestra.run.smithi136.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-26T23:31:43.657 INFO:teuthology.orchestra.run.smithi136.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-26T23:31:43.657 INFO:teuthology.orchestra.run.smithi136.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-26T23:31:43.657 INFO:teuthology.orchestra.run.smithi136.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-26T23:31:43.658 INFO:teuthology.orchestra.run.smithi136.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-26T23:31:43.658 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-26T23:31:43.658 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-04-26T23:31:44.025 DEBUG:teuthology.parallel:result is None 2024-04-26T23:31:47.200 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-26T23:31:47.200 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /sys 2024-04-26T23:31:47.200 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /proc 2024-04-26T23:31:47.201 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /mnt 2024-04-26T23:31:47.201 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /var/tmp 2024-04-26T23:31:47.201 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /home 2024-04-26T23:31:47.201 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /root 2024-04-26T23:31:47.201 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /tmp 2024-04-26T23:31:47.201 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:31:47.433 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-26T23:31:47.478 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-26T23:31:47.479 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:31:47.479 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-26T23:31:47.479 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:31:49.103 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-26T23:31:49.185 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-26T23:31:49.185 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:31:49.185 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-26T23:31:49.185 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T23:31:49.185 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T23:31:49.185 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:31:49.439 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-26T23:31:49.508 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-26T23:31:49.508 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:31:49.508 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-26T23:31:49.508 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T23:31:49.508 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T23:31:49.508 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:31:49.999 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-26T23:31:50.068 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-26T23:31:50.068 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:31:50.068 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-26T23:31:50.068 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T23:31:50.069 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T23:31:50.069 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:31:50.208 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-26T23:31:50.280 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-26T23:31:50.367 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-26T23:31:50.610 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-26T23:31:50.676 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-26T23:31:50.769 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-26T23:31:50.879 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T23:31:51.050 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T23:31:51.103 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-26T23:31:51.155 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-26T23:31:51.214 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-26T23:31:51.863 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-26T23:31:51.915 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-26T23:31:51.974 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 124/156 2024-04-26T23:31:52.120 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 125/156 2024-04-26T23:31:52.159 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-26T23:31:53.472 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-26T23:31:53.531 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-26T23:31:54.635 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-26T23:31:54.664 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-26T23:31:54.809 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-26T23:31:55.018 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 129/156 2024-04-26T23:31:55.269 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-26T23:31:55.357 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-26T23:31:55.357 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:31:55.357 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-26T23:31:55.357 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T23:31:55.357 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T23:31:55.357 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:31:55.430 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-26T23:31:55.471 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-26T23:31:55.586 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 132/156 2024-04-26T23:31:55.657 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 133/156 2024-04-26T23:31:56.660 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-26T23:31:56.693 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-26T23:31:56.762 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-26T23:31:56.762 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:31:56.762 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-26T23:31:56.762 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T23:31:56.762 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T23:31:56.762 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:31:56.819 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-26T23:31:56.889 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-26T23:31:56.890 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:31:56.890 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-26T23:31:56.890 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:31:56.965 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-26T23:31:57.042 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-26T23:31:57.043 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:31:57.043 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-26T23:31:57.043 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T23:31:57.043 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T23:31:57.043 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:31:57.399 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-26T23:31:57.468 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-26T23:31:57.468 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T23:31:57.468 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-26T23:31:57.468 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T23:31:57.468 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T23:31:57.468 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:32:02.478 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 139/156 2024-04-26T23:32:02.533 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 140/156 2024-04-26T23:32:02.609 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 141/156 2024-04-26T23:32:02.657 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-26T23:32:02.706 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-26T23:32:03.027 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 144/156 2024-04-26T23:32:03.099 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-26T23:32:03.137 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 146/156 2024-04-26T23:32:04.055 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T23:32:04.110 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T23:32:04.172 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T23:32:05.478 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 148/156 2024-04-26T23:32:05.615 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-26T23:32:05.697 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-26T23:32:05.858 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-26T23:32:05.909 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-26T23:32:05.909 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T23:32:06.053 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T23:32:06.101 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T23:32:06.101 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T23:32:06.250 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T23:32:06.382 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T23:32:06.672 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T23:32:06.672 INFO:teuthology.orchestra.run.smithi072.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-04-26T23:32:06.672 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:32:19.685 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 1/156 2024-04-26T23:32:19.686 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2/156 2024-04-26T23:32:19.686 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 3/156 2024-04-26T23:32:19.686 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 4/156 2024-04-26T23:32:19.686 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 5/156 2024-04-26T23:32:19.686 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 6/156 2024-04-26T23:32:19.686 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 7/156 2024-04-26T23:32:19.686 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T23:32:19.686 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T23:32:19.686 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-26T23:32:19.686 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 11/156 2024-04-26T23:32:19.686 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 12/156 2024-04-26T23:32:19.686 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T23:32:19.687 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 14/156 2024-04-26T23:32:19.687 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 15/156 2024-04-26T23:32:19.687 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 16/156 2024-04-26T23:32:19.687 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 17/156 2024-04-26T23:32:19.687 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 18/156 2024-04-26T23:32:19.687 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 19/156 2024-04-26T23:32:19.687 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 20/156 2024-04-26T23:32:19.687 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 21/156 2024-04-26T23:32:19.687 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 22/156 2024-04-26T23:32:19.687 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 23/156 2024-04-26T23:32:19.687 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 24/156 2024-04-26T23:32:19.687 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 25/156 2024-04-26T23:32:19.687 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 26/156 2024-04-26T23:32:19.687 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 27/156 2024-04-26T23:32:19.688 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 28/156 2024-04-26T23:32:19.688 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 29/156 2024-04-26T23:32:19.688 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 30/156 2024-04-26T23:32:19.688 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 31/156 2024-04-26T23:32:19.688 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 32/156 2024-04-26T23:32:19.688 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 33/156 2024-04-26T23:32:19.688 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 34/156 2024-04-26T23:32:19.688 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 35/156 2024-04-26T23:32:19.688 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 36/156 2024-04-26T23:32:19.688 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 37/156 2024-04-26T23:32:19.688 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 38/156 2024-04-26T23:32:19.688 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 39/156 2024-04-26T23:32:19.688 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-26T23:32:19.688 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-26T23:32:19.689 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-26T23:32:19.689 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-26T23:32:19.689 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-26T23:32:19.689 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-26T23:32:19.689 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-26T23:32:19.689 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-26T23:32:19.689 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-26T23:32:19.689 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-26T23:32:19.689 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-26T23:32:19.689 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-26T23:32:19.689 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-26T23:32:19.689 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-26T23:32:19.689 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-26T23:32:19.689 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-26T23:32:19.689 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-26T23:32:19.690 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-26T23:32:19.690 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-26T23:32:19.690 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-26T23:32:19.690 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-26T23:32:19.690 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-26T23:32:19.690 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-26T23:32:19.690 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-26T23:32:19.690 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-26T23:32:19.690 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-26T23:32:19.690 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-26T23:32:19.690 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-26T23:32:19.690 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-26T23:32:19.691 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-26T23:32:19.691 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-26T23:32:19.691 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-26T23:32:19.691 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-26T23:32:19.691 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-26T23:32:19.691 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-26T23:32:19.691 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-26T23:32:19.691 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-26T23:32:19.691 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-26T23:32:19.691 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-26T23:32:19.691 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-26T23:32:19.691 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-26T23:32:19.691 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-26T23:32:19.691 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-26T23:32:19.692 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-26T23:32:19.692 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-26T23:32:19.692 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-26T23:32:19.692 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-26T23:32:19.692 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-26T23:32:19.692 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-26T23:32:19.692 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-26T23:32:19.692 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-26T23:32:19.692 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-26T23:32:19.692 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-26T23:32:19.692 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-26T23:32:19.692 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-26T23:32:19.692 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-26T23:32:19.692 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-26T23:32:19.692 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-26T23:32:19.692 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-26T23:32:19.693 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-26T23:32:19.693 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-26T23:32:19.693 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-26T23:32:19.693 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-26T23:32:19.693 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-26T23:32:19.693 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-26T23:32:19.693 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-26T23:32:19.693 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-26T23:32:19.693 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-26T23:32:19.693 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/156 2024-04-26T23:32:19.693 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/156 2024-04-26T23:32:19.693 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-26T23:32:19.693 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-26T23:32:19.693 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-26T23:32:19.693 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/156 2024-04-26T23:32:19.693 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-26T23:32:19.694 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-26T23:32:19.694 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-26T23:32:19.694 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-26T23:32:19.694 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T23:32:19.694 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T23:32:19.694 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-26T23:32:19.694 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-26T23:32:19.694 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-26T23:32:19.694 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-26T23:32:19.694 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-26T23:32:19.694 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-26T23:32:19.694 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-26T23:32:19.694 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-26T23:32:19.694 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-26T23:32:19.694 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-26T23:32:19.695 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-26T23:32:19.695 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-26T23:32:19.695 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-26T23:32:19.695 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-26T23:32:19.695 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-26T23:32:19.695 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-26T23:32:19.695 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-26T23:32:19.695 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-26T23:32:19.695 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-26T23:32:19.695 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-26T23:32:19.695 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-26T23:32:19.695 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-26T23:32:19.695 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-26T23:32:19.695 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-26T23:32:19.695 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-26T23:32:19.696 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-26T23:32:19.696 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-26T23:32:19.696 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-26T23:32:19.696 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-26T23:32:19.696 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 149/156 2024-04-26T23:32:19.696 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-26T23:32:19.696 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 151/156 2024-04-26T23:32:19.696 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-26T23:32:19.696 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-26T23:32:19.696 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T23:32:19.696 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-26T23:32:22.164 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T23:32:22.164 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:32:22.164 INFO:teuthology.orchestra.run.smithi072.stdout:Upgraded: 2024-04-26T23:32:22.164 INFO:teuthology.orchestra.run.smithi072.stdout: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:32:22.164 INFO:teuthology.orchestra.run.smithi072.stdout: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:32:22.165 INFO:teuthology.orchestra.run.smithi072.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-26T23:32:22.165 INFO:teuthology.orchestra.run.smithi072.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-26T23:32:22.165 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-04-26T23:32:22.165 INFO:teuthology.orchestra.run.smithi072.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-26T23:32:22.165 INFO:teuthology.orchestra.run.smithi072.stdout: automake-1.16.2-8.el9.noarch 2024-04-26T23:32:22.165 INFO:teuthology.orchestra.run.smithi072.stdout: bison-3.7.4-5.el9.x86_64 2024-04-26T23:32:22.165 INFO:teuthology.orchestra.run.smithi072.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-26T23:32:22.165 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:32:22.165 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:32:22.165 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:32:22.165 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:32:22.165 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:32:22.165 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-immutable-object-cache-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:32:22.165 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:32:22.166 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:32:22.166 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:32:22.166 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:32:22.166 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:32:22.166 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:32:22.166 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:32:22.166 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:32:22.166 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:32:22.166 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:32:22.166 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:32:22.166 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:32:22.166 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:32:22.166 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:32:22.166 INFO:teuthology.orchestra.run.smithi072.stdout: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:32:22.167 INFO:teuthology.orchestra.run.smithi072.stdout: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:32:22.167 INFO:teuthology.orchestra.run.smithi072.stdout: cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:32:22.167 INFO:teuthology.orchestra.run.smithi072.stdout: cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T23:32:22.167 INFO:teuthology.orchestra.run.smithi072.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-26T23:32:22.167 INFO:teuthology.orchestra.run.smithi072.stdout: flex-2.6.4-9.el9.x86_64 2024-04-26T23:32:22.167 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-26T23:32:22.167 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-26T23:32:22.167 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-26T23:32:22.167 INFO:teuthology.orchestra.run.smithi072.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-26T23:32:22.167 INFO:teuthology.orchestra.run.smithi072.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-26T23:32:22.167 INFO:teuthology.orchestra.run.smithi072.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-26T23:32:22.167 INFO:teuthology.orchestra.run.smithi072.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-26T23:32:22.167 INFO:teuthology.orchestra.run.smithi072.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-26T23:32:22.168 INFO:teuthology.orchestra.run.smithi072.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-26T23:32:22.168 INFO:teuthology.orchestra.run.smithi072.stdout: libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:32:22.168 INFO:teuthology.orchestra.run.smithi072.stdout: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:32:22.168 INFO:teuthology.orchestra.run.smithi072.stdout: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:32:22.168 INFO:teuthology.orchestra.run.smithi072.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-26T23:32:22.168 INFO:teuthology.orchestra.run.smithi072.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-26T23:32:22.168 INFO:teuthology.orchestra.run.smithi072.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-26T23:32:22.168 INFO:teuthology.orchestra.run.smithi072.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-26T23:32:22.168 INFO:teuthology.orchestra.run.smithi072.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-26T23:32:22.168 INFO:teuthology.orchestra.run.smithi072.stdout: librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:32:22.168 INFO:teuthology.orchestra.run.smithi072.stdout: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:32:22.168 INFO:teuthology.orchestra.run.smithi072.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-26T23:32:22.168 INFO:teuthology.orchestra.run.smithi072.stdout: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:32:22.168 INFO:teuthology.orchestra.run.smithi072.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-26T23:32:22.169 INFO:teuthology.orchestra.run.smithi072.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-26T23:32:22.169 INFO:teuthology.orchestra.run.smithi072.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-26T23:32:22.169 INFO:teuthology.orchestra.run.smithi072.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-26T23:32:22.169 INFO:teuthology.orchestra.run.smithi072.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-26T23:32:22.169 INFO:teuthology.orchestra.run.smithi072.stdout: lua-5.4.4-4.el9.x86_64 2024-04-26T23:32:22.169 INFO:teuthology.orchestra.run.smithi072.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-26T23:32:22.169 INFO:teuthology.orchestra.run.smithi072.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-26T23:32:22.169 INFO:teuthology.orchestra.run.smithi072.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-26T23:32:22.169 INFO:teuthology.orchestra.run.smithi072.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-26T23:32:22.169 INFO:teuthology.orchestra.run.smithi072.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-26T23:32:22.169 INFO:teuthology.orchestra.run.smithi072.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-26T23:32:22.169 INFO:teuthology.orchestra.run.smithi072.stdout: postgresql-13.14-1.el9.x86_64 2024-04-26T23:32:22.169 INFO:teuthology.orchestra.run.smithi072.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-26T23:32:22.169 INFO:teuthology.orchestra.run.smithi072.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-26T23:32:22.169 INFO:teuthology.orchestra.run.smithi072.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-26T23:32:22.170 INFO:teuthology.orchestra.run.smithi072.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-26T23:32:22.170 INFO:teuthology.orchestra.run.smithi072.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-26T23:32:22.170 INFO:teuthology.orchestra.run.smithi072.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-26T23:32:22.170 INFO:teuthology.orchestra.run.smithi072.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-26T23:32:22.170 INFO:teuthology.orchestra.run.smithi072.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-26T23:32:22.170 INFO:teuthology.orchestra.run.smithi072.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-26T23:32:22.170 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-26T23:32:22.170 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:32:22.170 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:32:22.170 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:32:22.170 INFO:teuthology.orchestra.run.smithi072.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-26T23:32:22.171 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-26T23:32:22.171 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-26T23:32:22.171 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-26T23:32:22.171 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-26T23:32:22.171 INFO:teuthology.orchestra.run.smithi072.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-26T23:32:22.171 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-26T23:32:22.171 INFO:teuthology.orchestra.run.smithi072.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-26T23:32:22.171 INFO:teuthology.orchestra.run.smithi072.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-26T23:32:22.171 INFO:teuthology.orchestra.run.smithi072.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-26T23:32:22.171 INFO:teuthology.orchestra.run.smithi072.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-26T23:32:22.171 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-26T23:32:22.171 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-26T23:32:22.171 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-26T23:32:22.171 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-26T23:32:22.171 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-26T23:32:22.172 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-26T23:32:22.172 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-26T23:32:22.172 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-26T23:32:22.172 INFO:teuthology.orchestra.run.smithi072.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-26T23:32:22.172 INFO:teuthology.orchestra.run.smithi072.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-26T23:32:22.172 INFO:teuthology.orchestra.run.smithi072.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-26T23:32:22.172 INFO:teuthology.orchestra.run.smithi072.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-26T23:32:22.172 INFO:teuthology.orchestra.run.smithi072.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-26T23:32:22.172 INFO:teuthology.orchestra.run.smithi072.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-26T23:32:22.172 INFO:teuthology.orchestra.run.smithi072.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-26T23:32:22.172 INFO:teuthology.orchestra.run.smithi072.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-26T23:32:22.172 INFO:teuthology.orchestra.run.smithi072.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-26T23:32:22.172 INFO:teuthology.orchestra.run.smithi072.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-26T23:32:22.172 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-26T23:32:22.173 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ply-3.11-14.el9.noarch 2024-04-26T23:32:22.173 INFO:teuthology.orchestra.run.smithi072.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-26T23:32:22.173 INFO:teuthology.orchestra.run.smithi072.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-26T23:32:22.173 INFO:teuthology.orchestra.run.smithi072.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-26T23:32:22.173 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-26T23:32:22.173 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-26T23:32:22.173 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-26T23:32:22.173 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-26T23:32:22.173 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-26T23:32:22.173 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-26T23:32:22.173 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:32:22.173 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:32:22.173 INFO:teuthology.orchestra.run.smithi072.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-26T23:32:22.173 INFO:teuthology.orchestra.run.smithi072.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-26T23:32:22.173 INFO:teuthology.orchestra.run.smithi072.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-26T23:32:22.174 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:32:22.174 INFO:teuthology.orchestra.run.smithi072.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-26T23:32:22.174 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-26T23:32:22.174 INFO:teuthology.orchestra.run.smithi072.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-26T23:32:22.174 INFO:teuthology.orchestra.run.smithi072.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-26T23:32:22.174 INFO:teuthology.orchestra.run.smithi072.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-26T23:32:22.174 INFO:teuthology.orchestra.run.smithi072.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-26T23:32:22.174 INFO:teuthology.orchestra.run.smithi072.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-26T23:32:22.174 INFO:teuthology.orchestra.run.smithi072.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-26T23:32:22.174 INFO:teuthology.orchestra.run.smithi072.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-26T23:32:22.174 INFO:teuthology.orchestra.run.smithi072.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-26T23:32:22.174 INFO:teuthology.orchestra.run.smithi072.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-26T23:32:22.174 INFO:teuthology.orchestra.run.smithi072.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-26T23:32:22.175 INFO:teuthology.orchestra.run.smithi072.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-26T23:32:22.175 INFO:teuthology.orchestra.run.smithi072.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-26T23:32:22.175 INFO:teuthology.orchestra.run.smithi072.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-26T23:32:22.175 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:32:22.175 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:32:22.175 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T23:32:22.175 INFO:teuthology.orchestra.run.smithi072.stdout: re2-1:20211101-3.el9.x86_64 2024-04-26T23:32:22.175 INFO:teuthology.orchestra.run.smithi072.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-26T23:32:22.175 INFO:teuthology.orchestra.run.smithi072.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-26T23:32:22.175 INFO:teuthology.orchestra.run.smithi072.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-26T23:32:22.175 INFO:teuthology.orchestra.run.smithi072.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-26T23:32:22.175 INFO:teuthology.orchestra.run.smithi072.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-26T23:32:22.176 INFO:teuthology.orchestra.run.smithi072.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-26T23:32:22.176 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:32:22.176 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-04-26T23:32:22.868 DEBUG:teuthology.parallel:result is None 2024-04-26T23:32:22.869 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T23:32:23.008 DEBUG:teuthology.orchestra.run.smithi072:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-26T23:32:23.061 INFO:teuthology.orchestra.run.smithi072.stdout:19.0.0-3306.gbc612ffb.el9 2024-04-26T23:32:23.062 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3306.gbc612ffb.el9 2024-04-26T23:32:23.062 INFO:teuthology.task.install:The correct ceph version 19.0.0-3306.gbc612ffb is installed. 2024-04-26T23:32:23.064 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T23:32:23.208 DEBUG:teuthology.orchestra.run.smithi076:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-26T23:32:23.234 INFO:teuthology.orchestra.run.smithi076.stdout:19.0.0-3306.gbc612ffb.el9 2024-04-26T23:32:23.235 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3306.gbc612ffb.el9 2024-04-26T23:32:23.235 INFO:teuthology.task.install:The correct ceph version 19.0.0-3306.gbc612ffb is installed. 2024-04-26T23:32:23.238 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T23:32:23.388 DEBUG:teuthology.orchestra.run.smithi136:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-26T23:32:23.417 INFO:teuthology.orchestra.run.smithi136.stdout:19.0.0-3306.gbc612ffb.el9 2024-04-26T23:32:23.417 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3306.gbc612ffb.el9 2024-04-26T23:32:23.418 INFO:teuthology.task.install:The correct ceph version 19.0.0-3306.gbc612ffb is installed. 2024-04-26T23:32:23.421 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-26T23:32:23.421 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-26T23:32:23.421 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T23:32:23.502 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-04-26T23:32:23.502 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T23:32:23.532 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-26T23:32:23.533 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T23:32:23.564 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-26T23:32:23.564 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-26T23:32:23.564 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T23:32:23.638 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T23:32:23.749 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-04-26T23:32:23.749 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T23:32:23.777 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T23:32:23.847 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-26T23:32:23.847 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T23:32:23.877 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T23:32:23.944 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-26T23:32:23.944 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-26T23:32:23.944 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T23:32:24.012 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T23:32:24.116 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-04-26T23:32:24.117 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T23:32:24.142 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T23:32:24.206 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-26T23:32:24.206 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T23:32:24.230 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T23:32:24.294 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-26T23:32:24.295 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-26T23:32:24.295 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T23:32:24.358 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T23:32:24.458 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-04-26T23:32:24.459 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T23:32:24.482 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T23:32:24.546 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-26T23:32:24.546 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T23:32:24.570 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T23:32:24.632 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-26T23:32:24.737 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'subvols': {'create': 2}} 2024-04-26T23:32:24.737 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T23:32:24.738 INFO:tasks.cephadm:Cluster fsid is 3c8453ae-0425-11ef-bc93-c7b262605968 2024-04-26T23:32:24.738 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-26T23:32:24.738 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.72', 'mon.b': '172.21.15.76', 'mon.c': '172.21.15.136'} 2024-04-26T23:32:24.738 INFO:tasks.cephadm:First mon is mon.a on smithi072 2024-04-26T23:32:24.738 INFO:tasks.cephadm:First mgr is x 2024-04-26T23:32:24.738 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-26T23:32:24.739 DEBUG:teuthology.orchestra.run.smithi072:> sudo hostname $(hostname -s) 2024-04-26T23:32:24.799 DEBUG:teuthology.orchestra.run.smithi076:> sudo hostname $(hostname -s) 2024-04-26T23:32:24.822 DEBUG:teuthology.orchestra.run.smithi136:> sudo hostname $(hostname -s) 2024-04-26T23:32:24.846 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-26T23:32:24.847 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T23:32:24.982 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78918/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3306-gbc612ffb', 'node_name': '172.21.2.14+braggi14', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3306.gbc612ffb'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-26 13:02:42.256491', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-rishabh-testing-20240426.111959', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-04-26T23:32:25.117 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-rishabh-testing-20240426.111959, sha1 bc612ffb9c1d5bc27536bc917a705262a36d4387 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T23:32:25.119 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/x86_64/flavors/default/cephadm 2024-04-26T23:32:25.119 DEBUG:teuthology.orchestra.run.smithi072:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-26T23:32:25.716 INFO:teuthology.orchestra.run.smithi072.stdout:-rw-r--r--. 1 ubuntu ubuntu 781514 Apr 26 23:32 /home/ubuntu/cephtest/cephadm 2024-04-26T23:32:25.717 DEBUG:teuthology.orchestra.run.smithi076:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-26T23:32:26.342 INFO:teuthology.orchestra.run.smithi076.stdout:-rw-r--r--. 1 ubuntu ubuntu 781514 Apr 26 23:32 /home/ubuntu/cephtest/cephadm 2024-04-26T23:32:26.342 DEBUG:teuthology.orchestra.run.smithi136:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-26T23:32:26.783 INFO:teuthology.orchestra.run.smithi136.stdout:-rw-r--r--. 1 ubuntu ubuntu 781514 Apr 26 23:32 /home/ubuntu/cephtest/cephadm 2024-04-26T23:32:26.783 DEBUG:teuthology.orchestra.run.smithi072:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-26T23:32:26.835 DEBUG:teuthology.orchestra.run.smithi076:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-26T23:32:26.853 DEBUG:teuthology.orchestra.run.smithi136:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-26T23:32:26.888 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 on all hosts... 2024-04-26T23:32:26.888 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 pull 2024-04-26T23:32:26.893 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 pull 2024-04-26T23:32:26.896 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 pull 2024-04-26T23:32:28.379 INFO:teuthology.orchestra.run.smithi076.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387... 2024-04-26T23:32:28.469 INFO:teuthology.orchestra.run.smithi136.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387... 2024-04-26T23:32:30.606 INFO:teuthology.orchestra.run.smithi072.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387... 2024-04-26T23:33:01.859 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-04-26T23:33:01.860 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph_version": "ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev)", 2024-04-26T23:33:01.860 INFO:teuthology.orchestra.run.smithi136.stdout: "image_id": "55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07", 2024-04-26T23:33:01.860 INFO:teuthology.orchestra.run.smithi136.stdout: "repo_digests": [ 2024-04-26T23:33:01.860 INFO:teuthology.orchestra.run.smithi136.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2" 2024-04-26T23:33:01.860 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-04-26T23:33:01.860 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-04-26T23:33:04.362 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2024-04-26T23:33:04.363 INFO:teuthology.orchestra.run.smithi076.stdout: "ceph_version": "ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev)", 2024-04-26T23:33:04.363 INFO:teuthology.orchestra.run.smithi076.stdout: "image_id": "55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07", 2024-04-26T23:33:04.363 INFO:teuthology.orchestra.run.smithi076.stdout: "repo_digests": [ 2024-04-26T23:33:04.363 INFO:teuthology.orchestra.run.smithi076.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2" 2024-04-26T23:33:04.363 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2024-04-26T23:33:04.363 INFO:teuthology.orchestra.run.smithi076.stdout:} 2024-04-26T23:33:22.275 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-04-26T23:33:22.275 INFO:teuthology.orchestra.run.smithi072.stdout: "ceph_version": "ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev)", 2024-04-26T23:33:22.275 INFO:teuthology.orchestra.run.smithi072.stdout: "image_id": "55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07", 2024-04-26T23:33:22.275 INFO:teuthology.orchestra.run.smithi072.stdout: "repo_digests": [ 2024-04-26T23:33:22.275 INFO:teuthology.orchestra.run.smithi072.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2" 2024-04-26T23:33:22.275 INFO:teuthology.orchestra.run.smithi072.stdout: ] 2024-04-26T23:33:22.275 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-04-26T23:33:22.320 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /etc/ceph 2024-04-26T23:33:22.408 DEBUG:teuthology.orchestra.run.smithi076:> sudo mkdir -p /etc/ceph 2024-04-26T23:33:22.457 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /etc/ceph 2024-04-26T23:33:22.491 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 777 /etc/ceph 2024-04-26T23:33:22.578 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 777 /etc/ceph 2024-04-26T23:33:22.605 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 777 /etc/ceph 2024-04-26T23:33:22.632 INFO:tasks.cephadm:Writing seed config... 2024-04-26T23:33:22.633 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-26T23:33:22.633 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-26T23:33:22.634 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-26T23:33:22.634 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-26T23:33:22.634 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-26T23:33:22.634 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-04-26T23:33:22.634 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-26T23:33:22.634 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-26T23:33:22.634 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-26T23:33:22.634 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-26T23:33:22.634 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-26T23:33:22.634 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-26T23:33:22.634 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-26T23:33:22.634 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-26T23:33:22.634 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-26T23:33:22.634 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-26T23:33:22.635 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-26T23:33:22.635 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-04-26T23:33:22.635 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-26T23:33:22.635 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-26T23:33:22.635 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-26T23:33:22.635 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-26T23:33:22.635 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-26T23:33:22.635 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-26T23:33:22.635 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-26T23:33:22.635 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-26T23:33:22.635 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-26T23:33:22.635 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-26T23:33:22.635 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-26T23:33:22.635 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-26T23:33:22.635 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-26T23:33:22.635 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-04-26T23:33:22.636 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-26T23:33:22.636 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-04-26T23:33:22.636 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-04-26T23:33:22.636 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-26T23:33:22.636 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-26T23:33:22.636 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-04-26T23:33:22.636 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-26T23:33:22.636 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-26T23:33:22.636 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-26T23:33:22.636 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-26T23:33:22.636 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-26T23:33:22.636 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-26T23:33:22.636 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-26T23:33:22.636 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-04-26T23:33:22.637 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-26T23:33:22.637 DEBUG:teuthology.orchestra.run.smithi072:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-26T23:33:22.695 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 3c8453ae-0425-11ef-bc93-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-26T23:33:22.695 DEBUG:teuthology.orchestra.run.smithi072:mon.a> sudo journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mon.a.service 2024-04-26T23:33:22.743 DEBUG:teuthology.orchestra.run.smithi072:mgr.x> sudo journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mgr.x.service 2024-04-26T23:33:22.788 INFO:tasks.cephadm:Bootstrapping... 2024-04-26T23:33:22.788 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 -v bootstrap --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.72 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-26T23:33:23.052 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-04-26T23:33:23.052 INFO:teuthology.orchestra.run.smithi072.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387', '-v', 'bootstrap', '--fsid', '3c8453ae-0425-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.72', '--skip-admin-label'] 2024-04-26T23:33:23.052 INFO:teuthology.orchestra.run.smithi072.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-26T23:33:23.053 INFO:teuthology.orchestra.run.smithi072.stdout:Verifying podman|docker is present... 2024-04-26T23:33:23.108 INFO:teuthology.orchestra.run.smithi072.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-26T23:33:23.109 INFO:teuthology.orchestra.run.smithi072.stdout:Verifying lvm2 is present... 2024-04-26T23:33:23.109 INFO:teuthology.orchestra.run.smithi072.stdout:Verifying time synchronization is in place... 2024-04-26T23:33:23.124 INFO:teuthology.orchestra.run.smithi072.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-26T23:33:23.124 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-26T23:33:23.140 INFO:teuthology.orchestra.run.smithi072.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-26T23:33:23.140 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stdout inactive 2024-04-26T23:33:23.163 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stdout enabled 2024-04-26T23:33:23.183 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stdout active 2024-04-26T23:33:23.183 INFO:teuthology.orchestra.run.smithi072.stdout:Unit chronyd.service is enabled and running 2024-04-26T23:33:23.183 INFO:teuthology.orchestra.run.smithi072.stdout:Repeating the final host check... 2024-04-26T23:33:23.242 INFO:teuthology.orchestra.run.smithi072.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-26T23:33:23.242 INFO:teuthology.orchestra.run.smithi072.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-26T23:33:23.242 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl is present 2024-04-26T23:33:23.242 INFO:teuthology.orchestra.run.smithi072.stdout:lvcreate is present 2024-04-26T23:33:23.256 INFO:teuthology.orchestra.run.smithi072.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-26T23:33:23.256 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-26T23:33:23.271 INFO:teuthology.orchestra.run.smithi072.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-26T23:33:23.271 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stdout inactive 2024-04-26T23:33:23.293 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stdout enabled 2024-04-26T23:33:23.310 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stdout active 2024-04-26T23:33:23.311 INFO:teuthology.orchestra.run.smithi072.stdout:Unit chronyd.service is enabled and running 2024-04-26T23:33:23.311 INFO:teuthology.orchestra.run.smithi072.stdout:Host looks OK 2024-04-26T23:33:23.311 INFO:teuthology.orchestra.run.smithi072.stdout:Cluster fsid: 3c8453ae-0425-11ef-bc93-c7b262605968 2024-04-26T23:33:23.311 INFO:teuthology.orchestra.run.smithi072.stdout:Acquiring lock 140163793104032 on /run/cephadm/3c8453ae-0425-11ef-bc93-c7b262605968.lock 2024-04-26T23:33:23.311 INFO:teuthology.orchestra.run.smithi072.stdout:Lock 140163793104032 acquired on /run/cephadm/3c8453ae-0425-11ef-bc93-c7b262605968.lock 2024-04-26T23:33:23.311 INFO:teuthology.orchestra.run.smithi072.stdout:Verifying IP 172.21.15.72 port 3300 ... 2024-04-26T23:33:23.312 INFO:teuthology.orchestra.run.smithi072.stdout:Verifying IP 172.21.15.72 port 6789 ... 2024-04-26T23:33:23.312 INFO:teuthology.orchestra.run.smithi072.stdout:Base mon IP(s) is [172.21.15.72:3300, 172.21.15.72:6789], mon addrv is [v2:172.21.15.72:3300,v1:172.21.15.72:6789] 2024-04-26T23:33:23.320 INFO:teuthology.orchestra.run.smithi072.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.72 metric 100 2024-04-26T23:33:23.320 INFO:teuthology.orchestra.run.smithi072.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.72 metric 100 2024-04-26T23:33:23.326 INFO:teuthology.orchestra.run.smithi072.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-26T23:33:23.327 INFO:teuthology.orchestra.run.smithi072.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-04-26T23:33:23.327 INFO:teuthology.orchestra.run.smithi072.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1775sec hoplimit 64 pref medium 2024-04-26T23:33:23.333 INFO:teuthology.orchestra.run.smithi072.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-26T23:33:23.333 INFO:teuthology.orchestra.run.smithi072.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-26T23:33:23.333 INFO:teuthology.orchestra.run.smithi072.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-26T23:33:23.334 INFO:teuthology.orchestra.run.smithi072.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-04-26T23:33:23.334 INFO:teuthology.orchestra.run.smithi072.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:b765/64 scope link 2024-04-26T23:33:23.334 INFO:teuthology.orchestra.run.smithi072.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-26T23:33:23.334 INFO:teuthology.orchestra.run.smithi072.stdout:Mon IP `172.21.15.72` is in CIDR network `172.21.0.0/20` 2024-04-26T23:33:23.334 INFO:teuthology.orchestra.run.smithi072.stdout:Mon IP `172.21.15.72` is in CIDR network `172.21.0.0/20` 2024-04-26T23:33:23.334 INFO:teuthology.orchestra.run.smithi072.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-26T23:33:23.335 INFO:teuthology.orchestra.run.smithi072.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-26T23:33:23.336 INFO:teuthology.orchestra.run.smithi072.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387... 2024-04-26T23:33:24.050 INFO:teuthology.orchestra.run.smithi072.stdout:/bin/podman: stdout 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 2024-04-26T23:33:24.050 INFO:teuthology.orchestra.run.smithi072.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387... 2024-04-26T23:33:24.050 INFO:teuthology.orchestra.run.smithi072.stdout:/bin/podman: stderr Getting image source signatures 2024-04-26T23:33:24.050 INFO:teuthology.orchestra.run.smithi072.stdout:/bin/podman: stderr Copying blob sha256:3d665f4c032644be9cae748af04bee69f2bfe96cdc3616dabe0ab17899633ab5 2024-04-26T23:33:24.050 INFO:teuthology.orchestra.run.smithi072.stdout:/bin/podman: stderr Copying blob sha256:f340d36a00bce3feed0ad509aeafc9092c1c625bf553b8a613c380a79149b87e 2024-04-26T23:33:24.050 INFO:teuthology.orchestra.run.smithi072.stdout:/bin/podman: stderr Copying config sha256:55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 2024-04-26T23:33:24.050 INFO:teuthology.orchestra.run.smithi072.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-26T23:33:25.335 INFO:teuthology.orchestra.run.smithi072.stdout:ceph: stdout ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev) 2024-04-26T23:33:25.335 INFO:teuthology.orchestra.run.smithi072.stdout:Ceph version: ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev) 2024-04-26T23:33:25.335 INFO:teuthology.orchestra.run.smithi072.stdout:Extracting ceph user uid/gid from container image... 2024-04-26T23:33:26.464 INFO:teuthology.orchestra.run.smithi072.stdout:stat: stdout 167 167 2024-04-26T23:33:26.464 INFO:teuthology.orchestra.run.smithi072.stdout:Creating initial keys... 2024-04-26T23:33:27.560 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph-authtool: stdout AQDGOSxmu+pwNxAAn+2mKgSda+DTK+7dVR4O8A== 2024-04-26T23:33:28.712 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph-authtool: stdout AQDIOSxmBCOuAhAA10c1HhqaGio1zixF4A5ZHg== 2024-04-26T23:33:29.865 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph-authtool: stdout AQDJOSxm97RSDBAAj6T3uQ4urayz/zGJnjIgFA== 2024-04-26T23:33:29.866 INFO:teuthology.orchestra.run.smithi072.stdout:Creating initial monmap... 2024-04-26T23:33:30.982 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-26T23:33:30.982 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-26T23:33:30.982 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 3c8453ae-0425-11ef-bc93-c7b262605968 2024-04-26T23:33:30.982 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-26T23:33:30.982 INFO:teuthology.orchestra.run.smithi072.stdout:monmaptool for a [v2:172.21.15.72:3300,v1:172.21.15.72:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-26T23:33:30.982 INFO:teuthology.orchestra.run.smithi072.stdout:setting min_mon_release = quincy 2024-04-26T23:33:30.982 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/monmaptool: set fsid to 3c8453ae-0425-11ef-bc93-c7b262605968 2024-04-26T23:33:30.983 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-26T23:33:30.983 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:33:30.983 INFO:teuthology.orchestra.run.smithi072.stdout:Creating mon... 2024-04-26T23:33:32.136 INFO:teuthology.orchestra.run.smithi072.stdout:create mon.a on 2024-04-26T23:33:32.534 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-26T23:33:32.769 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-26T23:33:33.067 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-3c8453ae-0425-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-3c8453ae-0425-11ef-bc93-c7b262605968.target. 2024-04-26T23:33:33.067 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-3c8453ae-0425-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-3c8453ae-0425-11ef-bc93-c7b262605968.target. 2024-04-26T23:33:33.492 INFO:teuthology.orchestra.run.smithi072.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mon.a 2024-04-26T23:33:33.492 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stderr Failed to reset failed state of unit ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-26T23:33:33.796 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-3c8453ae-0425-11ef-bc93-c7b262605968.target.wants/ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-3c8453ae-0425-11ef-bc93-c7b262605968@.service. 2024-04-26T23:33:34.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:34 smithi072 podman[27788]: 2024-04-26 23:33:34.139842854 +0000 UTC m=+0.037470187 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T23:33:48.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:48 smithi072 podman[27788]: 2024-04-26 23:33:48.000613953 +0000 UTC m=+13.898241350 container init b0847df67adeb4922cafad3260e16fa60056d4e6903883ca1807e322a4e86eb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mon-a, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=) 2024-04-26T23:33:48.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:48 smithi072 podman[27788]: 2024-04-26 23:33:48.011992174 +0000 UTC m=+13.909619541 container start b0847df67adeb4922cafad3260e16fa60056d4e6903883ca1807e322a4e86eb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mon-a, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-26T23:33:48.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:48 smithi072 ceph-mon[27806]: mkfs 3c8453ae-0425-11ef-bc93-c7b262605968 2024-04-26T23:33:48.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:48 smithi072 ceph-mon[27806]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-26T23:33:49.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:49 smithi072 ceph-mon[27806]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-26T23:33:49.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:49 smithi072 ceph-mon[27806]: monmap epoch 1 2024-04-26T23:33:49.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:49 smithi072 ceph-mon[27806]: fsid 3c8453ae-0425-11ef-bc93-c7b262605968 2024-04-26T23:33:49.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:49 smithi072 ceph-mon[27806]: last_changed 2024-04-26T23:33:30.337416+0000 2024-04-26T23:33:49.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:49 smithi072 ceph-mon[27806]: created 2024-04-26T23:33:30.337416+0000 2024-04-26T23:33:49.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:49 smithi072 ceph-mon[27806]: min_mon_release 19 (squid) 2024-04-26T23:33:49.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:49 smithi072 ceph-mon[27806]: election_strategy: 1 2024-04-26T23:33:49.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:49 smithi072 ceph-mon[27806]: 0: [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] mon.a 2024-04-26T23:33:49.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:49 smithi072 ceph-mon[27806]: fsmap 2024-04-26T23:33:49.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:49 smithi072 ceph-mon[27806]: osdmap e1: 0 total, 0 up, 0 in 2024-04-26T23:33:49.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:49 smithi072 ceph-mon[27806]: mgrmap e1: no daemons active 2024-04-26T23:33:50.707 INFO:teuthology.orchestra.run.smithi072.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T23:33:50.707 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stdout disabled 2024-04-26T23:33:50.734 INFO:teuthology.orchestra.run.smithi072.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T23:33:50.734 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stdout inactive 2024-04-26T23:33:50.734 INFO:teuthology.orchestra.run.smithi072.stdout:firewalld.service is not enabled 2024-04-26T23:33:50.734 INFO:teuthology.orchestra.run.smithi072.stdout:Not possible to enable service . firewalld.service is not available 2024-04-26T23:33:50.736 INFO:teuthology.orchestra.run.smithi072.stdout:Waiting for mon to start... 2024-04-26T23:33:50.736 INFO:teuthology.orchestra.run.smithi072.stdout:Waiting for mon... 2024-04-26T23:33:50.896 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:50 smithi072 bash[27788]: b0847df67adeb4922cafad3260e16fa60056d4e6903883ca1807e322a4e86eb8 2024-04-26T23:33:50.896 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:50 smithi072 systemd[1]: Started Ceph mon.a for 3c8453ae-0425-11ef-bc93-c7b262605968. 2024-04-26T23:33:51.766 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:51 smithi072 ceph-mon[27806]: from='client.? 172.21.15.72:0/312429783' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-26T23:33:52.101 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout cluster: 2024-04-26T23:33:52.102 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout id: 3c8453ae-0425-11ef-bc93-c7b262605968 2024-04-26T23:33:52.102 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-26T23:33:52.102 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 2024-04-26T23:33:52.102 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout services: 2024-04-26T23:33:52.102 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 3s) 2024-04-26T23:33:52.102 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-26T23:33:52.102 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-26T23:33:52.108 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 2024-04-26T23:33:52.108 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout data: 2024-04-26T23:33:52.108 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-26T23:33:52.108 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-26T23:33:52.108 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-26T23:33:52.108 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout pgs: 2024-04-26T23:33:52.108 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 2024-04-26T23:33:52.108 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:51.333+0000 7fb465600700 1 Processor -- start 2024-04-26T23:33:52.108 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:51.333+0000 7fb465600700 1 -- start start 2024-04-26T23:33:52.108 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:51.334+0000 7fb465600700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb46007ee70 0x7fb46007d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:33:52.108 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:51.334+0000 7fb465600700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fb46007d8a0 con 0x7fb46007ee70 2024-04-26T23:33:52.108 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:51.335+0000 7fb45ea00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb46007ee70 0x7fb46007d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:33:52.109 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:51.335+0000 7fb45ea00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb46007ee70 0x7fb46007d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56464/0 (socket says 172.21.15.72:56464) 2024-04-26T23:33:52.109 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:51.335+0000 7fb45ea00700 1 -- 172.21.15.72:0/3392957483 learned_addr learned my addr 172.21.15.72:0/3392957483 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:33:52.109 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:51.336+0000 7fb45ea00700 1 -- 172.21.15.72:0/3392957483 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb46007da20 con 0x7fb46007ee70 2024-04-26T23:33:52.109 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:51.336+0000 7fb45ea00700 1 --2- 172.21.15.72:0/3392957483 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb46007ee70 0x7fb46007d2d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb4500098a0 tx=0x7fb450009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90ce170c51606f31 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:33:52.109 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:51.336+0000 7fb45d600700 1 -- 172.21.15.72:0/3392957483 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb450004160 con 0x7fb46007ee70 2024-04-26T23:33:52.109 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:51.336+0000 7fb45d600700 1 -- 172.21.15.72:0/3392957483 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb450004300 con 0x7fb46007ee70 2024-04-26T23:33:52.109 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:51.336+0000 7fb45d600700 1 -- 172.21.15.72:0/3392957483 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb4500036a0 con 0x7fb46007ee70 2024-04-26T23:33:52.109 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:51.337+0000 7fb465600700 1 -- 172.21.15.72:0/3392957483 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb46007ee70 msgr2=0x7fb46007d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:33:52.109 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:51.337+0000 7fb465600700 1 --2- 172.21.15.72:0/3392957483 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb46007ee70 0x7fb46007d2d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb4500098a0 tx=0x7fb450009bb0 comp rx=0 tx=0).stop 2024-04-26T23:33:52.109 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:51.337+0000 7fb465600700 1 -- 172.21.15.72:0/3392957483 shutdown_connections 2024-04-26T23:33:52.109 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:51.337+0000 7fb465600700 1 --2- 172.21.15.72:0/3392957483 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb46007ee70 0x7fb46007d2d0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:33:52.109 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:51.337+0000 7fb465600700 1 -- 172.21.15.72:0/3392957483 >> 172.21.15.72:0/3392957483 conn(0x7fb4601042d0 msgr2=0x7fb460106750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:33:52.109 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:51.337+0000 7fb465600700 1 -- 172.21.15.72:0/3392957483 shutdown_connections 2024-04-26T23:33:52.110 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:51.338+0000 7fb465600700 1 -- 172.21.15.72:0/3392957483 wait complete. 2024-04-26T23:33:52.110 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:51.339+0000 7fb465600700 1 Processor -- start 2024-04-26T23:33:52.110 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:51.339+0000 7fb465600700 1 -- start start 2024-04-26T23:33:52.110 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:51.339+0000 7fb465600700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb46007ee70 0x7fb46019e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:33:52.110 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:51.339+0000 7fb465600700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fb46007df50 con 0x7fb46007ee70 2024-04-26T23:33:52.110 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:51.339+0000 7fb45ea00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb46007ee70 0x7fb46019e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:33:52.110 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:51.339+0000 7fb45ea00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb46007ee70 0x7fb46019e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56480/0 (socket says 172.21.15.72:56480) 2024-04-26T23:33:52.110 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:51.339+0000 7fb45ea00700 1 -- 172.21.15.72:0/312429783 learned_addr learned my addr 172.21.15.72:0/312429783 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:33:52.110 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:51.340+0000 7fb45ea00700 1 -- 172.21.15.72:0/312429783 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb46019ebf0 con 0x7fb46007ee70 2024-04-26T23:33:52.110 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:51.340+0000 7fb45ea00700 1 --2- 172.21.15.72:0/312429783 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb46007ee70 0x7fb46019e6b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fb450000c00 tx=0x7fb450004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:33:52.110 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:51.340+0000 7fb457400700 1 -- 172.21.15.72:0/312429783 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb450003db0 con 0x7fb46007ee70 2024-04-26T23:33:52.110 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:51.340+0000 7fb457400700 1 -- 172.21.15.72:0/312429783 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb450011630 con 0x7fb46007ee70 2024-04-26T23:33:52.111 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:51.341+0000 7fb457400700 1 -- 172.21.15.72:0/312429783 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb450011950 con 0x7fb46007ee70 2024-04-26T23:33:52.111 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:51.341+0000 7fb465600700 1 -- 172.21.15.72:0/312429783 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb46019ee80 con 0x7fb46007ee70 2024-04-26T23:33:52.111 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:51.341+0000 7fb465600700 1 -- 172.21.15.72:0/312429783 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4601a1b20 con 0x7fb46007ee70 2024-04-26T23:33:52.111 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:51.341+0000 7fb457400700 1 -- 172.21.15.72:0/312429783 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb45001a460 con 0x7fb46007ee70 2024-04-26T23:33:52.111 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:51.341+0000 7fb457400700 1 -- 172.21.15.72:0/312429783 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb45001b780 con 0x7fb46007ee70 2024-04-26T23:33:52.111 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:51.342+0000 7fb465600700 1 -- 172.21.15.72:0/312429783 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb460064a80 con 0x7fb46007ee70 2024-04-26T23:33:52.111 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:51.345+0000 7fb457400700 1 -- 172.21.15.72:0/312429783 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fb45001ba20 con 0x7fb46007ee70 2024-04-26T23:33:52.111 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:51.425+0000 7fb465600700 1 -- 172.21.15.72:0/312429783 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7fb46002e620 con 0x7fb46007ee70 2024-04-26T23:33:52.111 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:51.426+0000 7fb457400700 1 -- 172.21.15.72:0/312429783 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+310 (secure 0 0 0) 0x7fb45001bc00 con 0x7fb46007ee70 2024-04-26T23:33:52.111 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:51.427+0000 7fb465600700 1 -- 172.21.15.72:0/312429783 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb46007ee70 msgr2=0x7fb46019e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:33:52.111 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:51.427+0000 7fb465600700 1 --2- 172.21.15.72:0/312429783 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb46007ee70 0x7fb46019e6b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fb450000c00 tx=0x7fb450004750 comp rx=0 tx=0).stop 2024-04-26T23:33:52.111 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:51.428+0000 7fb465600700 1 -- 172.21.15.72:0/312429783 shutdown_connections 2024-04-26T23:33:52.111 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:51.428+0000 7fb465600700 1 --2- 172.21.15.72:0/312429783 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb46007ee70 0x7fb46019e6b0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:33:52.111 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:51.428+0000 7fb465600700 1 -- 172.21.15.72:0/312429783 >> 172.21.15.72:0/312429783 conn(0x7fb4601042d0 msgr2=0x7fb460105210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:33:52.112 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:51.428+0000 7fb465600700 1 -- 172.21.15.72:0/312429783 shutdown_connections 2024-04-26T23:33:52.112 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:51.428+0000 7fb465600700 1 -- 172.21.15.72:0/312429783 wait complete. 2024-04-26T23:33:52.112 INFO:teuthology.orchestra.run.smithi072.stdout:mon is available 2024-04-26T23:33:52.112 INFO:teuthology.orchestra.run.smithi072.stdout:Assimilating anything we can from ceph.conf... 2024-04-26T23:33:53.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:52 smithi072 ceph-mon[27806]: from='client.? 172.21.15.72:0/2273881013' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-26T23:33:53.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:52 smithi072 ceph-mon[27806]: from='client.? 172.21.15.72:0/2273881013' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-26T23:33:53.395 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 2024-04-26T23:33:53.396 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout [global] 2024-04-26T23:33:53.396 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout fsid = 3c8453ae-0425-11ef-bc93-c7b262605968 2024-04-26T23:33:53.396 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-26T23:33:53.397 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.72:3300,v1:172.21.15.72:6789] 2024-04-26T23:33:53.398 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-26T23:33:53.398 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-26T23:33:53.398 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-26T23:33:53.398 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-26T23:33:53.398 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 2024-04-26T23:33:53.398 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-26T23:33:53.399 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-26T23:33:53.399 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 2024-04-26T23:33:53.399 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout [osd] 2024-04-26T23:33:53.400 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-26T23:33:53.400 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-26T23:33:53.400 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-26T23:33:53.400 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-26T23:33:53.400 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-26T23:33:53.400 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:52.678+0000 7f1993000700 1 Processor -- start 2024-04-26T23:33:53.400 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:52.679+0000 7f1993000700 1 -- start start 2024-04-26T23:33:53.402 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:52.679+0000 7f1993000700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f198c108d70 0x7f198c109190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:33:53.402 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:52.679+0000 7f1993000700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f198c109760 con 0x7f198c108d70 2024-04-26T23:33:53.402 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:52.680+0000 7f198be00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f198c108d70 0x7f198c109190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:33:53.402 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:52.680+0000 7f198be00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f198c108d70 0x7f198c109190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56492/0 (socket says 172.21.15.72:56492) 2024-04-26T23:33:53.402 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:52.680+0000 7f198be00700 1 -- 172.21.15.72:0/2718212543 learned_addr learned my addr 172.21.15.72:0/2718212543 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:33:53.402 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:52.680+0000 7f198be00700 1 -- 172.21.15.72:0/2718212543 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f198c109f70 con 0x7f198c108d70 2024-04-26T23:33:53.402 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:52.681+0000 7f198be00700 1 --2- 172.21.15.72:0/2718212543 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f198c108d70 0x7f198c109190 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f197c0098a0 tx=0x7f197c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=404284f41d44930b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:33:53.402 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:52.681+0000 7f198aa00700 1 -- 172.21.15.72:0/2718212543 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f197c004160 con 0x7f198c108d70 2024-04-26T23:33:53.402 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:52.681+0000 7f198aa00700 1 -- 172.21.15.72:0/2718212543 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f197c004300 con 0x7f198c108d70 2024-04-26T23:33:53.402 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:52.681+0000 7f198aa00700 1 -- 172.21.15.72:0/2718212543 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f197c0036a0 con 0x7f198c108d70 2024-04-26T23:33:53.402 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:52.682+0000 7f1993000700 1 -- 172.21.15.72:0/2718212543 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f198c108d70 msgr2=0x7f198c109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:33:53.403 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:52.682+0000 7f1993000700 1 --2- 172.21.15.72:0/2718212543 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f198c108d70 0x7f198c109190 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f197c0098a0 tx=0x7f197c009bb0 comp rx=0 tx=0).stop 2024-04-26T23:33:53.404 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:52.682+0000 7f1993000700 1 -- 172.21.15.72:0/2718212543 shutdown_connections 2024-04-26T23:33:53.404 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:52.682+0000 7f1993000700 1 --2- 172.21.15.72:0/2718212543 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f198c108d70 0x7f198c109190 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:33:53.404 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:52.682+0000 7f1993000700 1 -- 172.21.15.72:0/2718212543 >> 172.21.15.72:0/2718212543 conn(0x7f198c1042d0 msgr2=0x7f198c106750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:33:53.404 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:52.682+0000 7f1993000700 1 -- 172.21.15.72:0/2718212543 shutdown_connections 2024-04-26T23:33:53.405 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:52.683+0000 7f1993000700 1 -- 172.21.15.72:0/2718212543 wait complete. 2024-04-26T23:33:53.405 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:52.684+0000 7f1993000700 1 Processor -- start 2024-04-26T23:33:53.405 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:52.684+0000 7f1993000700 1 -- start start 2024-04-26T23:33:53.405 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:52.685+0000 7f1993000700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f198c108d70 0x7f198c19e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:33:53.405 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:52.685+0000 7f1993000700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f198c10a4a0 con 0x7f198c108d70 2024-04-26T23:33:53.405 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:52.685+0000 7f198be00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f198c108d70 0x7f198c19e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:33:53.405 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:52.685+0000 7f198be00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f198c108d70 0x7f198c19e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56498/0 (socket says 172.21.15.72:56498) 2024-04-26T23:33:53.405 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:52.685+0000 7f198be00700 1 -- 172.21.15.72:0/2273881013 learned_addr learned my addr 172.21.15.72:0/2273881013 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:33:53.405 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:52.685+0000 7f198be00700 1 -- 172.21.15.72:0/2273881013 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f198c19ec20 con 0x7f198c108d70 2024-04-26T23:33:53.408 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:52.686+0000 7f198be00700 1 --2- 172.21.15.72:0/2273881013 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f198c108d70 0x7f198c19e6e0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f197c000c00 tx=0x7f197c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:33:53.408 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:52.686+0000 7f1988c00700 1 -- 172.21.15.72:0/2273881013 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f197c003db0 con 0x7f198c108d70 2024-04-26T23:33:53.408 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:52.686+0000 7f1988c00700 1 -- 172.21.15.72:0/2273881013 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f197c011630 con 0x7f198c108d70 2024-04-26T23:33:53.408 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:52.686+0000 7f1993000700 1 -- 172.21.15.72:0/2273881013 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f198c19eeb0 con 0x7f198c108d70 2024-04-26T23:33:53.408 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:52.686+0000 7f1988c00700 1 -- 172.21.15.72:0/2273881013 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f197c011950 con 0x7f198c108d70 2024-04-26T23:33:53.408 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:52.686+0000 7f1993000700 1 -- 172.21.15.72:0/2273881013 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f198c1a1ba0 con 0x7f198c108d70 2024-04-26T23:33:53.408 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:52.686+0000 7f1988c00700 1 -- 172.21.15.72:0/2273881013 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f197c01a460 con 0x7f198c108d70 2024-04-26T23:33:53.408 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:52.686+0000 7f1988c00700 1 -- 172.21.15.72:0/2273881013 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f197c01b780 con 0x7f198c108d70 2024-04-26T23:33:53.408 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:52.687+0000 7f1993000700 1 -- 172.21.15.72:0/2273881013 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f196c005330 con 0x7f198c108d70 2024-04-26T23:33:53.409 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:52.690+0000 7f1988c00700 1 -- 172.21.15.72:0/2273881013 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f197c031020 con 0x7f198c108d70 2024-04-26T23:33:53.409 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:52.766+0000 7f1993000700 1 -- 172.21.15.72:0/2273881013 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f196c002400 con 0x7f198c108d70 2024-04-26T23:33:53.409 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:52.775+0000 7f1988c00700 1 -- 172.21.15.72:0/2273881013 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+465 (secure 0 0 0) 0x7f197c01b9d0 con 0x7f198c108d70 2024-04-26T23:33:53.409 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:52.776+0000 7f1993000700 1 -- 172.21.15.72:0/2273881013 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f198c108d70 msgr2=0x7f198c19e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:33:53.409 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:52.776+0000 7f1993000700 1 --2- 172.21.15.72:0/2273881013 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f198c108d70 0x7f198c19e6e0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f197c000c00 tx=0x7f197c004750 comp rx=0 tx=0).stop 2024-04-26T23:33:53.409 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:52.776+0000 7f1993000700 1 -- 172.21.15.72:0/2273881013 shutdown_connections 2024-04-26T23:33:53.409 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:52.777+0000 7f1993000700 1 --2- 172.21.15.72:0/2273881013 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f198c108d70 0x7f198c19e6e0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:33:53.409 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:52.777+0000 7f1993000700 1 -- 172.21.15.72:0/2273881013 >> 172.21.15.72:0/2273881013 conn(0x7f198c1042d0 msgr2=0x7f198c07e2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:33:53.409 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:52.777+0000 7f1993000700 1 -- 172.21.15.72:0/2273881013 shutdown_connections 2024-04-26T23:33:53.410 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:52.777+0000 7f1993000700 1 -- 172.21.15.72:0/2273881013 wait complete. 2024-04-26T23:33:53.410 INFO:teuthology.orchestra.run.smithi072.stdout:Generating new minimal ceph.conf... 2024-04-26T23:33:54.532 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:54 smithi072 ceph-mon[27806]: from='client.? 172.21.15.72:0/154030194' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:33:54.839 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:54.150+0000 7f2a11800700 1 Processor -- start 2024-04-26T23:33:54.839 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:54.151+0000 7f2a11800700 1 -- start start 2024-04-26T23:33:54.840 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:54.151+0000 7f2a11800700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a0c082dd0 0x7f2a0c0831f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:33:54.840 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:54.151+0000 7f2a11800700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f2a0c0837c0 con 0x7f2a0c082dd0 2024-04-26T23:33:54.840 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:54.151+0000 7f2a0aa00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a0c082dd0 0x7f2a0c0831f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:33:54.840 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:54.151+0000 7f2a0aa00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a0c082dd0 0x7f2a0c0831f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56512/0 (socket says 172.21.15.72:56512) 2024-04-26T23:33:54.840 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:54.151+0000 7f2a0aa00700 1 -- 172.21.15.72:0/3765216273 learned_addr learned my addr 172.21.15.72:0/3765216273 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:33:54.840 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:54.152+0000 7f2a0aa00700 1 -- 172.21.15.72:0/3765216273 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a0c110550 con 0x7f2a0c082dd0 2024-04-26T23:33:54.840 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:54.152+0000 7f2a0aa00700 1 --2- 172.21.15.72:0/3765216273 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a0c082dd0 0x7f2a0c0831f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f29fc009e80 tx=0x7f29fc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=621970f9e02c017d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:33:54.840 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:54.153+0000 7f2a0a000700 1 -- 172.21.15.72:0/3765216273 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f29fc017070 con 0x7f2a0c082dd0 2024-04-26T23:33:54.840 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:54.153+0000 7f2a0a000700 1 -- 172.21.15.72:0/3765216273 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f29fc00b8c0 con 0x7f2a0c082dd0 2024-04-26T23:33:54.840 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:54.153+0000 7f2a0a000700 1 -- 172.21.15.72:0/3765216273 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f29fc004490 con 0x7f2a0c082dd0 2024-04-26T23:33:54.840 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:54.153+0000 7f2a11800700 1 -- 172.21.15.72:0/3765216273 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a0c082dd0 msgr2=0x7f2a0c0831f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:33:54.840 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:54.153+0000 7f2a11800700 1 --2- 172.21.15.72:0/3765216273 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a0c082dd0 0x7f2a0c0831f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f29fc009e80 tx=0x7f29fc00b330 comp rx=0 tx=0).stop 2024-04-26T23:33:54.841 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:54.153+0000 7f2a11800700 1 -- 172.21.15.72:0/3765216273 shutdown_connections 2024-04-26T23:33:54.841 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:54.154+0000 7f2a11800700 1 --2- 172.21.15.72:0/3765216273 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a0c082dd0 0x7f2a0c0831f0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:33:54.841 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:54.154+0000 7f2a11800700 1 -- 172.21.15.72:0/3765216273 >> 172.21.15.72:0/3765216273 conn(0x7f2a0c07e390 msgr2=0x7f2a0c0807b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:33:54.841 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:54.154+0000 7f2a11800700 1 -- 172.21.15.72:0/3765216273 shutdown_connections 2024-04-26T23:33:54.841 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:54.154+0000 7f2a11800700 1 -- 172.21.15.72:0/3765216273 wait complete. 2024-04-26T23:33:54.841 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:54.155+0000 7f2a11800700 1 Processor -- start 2024-04-26T23:33:54.841 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:54.155+0000 7f2a11800700 1 -- start start 2024-04-26T23:33:54.841 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:54.155+0000 7f2a11800700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a0c082dd0 0x7f2a0c1a3300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:33:54.841 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:54.155+0000 7f2a11800700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f2a0c113900 con 0x7f2a0c082dd0 2024-04-26T23:33:54.841 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:54.155+0000 7f2a0aa00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a0c082dd0 0x7f2a0c1a3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:33:54.841 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:54.155+0000 7f2a0aa00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a0c082dd0 0x7f2a0c1a3300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56524/0 (socket says 172.21.15.72:56524) 2024-04-26T23:33:54.841 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:54.155+0000 7f2a0aa00700 1 -- 172.21.15.72:0/154030194 learned_addr learned my addr 172.21.15.72:0/154030194 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:33:54.842 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:54.156+0000 7f2a0aa00700 1 -- 172.21.15.72:0/154030194 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a0c1a3840 con 0x7f2a0c082dd0 2024-04-26T23:33:54.842 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:54.156+0000 7f2a0aa00700 1 --2- 172.21.15.72:0/154030194 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a0c082dd0 0x7f2a0c1a3300 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f29fc00b7b0 tx=0x7f29fc00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:33:54.842 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:54.156+0000 7f2a03e00700 1 -- 172.21.15.72:0/154030194 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f29fc020020 con 0x7f2a0c082dd0 2024-04-26T23:33:54.842 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:54.156+0000 7f2a03e00700 1 -- 172.21.15.72:0/154030194 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f29fc012430 con 0x7f2a0c082dd0 2024-04-26T23:33:54.842 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:54.156+0000 7f2a11800700 1 -- 172.21.15.72:0/154030194 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a0c1a3ad0 con 0x7f2a0c082dd0 2024-04-26T23:33:54.842 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:54.156+0000 7f2a03e00700 1 -- 172.21.15.72:0/154030194 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f29fc017040 con 0x7f2a0c082dd0 2024-04-26T23:33:54.842 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:54.156+0000 7f2a11800700 1 -- 172.21.15.72:0/154030194 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a0c1a67c0 con 0x7f2a0c082dd0 2024-04-26T23:33:54.842 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:54.157+0000 7f2a03e00700 1 -- 172.21.15.72:0/154030194 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f29fc023020 con 0x7f2a0c082dd0 2024-04-26T23:33:54.842 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:54.157+0000 7f2a03e00700 1 -- 172.21.15.72:0/154030194 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f29fc00f050 con 0x7f2a0c082dd0 2024-04-26T23:33:54.842 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:54.158+0000 7f2a11800700 1 -- 172.21.15.72:0/154030194 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a0c064a80 con 0x7f2a0c082dd0 2024-04-26T23:33:54.842 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:54.160+0000 7f2a03e00700 1 -- 172.21.15.72:0/154030194 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f29fc02a020 con 0x7f2a0c082dd0 2024-04-26T23:33:54.842 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:54.231+0000 7f2a11800700 1 -- 172.21.15.72:0/154030194 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f2a0c1a6990 con 0x7f2a0c082dd0 2024-04-26T23:33:54.843 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:54.232+0000 7f2a03e00700 1 -- 172.21.15.72:0/154030194 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f29fc00b860 con 0x7f2a0c082dd0 2024-04-26T23:33:54.843 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:54.233+0000 7f2a11800700 1 -- 172.21.15.72:0/154030194 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a0c082dd0 msgr2=0x7f2a0c1a3300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:33:54.843 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:54.233+0000 7f2a11800700 1 --2- 172.21.15.72:0/154030194 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a0c082dd0 0x7f2a0c1a3300 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f29fc00b7b0 tx=0x7f29fc00be20 comp rx=0 tx=0).stop 2024-04-26T23:33:54.843 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:54.233+0000 7f2a11800700 1 -- 172.21.15.72:0/154030194 shutdown_connections 2024-04-26T23:33:54.843 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:54.233+0000 7f2a11800700 1 --2- 172.21.15.72:0/154030194 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a0c082dd0 0x7f2a0c1a3300 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:33:54.843 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:54.233+0000 7f2a11800700 1 -- 172.21.15.72:0/154030194 >> 172.21.15.72:0/154030194 conn(0x7f2a0c07e390 msgr2=0x7f2a0c07eef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:33:54.843 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:54.233+0000 7f2a11800700 1 -- 172.21.15.72:0/154030194 shutdown_connections 2024-04-26T23:33:54.843 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:54.233+0000 7f2a11800700 1 -- 172.21.15.72:0/154030194 wait complete. 2024-04-26T23:33:54.843 INFO:teuthology.orchestra.run.smithi072.stdout:Restarting the monitor... 2024-04-26T23:33:55.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:54 smithi072 systemd[1]: Stopping Ceph mon.a for 3c8453ae-0425-11ef-bc93-c7b262605968... 2024-04-26T23:33:55.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:55 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mon-a[27802]: 2024-04-26T23:33:55.299+0000 7f83d6a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T23:33:55.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:55 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mon-a[27802]: 2024-04-26T23:33:55.299+0000 7f83d6a00700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-26T23:33:55.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:55 smithi072 podman[28059]: 2024-04-26 23:33:55.48487274 +0000 UTC m=+0.313338805 container died b0847df67adeb4922cafad3260e16fa60056d4e6903883ca1807e322a4e86eb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424) 2024-04-26T23:33:56.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:56 smithi072 podman[28059]: 2024-04-26 23:33:56.669218474 +0000 UTC m=+1.497684553 container cleanup b0847df67adeb4922cafad3260e16fa60056d4e6903883ca1807e322a4e86eb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-26T23:33:56.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:56 smithi072 bash[28059]: ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mon-a 2024-04-26T23:33:57.247 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:56 smithi072 podman[28073]: 2024-04-26 23:33:56.879084812 +0000 UTC m=+1.373049579 container remove b0847df67adeb4922cafad3260e16fa60056d4e6903883ca1807e322a4e86eb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mon-a, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-26T23:33:57.542 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 systemd[1]: ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-26T23:33:57.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 systemd[1]: Stopped Ceph mon.a for 3c8453ae-0425-11ef-bc93-c7b262605968. 2024-04-26T23:33:57.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 systemd[1]: ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mon.a.service: Consumed 1.845s CPU time. 2024-04-26T23:33:57.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 systemd[1]: Starting Ceph mon.a for 3c8453ae-0425-11ef-bc93-c7b262605968... 2024-04-26T23:33:57.899 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 podman[28165]: 2024-04-26 23:33:57.536886687 +0000 UTC m=+0.031084397 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T23:33:57.899 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 podman[28165]: 2024-04-26 23:33:57.633011042 +0000 UTC m=+0.127208789 container create a100b14c4c0236a1b57ef9f0e10d64dcbfd011e79faf68b0a303924f27d2349b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mon-a, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS) 2024-04-26T23:33:57.899 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 podman[28165]: 2024-04-26 23:33:57.896440391 +0000 UTC m=+0.390638144 container init a100b14c4c0236a1b57ef9f0e10d64dcbfd011e79faf68b0a303924f27d2349b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mon-a, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-26T23:33:58.000 INFO:teuthology.orchestra.run.smithi072.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-26T23:33:58.151 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 podman[28165]: 2024-04-26 23:33:57.906223208 +0000 UTC m=+0.400420923 container start a100b14c4c0236a1b57ef9f0e10d64dcbfd011e79faf68b0a303924f27d2349b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mon-a, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-26T23:33:58.151 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: set uid:gid to 167:167 (ceph:ceph) 2024-04-26T23:33:58.151 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev), process ceph-mon, pid 2 2024-04-26T23:33:58.151 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: pidfile_write: ignore empty --pid-file 2024-04-26T23:33:58.151 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: load: jerasure load: lrc 2024-04-26T23:33:58.151 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: RocksDB version: 7.9.2 2024-04-26T23:33:58.151 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Git sha 0 2024-04-26T23:33:58.152 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Compile date 2024-04-26 11:39:23 2024-04-26T23:33:58.152 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: DB SUMMARY 2024-04-26T23:33:58.152 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: DB Session ID: B51UW9IVKJI5XBXEP0FB 2024-04-26T23:33:58.152 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: CURRENT file: CURRENT 2024-04-26T23:33:58.152 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: IDENTITY file: IDENTITY 2024-04-26T23:33:58.152 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-26T23:33:58.152 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-26T23:33:58.152 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 104971 ; 2024-04-26T23:33:58.152 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.error_if_exists: 0 2024-04-26T23:33:58.152 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.create_if_missing: 0 2024-04-26T23:33:58.152 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.paranoid_checks: 1 2024-04-26T23:33:58.152 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-26T23:33:58.152 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-26T23:33:58.152 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-26T23:33:58.153 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.env: 0x55a47dc50cc0 2024-04-26T23:33:58.153 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.fs: PosixFileSystem 2024-04-26T23:33:58.153 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.info_log: 0x55a49d3c9dc0 2024-04-26T23:33:58.153 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.max_file_opening_threads: 16 2024-04-26T23:33:58.153 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.statistics: (nil) 2024-04-26T23:33:58.153 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.use_fsync: 0 2024-04-26T23:33:58.153 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.max_log_file_size: 0 2024-04-26T23:33:58.153 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-26T23:33:58.153 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-26T23:33:58.153 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.keep_log_file_num: 1000 2024-04-26T23:33:58.153 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.recycle_log_file_num: 0 2024-04-26T23:33:58.153 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.allow_fallocate: 1 2024-04-26T23:33:58.154 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.allow_mmap_reads: 0 2024-04-26T23:33:58.154 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.allow_mmap_writes: 0 2024-04-26T23:33:58.155 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.use_direct_reads: 0 2024-04-26T23:33:58.155 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-26T23:33:58.155 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.create_missing_column_families: 0 2024-04-26T23:33:58.155 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.db_log_dir: 2024-04-26T23:33:58.155 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.wal_dir: 2024-04-26T23:33:58.155 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-26T23:33:58.155 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-26T23:33:58.155 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-26T23:33:58.155 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-26T23:33:58.155 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-26T23:33:58.155 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-26T23:33:58.155 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.advise_random_on_open: 1 2024-04-26T23:33:58.156 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.db_write_buffer_size: 0 2024-04-26T23:33:58.156 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.write_buffer_manager: 0x55a49d39b0e0 2024-04-26T23:33:58.156 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-26T23:33:58.156 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-26T23:33:58.156 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-26T23:33:58.156 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.rate_limiter: (nil) 2024-04-26T23:33:58.156 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-26T23:33:58.156 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.wal_recovery_mode: 2 2024-04-26T23:33:58.156 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.enable_thread_tracking: 0 2024-04-26T23:33:58.156 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.enable_pipelined_write: 0 2024-04-26T23:33:58.156 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.unordered_write: 0 2024-04-26T23:33:58.156 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-26T23:33:58.156 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-26T23:33:58.157 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-26T23:33:58.157 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-26T23:33:58.157 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.row_cache: None 2024-04-26T23:33:58.157 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.wal_filter: None 2024-04-26T23:33:58.157 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-26T23:33:58.157 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.allow_ingest_behind: 0 2024-04-26T23:33:58.157 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.two_write_queues: 0 2024-04-26T23:33:58.157 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.manual_wal_flush: 0 2024-04-26T23:33:58.157 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.wal_compression: 0 2024-04-26T23:33:58.157 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.atomic_flush: 0 2024-04-26T23:33:58.157 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-26T23:33:58.157 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-26T23:33:58.157 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-26T23:33:58.157 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.log_readahead_size: 0 2024-04-26T23:33:58.158 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-26T23:33:58.158 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.best_efforts_recovery: 0 2024-04-26T23:33:58.158 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-26T23:33:58.158 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-26T23:33:58.158 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.allow_data_in_errors: 0 2024-04-26T23:33:58.158 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.db_host_id: __hostname__ 2024-04-26T23:33:58.158 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-26T23:33:58.158 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.max_background_jobs: 2 2024-04-26T23:33:58.158 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.max_background_compactions: -1 2024-04-26T23:33:58.158 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.max_subcompactions: 1 2024-04-26T23:33:58.158 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-26T23:33:58.158 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-26T23:33:58.159 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-26T23:33:58.159 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.max_total_wal_size: 0 2024-04-26T23:33:58.159 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-26T23:33:58.159 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-26T23:33:58.159 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-26T23:33:58.159 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-26T23:33:58.159 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.max_open_files: -1 2024-04-26T23:33:58.159 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.bytes_per_sync: 0 2024-04-26T23:33:58.159 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-26T23:33:58.159 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-26T23:33:58.159 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.compaction_readahead_size: 0 2024-04-26T23:33:58.159 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.max_background_flushes: -1 2024-04-26T23:33:58.160 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Compression algorithms supported: 2024-04-26T23:33:58.160 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-26T23:33:58.160 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: kZSTD supported: 0 2024-04-26T23:33:58.160 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: kXpressCompression supported: 0 2024-04-26T23:33:58.160 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: kLZ4HCCompression supported: 1 2024-04-26T23:33:58.160 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: kZlibCompression supported: 1 2024-04-26T23:33:58.160 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: kSnappyCompression supported: 1 2024-04-26T23:33:58.160 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: kLZ4Compression supported: 1 2024-04-26T23:33:58.160 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: kBZip2Compression supported: 0 2024-04-26T23:33:58.160 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-26T23:33:58.160 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-26T23:33:58.160 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-26T23:33:58.160 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-26T23:33:58.160 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-26T23:33:58.161 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.merge_operator: 2024-04-26T23:33:58.161 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.compaction_filter: None 2024-04-26T23:33:58.161 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.compaction_filter_factory: None 2024-04-26T23:33:58.161 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.sst_partitioner_factory: None 2024-04-26T23:33:58.161 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-26T23:33:58.161 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-26T23:33:58.161 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a49d3c9f80) 2024-04-26T23:33:58.161 INFO:journalctl@ceph.mon.a.smithi072.stdout: cache_index_and_filter_blocks: 1 2024-04-26T23:33:58.161 INFO:journalctl@ceph.mon.a.smithi072.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-26T23:33:58.161 INFO:journalctl@ceph.mon.a.smithi072.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-26T23:33:58.161 INFO:journalctl@ceph.mon.a.smithi072.stdout: pin_top_level_index_and_filter: 1 2024-04-26T23:33:58.161 INFO:journalctl@ceph.mon.a.smithi072.stdout: index_type: 0 2024-04-26T23:33:58.161 INFO:journalctl@ceph.mon.a.smithi072.stdout: data_block_index_type: 0 2024-04-26T23:33:58.161 INFO:journalctl@ceph.mon.a.smithi072.stdout: index_shortening: 1 2024-04-26T23:33:58.162 INFO:journalctl@ceph.mon.a.smithi072.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-26T23:33:58.162 INFO:journalctl@ceph.mon.a.smithi072.stdout: checksum: 4 2024-04-26T23:33:58.162 INFO:journalctl@ceph.mon.a.smithi072.stdout: no_block_cache: 0 2024-04-26T23:33:58.162 INFO:journalctl@ceph.mon.a.smithi072.stdout: block_cache: 0x55a49d3cd1f0 2024-04-26T23:33:58.162 INFO:journalctl@ceph.mon.a.smithi072.stdout: block_cache_name: BinnedLRUCache 2024-04-26T23:33:58.162 INFO:journalctl@ceph.mon.a.smithi072.stdout: block_cache_options: 2024-04-26T23:33:58.162 INFO:journalctl@ceph.mon.a.smithi072.stdout: capacity : 536870912 2024-04-26T23:33:58.162 INFO:journalctl@ceph.mon.a.smithi072.stdout: num_shard_bits : 4 2024-04-26T23:33:58.162 INFO:journalctl@ceph.mon.a.smithi072.stdout: strict_capacity_limit : 0 2024-04-26T23:33:58.162 INFO:journalctl@ceph.mon.a.smithi072.stdout: high_pri_pool_ratio: 0.000 2024-04-26T23:33:58.162 INFO:journalctl@ceph.mon.a.smithi072.stdout: block_cache_compressed: (nil) 2024-04-26T23:33:58.162 INFO:journalctl@ceph.mon.a.smithi072.stdout: persistent_cache: (nil) 2024-04-26T23:33:58.162 INFO:journalctl@ceph.mon.a.smithi072.stdout: block_size: 4096 2024-04-26T23:33:58.163 INFO:journalctl@ceph.mon.a.smithi072.stdout: block_size_deviation: 10 2024-04-26T23:33:58.163 INFO:journalctl@ceph.mon.a.smithi072.stdout: block_restart_interval: 16 2024-04-26T23:33:58.163 INFO:journalctl@ceph.mon.a.smithi072.stdout: index_block_restart_interval: 1 2024-04-26T23:33:58.163 INFO:journalctl@ceph.mon.a.smithi072.stdout: metadata_block_size: 4096 2024-04-26T23:33:58.163 INFO:journalctl@ceph.mon.a.smithi072.stdout: partition_filters: 0 2024-04-26T23:33:58.163 INFO:journalctl@ceph.mon.a.smithi072.stdout: use_delta_encoding: 1 2024-04-26T23:33:58.163 INFO:journalctl@ceph.mon.a.smithi072.stdout: filter_policy: bloomfilter 2024-04-26T23:33:58.163 INFO:journalctl@ceph.mon.a.smithi072.stdout: whole_key_filtering: 1 2024-04-26T23:33:58.163 INFO:journalctl@ceph.mon.a.smithi072.stdout: verify_compression: 0 2024-04-26T23:33:58.163 INFO:journalctl@ceph.mon.a.smithi072.stdout: read_amp_bytes_per_bit: 0 2024-04-26T23:33:58.163 INFO:journalctl@ceph.mon.a.smithi072.stdout: format_version: 5 2024-04-26T23:33:58.163 INFO:journalctl@ceph.mon.a.smithi072.stdout: enable_index_compression: 1 2024-04-26T23:33:58.163 INFO:journalctl@ceph.mon.a.smithi072.stdout: block_align: 0 2024-04-26T23:33:58.163 INFO:journalctl@ceph.mon.a.smithi072.stdout: max_auto_readahead_size: 262144 2024-04-26T23:33:58.163 INFO:journalctl@ceph.mon.a.smithi072.stdout: prepopulate_block_cache: 0 2024-04-26T23:33:58.164 INFO:journalctl@ceph.mon.a.smithi072.stdout: initial_auto_readahead_size: 8192 2024-04-26T23:33:58.164 INFO:journalctl@ceph.mon.a.smithi072.stdout: num_file_reads_for_auto_readahead: 2 2024-04-26T23:33:58.164 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.write_buffer_size: 33554432 2024-04-26T23:33:58.164 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.max_write_buffer_number: 2 2024-04-26T23:33:58.164 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.compression: NoCompression 2024-04-26T23:33:58.164 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.bottommost_compression: Disabled 2024-04-26T23:33:58.164 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.prefix_extractor: nullptr 2024-04-26T23:33:58.164 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-26T23:33:58.164 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.num_levels: 7 2024-04-26T23:33:58.164 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-26T23:33:58.164 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-26T23:33:58.164 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-26T23:33:58.164 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-26T23:33:58.165 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-26T23:33:58.165 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-26T23:33:58.165 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-26T23:33:58.165 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-26T23:33:58.165 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-26T23:33:58.165 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-26T23:33:58.165 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-26T23:33:58.165 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-26T23:33:58.165 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-26T23:33:58.165 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.compression_opts.level: 32767 2024-04-26T23:33:58.165 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.compression_opts.strategy: 0 2024-04-26T23:33:58.165 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-26T23:33:58.165 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-26T23:33:58.166 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-26T23:33:58.166 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-26T23:33:58.166 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.compression_opts.enabled: false 2024-04-26T23:33:58.166 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-26T23:33:58.166 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-26T23:33:58.166 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-26T23:33:58.166 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-26T23:33:58.166 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.target_file_size_base: 67108864 2024-04-26T23:33:58.166 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-26T23:33:58.166 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-26T23:33:58.166 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-26T23:33:58.166 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-26T23:33:58.167 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-26T23:33:58.167 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-26T23:33:58.167 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-26T23:33:58.167 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-26T23:33:58.167 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-26T23:33:58.167 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-26T23:33:58.167 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-26T23:33:58.167 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-26T23:33:58.167 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-26T23:33:58.167 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-26T23:33:58.168 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.arena_block_size: 1048576 2024-04-26T23:33:58.168 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-26T23:33:58.168 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-26T23:33:58.168 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.disable_auto_compactions: 0 2024-04-26T23:33:58.168 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-26T23:33:58.168 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-26T23:33:58.169 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-26T23:33:58.169 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-26T23:33:58.169 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-26T23:33:58.169 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-26T23:33:58.169 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-26T23:33:58.169 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-26T23:33:58.169 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-26T23:33:58.169 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-26T23:33:58.169 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-26T23:33:58.169 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.inplace_update_support: 0 2024-04-26T23:33:58.169 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-26T23:33:58.169 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-26T23:33:58.170 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-26T23:33:58.170 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-26T23:33:58.170 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.bloom_locality: 0 2024-04-26T23:33:58.170 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.max_successive_merges: 0 2024-04-26T23:33:58.170 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-26T23:33:58.170 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.paranoid_file_checks: 0 2024-04-26T23:33:58.170 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.force_consistency_checks: 1 2024-04-26T23:33:58.170 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.report_bg_io_stats: 0 2024-04-26T23:33:58.170 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.ttl: 2592000 2024-04-26T23:33:58.170 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-26T23:33:58.171 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-26T23:33:58.171 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-26T23:33:58.171 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.enable_blob_files: false 2024-04-26T23:33:58.171 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.min_blob_size: 0 2024-04-26T23:33:58.171 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.blob_file_size: 268435456 2024-04-26T23:33:58.171 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-26T23:33:58.171 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-26T23:33:58.171 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-26T23:33:58.171 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-26T23:33:58.171 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-26T23:33:58.171 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.blob_file_starting_level: 0 2024-04-26T23:33:58.171 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-26T23:33:58.171 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-26T23:33:58.171 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-26T23:33:58.172 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 3311dc80-fdfc-4084-ba76-c788d91b02ba 2024-04-26T23:33:58.172 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714174437967295, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-26T23:33:58.172 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-26T23:33:58.172 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714174437970615, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 100023, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 330, "table_properties": {"data_size": 97909, "index_size": 306, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14088, "raw_average_key_size": 49, "raw_value_size": 90234, "raw_average_value_size": 316, "num_data_blocks": 12, "num_entries": 285, "num_filter_entries": 285, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714174437, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "3311dc80-fdfc-4084-ba76-c788d91b02ba", "db_session_id": "B51UW9IVKJI5XBXEP0FB", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-26T23:33:58.172 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714174437970975, "job": 1, "event": "recovery_finished"} 2024-04-26T23:33:58.172 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-26T23:33:58.172 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-26T23:33:58.172 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55a49d4ea000 2024-04-26T23:33:58.172 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: DB pointer 0x55a49d4d6000 2024-04-26T23:33:58.172 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-26T23:33:58.172 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-26T23:33:58.172 INFO:journalctl@ceph.mon.a.smithi072.stdout: ** DB Stats ** 2024-04-26T23:33:58.172 INFO:journalctl@ceph.mon.a.smithi072.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T23:33:58.172 INFO:journalctl@ceph.mon.a.smithi072.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-26T23:33:58.172 INFO:journalctl@ceph.mon.a.smithi072.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T23:33:58.173 INFO:journalctl@ceph.mon.a.smithi072.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T23:33:58.173 INFO:journalctl@ceph.mon.a.smithi072.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-26T23:33:58.173 INFO:journalctl@ceph.mon.a.smithi072.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T23:33:58.173 INFO:journalctl@ceph.mon.a.smithi072.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T23:33:58.173 INFO:journalctl@ceph.mon.a.smithi072.stdout: 2024-04-26T23:33:58.173 INFO:journalctl@ceph.mon.a.smithi072.stdout: ** Compaction Stats [default] ** 2024-04-26T23:33:58.173 INFO:journalctl@ceph.mon.a.smithi072.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T23:33:58.173 INFO:journalctl@ceph.mon.a.smithi072.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-26T23:33:58.173 INFO:journalctl@ceph.mon.a.smithi072.stdout: L0 2/0 99.54 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 44.1 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-04-26T23:33:58.173 INFO:journalctl@ceph.mon.a.smithi072.stdout: Sum 2/0 99.54 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 44.1 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-04-26T23:33:58.173 INFO:journalctl@ceph.mon.a.smithi072.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 44.1 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-04-26T23:33:58.173 INFO:journalctl@ceph.mon.a.smithi072.stdout: 2024-04-26T23:33:58.173 INFO:journalctl@ceph.mon.a.smithi072.stdout: ** Compaction Stats [default] ** 2024-04-26T23:33:58.174 INFO:journalctl@ceph.mon.a.smithi072.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T23:33:58.174 INFO:journalctl@ceph.mon.a.smithi072.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-26T23:33:58.174 INFO:journalctl@ceph.mon.a.smithi072.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 44.1 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-04-26T23:33:58.174 INFO:journalctl@ceph.mon.a.smithi072.stdout: 2024-04-26T23:33:58.174 INFO:journalctl@ceph.mon.a.smithi072.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-26T23:33:58.174 INFO:journalctl@ceph.mon.a.smithi072.stdout: 2024-04-26T23:33:58.174 INFO:journalctl@ceph.mon.a.smithi072.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T23:33:58.174 INFO:journalctl@ceph.mon.a.smithi072.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-26T23:33:58.174 INFO:journalctl@ceph.mon.a.smithi072.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-26T23:33:58.174 INFO:journalctl@ceph.mon.a.smithi072.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-26T23:33:58.174 INFO:journalctl@ceph.mon.a.smithi072.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-26T23:33:58.174 INFO:journalctl@ceph.mon.a.smithi072.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-26T23:33:58.174 INFO:journalctl@ceph.mon.a.smithi072.stdout: Cumulative compaction: 0.00 GB write, 9.29 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T23:33:58.175 INFO:journalctl@ceph.mon.a.smithi072.stdout: Interval compaction: 0.00 GB write, 9.29 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T23:33:58.175 INFO:journalctl@ceph.mon.a.smithi072.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-26T23:33:58.175 INFO:journalctl@ceph.mon.a.smithi072.stdout: Block cache BinnedLRUCache@0x55a49d3cd1f0#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-04-26T23:33:58.175 INFO:journalctl@ceph.mon.a.smithi072.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-26T23:33:58.175 INFO:journalctl@ceph.mon.a.smithi072.stdout: 2024-04-26T23:33:58.175 INFO:journalctl@ceph.mon.a.smithi072.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-26T23:33:58.175 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: starting mon.a rank 0 at public addrs [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] at bind addrs [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 3c8453ae-0425-11ef-bc93-c7b262605968 2024-04-26T23:33:58.175 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: mon.a@-1(???) e1 preinit fsid 3c8453ae-0425-11ef-bc93-c7b262605968 2024-04-26T23:33:58.175 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: mon.a@-1(???).mds e1 new map 2024-04-26T23:33:58.175 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: mon.a@-1(???).mds e1 print_map 2024-04-26T23:33:58.175 INFO:journalctl@ceph.mon.a.smithi072.stdout: e1 2024-04-26T23:33:58.175 INFO:journalctl@ceph.mon.a.smithi072.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-26T23:33:58.175 INFO:journalctl@ceph.mon.a.smithi072.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-26T23:33:58.175 INFO:journalctl@ceph.mon.a.smithi072.stdout: legacy client fscid: -1 2024-04-26T23:33:58.176 INFO:journalctl@ceph.mon.a.smithi072.stdout: 2024-04-26T23:33:58.176 INFO:journalctl@ceph.mon.a.smithi072.stdout: No filesystems configured 2024-04-26T23:33:58.176 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-26T23:33:58.176 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T23:33:58.176 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T23:33:58.176 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T23:33:58.176 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 ceph-mon[28184]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-26T23:33:58.176 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 bash[28165]: a100b14c4c0236a1b57ef9f0e10d64dcbfd011e79faf68b0a303924f27d2349b 2024-04-26T23:33:58.176 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:57 smithi072 systemd[1]: Started Ceph mon.a for 3c8453ae-0425-11ef-bc93-c7b262605968. 2024-04-26T23:33:58.177 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:58 smithi072 ceph-mon[28184]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-26T23:33:58.177 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:58 smithi072 ceph-mon[28184]: monmap epoch 1 2024-04-26T23:33:58.177 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:58 smithi072 ceph-mon[28184]: fsid 3c8453ae-0425-11ef-bc93-c7b262605968 2024-04-26T23:33:58.177 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:58 smithi072 ceph-mon[28184]: last_changed 2024-04-26T23:33:30.337416+0000 2024-04-26T23:33:58.177 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:58 smithi072 ceph-mon[28184]: created 2024-04-26T23:33:30.337416+0000 2024-04-26T23:33:58.177 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:58 smithi072 ceph-mon[28184]: min_mon_release 19 (squid) 2024-04-26T23:33:58.177 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:58 smithi072 ceph-mon[28184]: election_strategy: 1 2024-04-26T23:33:58.177 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:58 smithi072 ceph-mon[28184]: 0: [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] mon.a 2024-04-26T23:33:58.177 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:58 smithi072 ceph-mon[28184]: fsmap 2024-04-26T23:33:58.177 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:58 smithi072 ceph-mon[28184]: osdmap e1: 0 total, 0 up, 0 in 2024-04-26T23:33:58.177 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:58 smithi072 ceph-mon[28184]: mgrmap e1: no daemons active 2024-04-26T23:33:59.383 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:58.638+0000 7fde7ae00700 1 Processor -- start 2024-04-26T23:33:59.388 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:58.638+0000 7fde7ae00700 1 -- start start 2024-04-26T23:33:59.388 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:58.638+0000 7fde7ae00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde7407fd60 0x7fde74080180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:33:59.388 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:58.639+0000 7fde7ae00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fde74080750 con 0x7fde7407fd60 2024-04-26T23:33:59.388 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:58.639+0000 7fde73e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde7407fd60 0x7fde74080180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:33:59.388 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:58.639+0000 7fde73e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde7407fd60 0x7fde74080180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37174/0 (socket says 172.21.15.72:37174) 2024-04-26T23:33:59.389 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:58.639+0000 7fde73e00700 1 -- 172.21.15.72:0/2546290816 learned_addr learned my addr 172.21.15.72:0/2546290816 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:33:59.389 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:58.640+0000 7fde73e00700 1 -- 172.21.15.72:0/2546290816 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde74080f60 con 0x7fde7407fd60 2024-04-26T23:33:59.389 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:58.640+0000 7fde73e00700 1 --2- 172.21.15.72:0/2546290816 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde7407fd60 0x7fde74080180 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fde640098a0 tx=0x7fde64009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69298f4eaa5de37 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:33:59.389 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:58.641+0000 7fde73400700 1 -- 172.21.15.72:0/2546290816 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde64004160 con 0x7fde7407fd60 2024-04-26T23:33:59.389 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:58.641+0000 7fde73400700 1 -- 172.21.15.72:0/2546290816 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fde64004300 con 0x7fde7407fd60 2024-04-26T23:33:59.389 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:58.641+0000 7fde73400700 1 -- 172.21.15.72:0/2546290816 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde64011af0 con 0x7fde7407fd60 2024-04-26T23:33:59.389 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:58.641+0000 7fde7ae00700 1 -- 172.21.15.72:0/2546290816 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde7407fd60 msgr2=0x7fde74080180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:33:59.389 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:58.641+0000 7fde7ae00700 1 --2- 172.21.15.72:0/2546290816 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde7407fd60 0x7fde74080180 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fde640098a0 tx=0x7fde64009bb0 comp rx=0 tx=0).stop 2024-04-26T23:33:59.389 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:58.641+0000 7fde7ae00700 1 -- 172.21.15.72:0/2546290816 shutdown_connections 2024-04-26T23:33:59.389 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:58.641+0000 7fde7ae00700 1 --2- 172.21.15.72:0/2546290816 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde7407fd60 0x7fde74080180 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:33:59.389 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:58.641+0000 7fde7ae00700 1 -- 172.21.15.72:0/2546290816 >> 172.21.15.72:0/2546290816 conn(0x7fde7407e390 msgr2=0x7fde7407e7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:33:59.390 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:58.642+0000 7fde7ae00700 1 -- 172.21.15.72:0/2546290816 shutdown_connections 2024-04-26T23:33:59.390 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:58.642+0000 7fde7ae00700 1 -- 172.21.15.72:0/2546290816 wait complete. 2024-04-26T23:33:59.390 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:58.643+0000 7fde7ae00700 1 Processor -- start 2024-04-26T23:33:59.390 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:58.643+0000 7fde7ae00700 1 -- start start 2024-04-26T23:33:59.390 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:58.643+0000 7fde7ae00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde7407fd60 0x7fde741a3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:33:59.390 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:58.643+0000 7fde7ae00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fde74082320 con 0x7fde7407fd60 2024-04-26T23:33:59.390 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:58.643+0000 7fde73e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde7407fd60 0x7fde741a3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:33:59.390 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:58.643+0000 7fde73e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde7407fd60 0x7fde741a3350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37182/0 (socket says 172.21.15.72:37182) 2024-04-26T23:33:59.390 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:58.643+0000 7fde73e00700 1 -- 172.21.15.72:0/2344739117 learned_addr learned my addr 172.21.15.72:0/2344739117 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:33:59.390 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:58.644+0000 7fde73e00700 1 -- 172.21.15.72:0/2344739117 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde741a3890 con 0x7fde7407fd60 2024-04-26T23:33:59.390 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:58.644+0000 7fde73e00700 1 --2- 172.21.15.72:0/2344739117 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde7407fd60 0x7fde741a3350 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fde64011ac0 tx=0x7fde64011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:33:59.390 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:58.644+0000 7fde71600700 1 -- 172.21.15.72:0/2344739117 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde64003970 con 0x7fde7407fd60 2024-04-26T23:33:59.390 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:58.644+0000 7fde71600700 1 -- 172.21.15.72:0/2344739117 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fde64003b10 con 0x7fde7407fd60 2024-04-26T23:33:59.391 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:58.644+0000 7fde7ae00700 1 -- 172.21.15.72:0/2344739117 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde741a3b20 con 0x7fde7407fd60 2024-04-26T23:33:59.391 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:58.644+0000 7fde71600700 1 -- 172.21.15.72:0/2344739117 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde64024610 con 0x7fde7407fd60 2024-04-26T23:33:59.391 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:58.645+0000 7fde7ae00700 1 -- 172.21.15.72:0/2344739117 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde741a6810 con 0x7fde7407fd60 2024-04-26T23:33:59.391 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:58.646+0000 7fde71600700 1 -- 172.21.15.72:0/2344739117 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fde64024890 con 0x7fde7407fd60 2024-04-26T23:33:59.391 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:58.646+0000 7fde7ae00700 1 -- 172.21.15.72:0/2344739117 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde74064a80 con 0x7fde7407fd60 2024-04-26T23:33:59.391 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:58.646+0000 7fde71600700 1 -- 172.21.15.72:0/2344739117 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fde6402dd20 con 0x7fde7407fd60 2024-04-26T23:33:59.391 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:58.649+0000 7fde71600700 1 -- 172.21.15.72:0/2344739117 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fde64024cc0 con 0x7fde7407fd60 2024-04-26T23:33:59.391 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:58.724+0000 7fde7ae00700 1 -- 172.21.15.72:0/2344739117 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7fde7402e490 con 0x7fde7407fd60 2024-04-26T23:33:59.391 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:58.729+0000 7fde71600700 1 -- 172.21.15.72:0/2344739117 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7fde6402d3f0 con 0x7fde7407fd60 2024-04-26T23:33:59.391 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:58.730+0000 7fde7ae00700 1 -- 172.21.15.72:0/2344739117 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde7407fd60 msgr2=0x7fde741a3350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:33:59.391 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:58.730+0000 7fde7ae00700 1 --2- 172.21.15.72:0/2344739117 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde7407fd60 0x7fde741a3350 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fde64011ac0 tx=0x7fde64011ba0 comp rx=0 tx=0).stop 2024-04-26T23:33:59.391 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:58.730+0000 7fde7ae00700 1 -- 172.21.15.72:0/2344739117 shutdown_connections 2024-04-26T23:33:59.391 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:58.730+0000 7fde7ae00700 1 --2- 172.21.15.72:0/2344739117 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde7407fd60 0x7fde741a3350 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:33:59.392 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:58.730+0000 7fde7ae00700 1 -- 172.21.15.72:0/2344739117 >> 172.21.15.72:0/2344739117 conn(0x7fde7407e390 msgr2=0x7fde74109a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:33:59.392 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:58.730+0000 7fde7ae00700 1 -- 172.21.15.72:0/2344739117 shutdown_connections 2024-04-26T23:33:59.392 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:33:58.730+0000 7fde7ae00700 1 -- 172.21.15.72:0/2344739117 wait complete. 2024-04-26T23:33:59.392 INFO:teuthology.orchestra.run.smithi072.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-26T23:33:59.413 INFO:teuthology.orchestra.run.smithi072.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-26T23:33:59.413 INFO:teuthology.orchestra.run.smithi072.stdout:Creating mgr... 2024-04-26T23:33:59.415 INFO:teuthology.orchestra.run.smithi072.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-26T23:33:59.416 INFO:teuthology.orchestra.run.smithi072.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-26T23:33:59.893 INFO:teuthology.orchestra.run.smithi072.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mgr.x 2024-04-26T23:33:59.893 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stderr Failed to reset failed state of unit ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mgr.x.service: Unit ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mgr.x.service not loaded. 2024-04-26T23:34:00.042 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:33:59 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/2344739117' entity='client.admin' 2024-04-26T23:34:00.159 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-3c8453ae-0425-11ef-bc93-c7b262605968.target.wants/ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-3c8453ae-0425-11ef-bc93-c7b262605968@.service. 2024-04-26T23:34:00.543 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:00 smithi072 systemd[1]: Starting Ceph mgr.x for 3c8453ae-0425-11ef-bc93-c7b262605968... 2024-04-26T23:34:00.804 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:00 smithi072 podman[28395]: 2024-04-26 23:34:00.451813321 +0000 UTC m=+0.034629950 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T23:34:00.804 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:00 smithi072 podman[28395]: 2024-04-26 23:34:00.557570986 +0000 UTC m=+0.140387648 container create bf3f10c9dfc7bd1ca4360ee6d8f423b51a4151a8408504c678a0165747df2e6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T23:34:00.804 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:00 smithi072 podman[28395]: 2024-04-26 23:34:00.788961341 +0000 UTC m=+0.371778036 container init bf3f10c9dfc7bd1ca4360ee6d8f423b51a4151a8408504c678a0165747df2e6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T23:34:00.804 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:00 smithi072 podman[28395]: 2024-04-26 23:34:00.800102491 +0000 UTC m=+0.382919147 container start bf3f10c9dfc7bd1ca4360ee6d8f423b51a4151a8408504c678a0165747df2e6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T23:34:00.904 INFO:teuthology.orchestra.run.smithi072.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T23:34:00.904 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stdout disabled 2024-04-26T23:34:00.919 INFO:teuthology.orchestra.run.smithi072.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T23:34:00.919 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stdout inactive 2024-04-26T23:34:00.919 INFO:teuthology.orchestra.run.smithi072.stdout:firewalld.service is not enabled 2024-04-26T23:34:00.919 INFO:teuthology.orchestra.run.smithi072.stdout:Not possible to enable service . firewalld.service is not available 2024-04-26T23:34:00.937 INFO:teuthology.orchestra.run.smithi072.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T23:34:00.938 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stdout disabled 2024-04-26T23:34:00.953 INFO:teuthology.orchestra.run.smithi072.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T23:34:00.953 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stdout inactive 2024-04-26T23:34:00.953 INFO:teuthology.orchestra.run.smithi072.stdout:firewalld.service is not enabled 2024-04-26T23:34:00.954 INFO:teuthology.orchestra.run.smithi072.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-26T23:34:00.954 INFO:teuthology.orchestra.run.smithi072.stdout:Waiting for mgr to start... 2024-04-26T23:34:00.954 INFO:teuthology.orchestra.run.smithi072.stdout:Waiting for mgr... 2024-04-26T23:34:01.104 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:00 smithi072 bash[28395]: bf3f10c9dfc7bd1ca4360ee6d8f423b51a4151a8408504c678a0165747df2e6d 2024-04-26T23:34:01.104 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:00 smithi072 systemd[1]: Started Ceph mgr.x for 3c8453ae-0425-11ef-bc93-c7b262605968. 2024-04-26T23:34:01.104 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:01 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:01.041+0000 7f19c4042200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T23:34:01.436 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:01 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:01.298+0000 7f19c4042200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T23:34:01.436 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:01 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:01.432+0000 7f19c4042200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T23:34:01.998 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:01 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:01.875+0000 7f19c4042200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T23:34:01.998 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:01 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/922526335' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T23:34:02.293 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:01 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:01.994+0000 7f19c4042200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T23:34:02.410 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 2024-04-26T23:34:02.411 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout { 2024-04-26T23:34:02.411 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "fsid": "3c8453ae-0425-11ef-bc93-c7b262605968", 2024-04-26T23:34:02.411 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T23:34:02.413 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T23:34:02.413 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T23:34:02.413 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T23:34:02.413 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:34:02.413 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T23:34:02.413 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T23:34:02.413 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 0 2024-04-26T23:34:02.413 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout ], 2024-04-26T23:34:02.414 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T23:34:02.414 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T23:34:02.414 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout ], 2024-04-26T23:34:02.414 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-04-26T23:34:02.414 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T23:34:02.414 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T23:34:02.414 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T23:34:02.414 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T23:34:02.414 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:34:02.414 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T23:34:02.414 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T23:34:02.414 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T23:34:02.414 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T23:34:02.423 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T23:34:02.423 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T23:34:02.423 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T23:34:02.423 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T23:34:02.423 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:34:02.423 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T23:34:02.423 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T23:34:02.423 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T23:34:02.423 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T23:34:02.423 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T23:34:02.424 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T23:34:02.424 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T23:34:02.424 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T23:34:02.424 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T23:34:02.424 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:34:02.424 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T23:34:02.424 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T23:34:02.424 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T23:34:02.424 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T23:34:02.424 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:34:02.424 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T23:34:02.424 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-26T23:34:02.424 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T23:34:02.424 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T23:34:02.425 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T23:34:02.425 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T23:34:02.425 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T23:34:02.425 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout ], 2024-04-26T23:34:02.425 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T23:34:02.425 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:34:02.425 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T23:34:02.425 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T23:34:02.425 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T23:33:48.088871+0000", 2024-04-26T23:34:02.425 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T23:34:02.425 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:34:02.425 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T23:34:02.425 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout } 2024-04-26T23:34:02.425 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:01.589+0000 7f6edbe00700 1 Processor -- start 2024-04-26T23:34:02.425 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:01.589+0000 7f6edbe00700 1 -- start start 2024-04-26T23:34:02.425 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:01.590+0000 7f6edbe00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ed4107c90 0x7f6ed41080b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:02.426 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:01.590+0000 7f6edbe00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f6ed4108680 con 0x7f6ed4107c90 2024-04-26T23:34:02.426 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:01.590+0000 7f6ed9400700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ed4107c90 0x7f6ed41080b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:02.426 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:01.590+0000 7f6ed9400700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ed4107c90 0x7f6ed41080b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37214/0 (socket says 172.21.15.72:37214) 2024-04-26T23:34:02.426 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:01.590+0000 7f6ed9400700 1 -- 172.21.15.72:0/2882793879 learned_addr learned my addr 172.21.15.72:0/2882793879 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:34:02.426 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:01.591+0000 7f6ed9400700 1 -- 172.21.15.72:0/2882793879 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ed4108e90 con 0x7f6ed4107c90 2024-04-26T23:34:02.426 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:01.591+0000 7f6ed9400700 1 --2- 172.21.15.72:0/2882793879 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ed4107c90 0x7f6ed41080b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6ec800bb40 tx=0x7f6ec800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b51da2ce04e34084 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:02.426 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:01.591+0000 7f6ed3e00700 1 -- 172.21.15.72:0/2882793879 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6ec800dbe0 con 0x7f6ed4107c90 2024-04-26T23:34:02.426 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:01.592+0000 7f6ed3e00700 1 -- 172.21.15.72:0/2882793879 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6ec80044c0 con 0x7f6ed4107c90 2024-04-26T23:34:02.426 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:01.592+0000 7f6ed3e00700 1 -- 172.21.15.72:0/2882793879 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6ec8010470 con 0x7f6ed4107c90 2024-04-26T23:34:02.426 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:01.592+0000 7f6edbe00700 1 -- 172.21.15.72:0/2882793879 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ed4107c90 msgr2=0x7f6ed41080b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:02.426 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:01.592+0000 7f6edbe00700 1 --2- 172.21.15.72:0/2882793879 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ed4107c90 0x7f6ed41080b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6ec800bb40 tx=0x7f6ec800be50 comp rx=0 tx=0).stop 2024-04-26T23:34:02.426 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:01.592+0000 7f6edbe00700 1 -- 172.21.15.72:0/2882793879 shutdown_connections 2024-04-26T23:34:02.426 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:01.592+0000 7f6edbe00700 1 --2- 172.21.15.72:0/2882793879 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ed4107c90 0x7f6ed41080b0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:02.427 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:01.592+0000 7f6edbe00700 1 -- 172.21.15.72:0/2882793879 >> 172.21.15.72:0/2882793879 conn(0x7f6ed41031f0 msgr2=0x7f6ed4105670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:34:02.427 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:01.593+0000 7f6edbe00700 1 -- 172.21.15.72:0/2882793879 shutdown_connections 2024-04-26T23:34:02.427 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:01.593+0000 7f6edbe00700 1 -- 172.21.15.72:0/2882793879 wait complete. 2024-04-26T23:34:02.427 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:01.594+0000 7f6edbe00700 1 Processor -- start 2024-04-26T23:34:02.427 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:01.594+0000 7f6edbe00700 1 -- start start 2024-04-26T23:34:02.427 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:01.594+0000 7f6edbe00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ed4107c90 0x7f6ed419aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:02.427 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:01.594+0000 7f6edbe00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f6ed407bfc0 con 0x7f6ed4107c90 2024-04-26T23:34:02.427 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:01.594+0000 7f6ed9400700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ed4107c90 0x7f6ed419aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:02.427 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:01.594+0000 7f6ed9400700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ed4107c90 0x7f6ed419aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37216/0 (socket says 172.21.15.72:37216) 2024-04-26T23:34:02.427 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:01.594+0000 7f6ed9400700 1 -- 172.21.15.72:0/922526335 learned_addr learned my addr 172.21.15.72:0/922526335 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:34:02.427 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:01.595+0000 7f6ed9400700 1 -- 172.21.15.72:0/922526335 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ed419b030 con 0x7f6ed4107c90 2024-04-26T23:34:02.427 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:01.595+0000 7f6ed9400700 1 --2- 172.21.15.72:0/922526335 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ed4107c90 0x7f6ed419aaf0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6ec800b340 tx=0x7f6ec8003dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:02.427 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:01.595+0000 7f6ed2000700 1 -- 172.21.15.72:0/922526335 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6ec8027070 con 0x7f6ed4107c90 2024-04-26T23:34:02.427 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:01.595+0000 7f6ed2000700 1 -- 172.21.15.72:0/922526335 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6ec8010bb0 con 0x7f6ed4107c90 2024-04-26T23:34:02.427 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:01.595+0000 7f6edbe00700 1 -- 172.21.15.72:0/922526335 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ed419b2c0 con 0x7f6ed4107c90 2024-04-26T23:34:02.428 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:01.595+0000 7f6ed2000700 1 -- 172.21.15.72:0/922526335 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6ec8022ba0 con 0x7f6ed4107c90 2024-04-26T23:34:02.428 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:01.595+0000 7f6edbe00700 1 -- 172.21.15.72:0/922526335 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ed419df60 con 0x7f6ed4107c90 2024-04-26T23:34:02.428 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:01.596+0000 7f6ed2000700 1 -- 172.21.15.72:0/922526335 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6ec802a070 con 0x7f6ed4107c90 2024-04-26T23:34:02.428 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:01.596+0000 7f6ed2000700 1 -- 172.21.15.72:0/922526335 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f6ec801f070 con 0x7f6ed4107c90 2024-04-26T23:34:02.428 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:01.597+0000 7f6edbe00700 1 -- 172.21.15.72:0/922526335 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ed4064a80 con 0x7f6ed4107c90 2024-04-26T23:34:02.428 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:01.599+0000 7f6ed2000700 1 -- 172.21.15.72:0/922526335 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f6ec8031030 con 0x7f6ed4107c90 2024-04-26T23:34:02.428 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:01.694+0000 7f6edbe00700 1 -- 172.21.15.72:0/922526335 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f6ed4194050 con 0x7f6ed4107c90 2024-04-26T23:34:02.428 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:01.695+0000 7f6ed2000700 1 -- 172.21.15.72:0/922526335 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f6ec8015070 con 0x7f6ed4107c90 2024-04-26T23:34:02.428 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:01.697+0000 7f6edbe00700 1 -- 172.21.15.72:0/922526335 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ed4107c90 msgr2=0x7f6ed419aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:02.428 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:01.697+0000 7f6edbe00700 1 --2- 172.21.15.72:0/922526335 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ed4107c90 0x7f6ed419aaf0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6ec800b340 tx=0x7f6ec8003dc0 comp rx=0 tx=0).stop 2024-04-26T23:34:02.428 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:01.697+0000 7f6edbe00700 1 -- 172.21.15.72:0/922526335 shutdown_connections 2024-04-26T23:34:02.428 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:01.697+0000 7f6edbe00700 1 --2- 172.21.15.72:0/922526335 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ed4107c90 0x7f6ed419aaf0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:02.428 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:01.697+0000 7f6edbe00700 1 -- 172.21.15.72:0/922526335 >> 172.21.15.72:0/922526335 conn(0x7f6ed41031f0 msgr2=0x7f6ed4191110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:34:02.428 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:01.697+0000 7f6edbe00700 1 -- 172.21.15.72:0/922526335 shutdown_connections 2024-04-26T23:34:02.428 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:01.697+0000 7f6edbe00700 1 -- 172.21.15.72:0/922526335 wait complete. 2024-04-26T23:34:02.429 INFO:teuthology.orchestra.run.smithi072.stdout:mgr not available, waiting (1/15)... 2024-04-26T23:34:03.873 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:03 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:03.583+0000 7f19c4042200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T23:34:03.873 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:03 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:03.807+0000 7f19c4042200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T23:34:04.293 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:04 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:04.027+0000 7f19c4042200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T23:34:04.727 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:04 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:04.396+0000 7f19c4042200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T23:34:04.727 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:04 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:04.520+0000 7f19c4042200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T23:34:05.020 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:04 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:04.779+0000 7f19c4042200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T23:34:05.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:05 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/2890329620' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T23:34:05.293 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:05 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:05.036+0000 7f19c4042200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T23:34:05.828 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 2024-04-26T23:34:05.828 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout { 2024-04-26T23:34:05.829 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "fsid": "3c8453ae-0425-11ef-bc93-c7b262605968", 2024-04-26T23:34:05.829 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T23:34:05.829 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T23:34:05.829 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T23:34:05.829 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T23:34:05.829 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:34:05.829 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T23:34:05.829 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T23:34:05.829 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 0 2024-04-26T23:34:05.829 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout ], 2024-04-26T23:34:05.829 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T23:34:05.829 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T23:34:05.831 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout ], 2024-04-26T23:34:05.831 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-04-26T23:34:05.831 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T23:34:05.831 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T23:34:05.831 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T23:34:05.831 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T23:34:05.831 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:34:05.831 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T23:34:05.831 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T23:34:05.831 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T23:34:05.831 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T23:34:05.831 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T23:34:05.841 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T23:34:05.841 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T23:34:05.841 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T23:34:05.841 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:34:05.841 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T23:34:05.841 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T23:34:05.841 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T23:34:05.841 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T23:34:05.841 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T23:34:05.842 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T23:34:05.842 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T23:34:05.842 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T23:34:05.842 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T23:34:05.842 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:34:05.842 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T23:34:05.842 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T23:34:05.842 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T23:34:05.842 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T23:34:05.842 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:34:05.842 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T23:34:05.842 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-26T23:34:05.842 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T23:34:05.842 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T23:34:05.843 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T23:34:05.843 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T23:34:05.843 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T23:34:05.843 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout ], 2024-04-26T23:34:05.843 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T23:34:05.843 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:34:05.843 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T23:34:05.843 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T23:34:05.843 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T23:33:48.088871+0000", 2024-04-26T23:34:05.843 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T23:34:05.843 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:34:05.843 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T23:34:05.843 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout } 2024-04-26T23:34:05.843 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:05.016+0000 7fd9ffc00700 1 Processor -- start 2024-04-26T23:34:05.844 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:05.016+0000 7fd9ffc00700 1 -- start start 2024-04-26T23:34:05.844 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:05.017+0000 7fd9ffc00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9f807eed0 0x7fd9f807d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:05.844 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:05.017+0000 7fd9ffc00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f807d900 con 0x7fd9f807eed0 2024-04-26T23:34:05.844 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:05.017+0000 7fd9fd200700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9f807eed0 0x7fd9f807d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:05.844 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:05.018+0000 7fd9fd200700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9f807eed0 0x7fd9f807d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37224/0 (socket says 172.21.15.72:37224) 2024-04-26T23:34:05.844 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:05.018+0000 7fd9fd200700 1 -- 172.21.15.72:0/2970026892 learned_addr learned my addr 172.21.15.72:0/2970026892 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:34:05.844 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:05.018+0000 7fd9fd200700 1 -- 172.21.15.72:0/2970026892 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9f807da80 con 0x7fd9f807eed0 2024-04-26T23:34:05.844 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:05.018+0000 7fd9fd200700 1 --2- 172.21.15.72:0/2970026892 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9f807eed0 0x7fd9f807d330 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec0098a0 tx=0x7fd9ec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=758fd94a17c76e39 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:05.844 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:05.019+0000 7fd9f7400700 1 -- 172.21.15.72:0/2970026892 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd9ec004160 con 0x7fd9f807eed0 2024-04-26T23:34:05.844 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:05.019+0000 7fd9f7400700 1 -- 172.21.15.72:0/2970026892 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd9ec004300 con 0x7fd9f807eed0 2024-04-26T23:34:05.844 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:05.019+0000 7fd9f7400700 1 -- 172.21.15.72:0/2970026892 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd9ec011af0 con 0x7fd9f807eed0 2024-04-26T23:34:05.844 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:05.019+0000 7fd9ffc00700 1 -- 172.21.15.72:0/2970026892 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9f807eed0 msgr2=0x7fd9f807d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:05.844 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:05.019+0000 7fd9ffc00700 1 --2- 172.21.15.72:0/2970026892 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9f807eed0 0x7fd9f807d330 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec0098a0 tx=0x7fd9ec009bb0 comp rx=0 tx=0).stop 2024-04-26T23:34:05.845 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:05.019+0000 7fd9ffc00700 1 -- 172.21.15.72:0/2970026892 shutdown_connections 2024-04-26T23:34:05.845 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:05.019+0000 7fd9ffc00700 1 --2- 172.21.15.72:0/2970026892 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9f807eed0 0x7fd9f807d330 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:05.845 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:05.019+0000 7fd9ffc00700 1 -- 172.21.15.72:0/2970026892 >> 172.21.15.72:0/2970026892 conn(0x7fd9f8104330 msgr2=0x7fd9f8106770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:34:05.845 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:05.020+0000 7fd9ffc00700 1 -- 172.21.15.72:0/2970026892 shutdown_connections 2024-04-26T23:34:05.845 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:05.020+0000 7fd9ffc00700 1 -- 172.21.15.72:0/2970026892 wait complete. 2024-04-26T23:34:05.845 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:05.021+0000 7fd9ffc00700 1 Processor -- start 2024-04-26T23:34:05.845 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:05.021+0000 7fd9ffc00700 1 -- start start 2024-04-26T23:34:05.845 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:05.021+0000 7fd9ffc00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9f807eed0 0x7fd9f819ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:05.845 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:05.021+0000 7fd9ffc00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f810f350 con 0x7fd9f807eed0 2024-04-26T23:34:05.845 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:05.021+0000 7fd9fd200700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9f807eed0 0x7fd9f819ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:05.845 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:05.021+0000 7fd9fd200700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9f807eed0 0x7fd9f819ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37238/0 (socket says 172.21.15.72:37238) 2024-04-26T23:34:05.845 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:05.021+0000 7fd9fd200700 1 -- 172.21.15.72:0/2890329620 learned_addr learned my addr 172.21.15.72:0/2890329620 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:34:05.845 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:05.022+0000 7fd9fd200700 1 -- 172.21.15.72:0/2890329620 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9f819f460 con 0x7fd9f807eed0 2024-04-26T23:34:05.845 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:05.022+0000 7fd9fd200700 1 --2- 172.21.15.72:0/2890329620 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9f807eed0 0x7fd9f819ef20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec009fb0 tx=0x7fd9ec011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:05.846 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:05.022+0000 7fd9f5600700 1 -- 172.21.15.72:0/2890329620 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd9ec0036a0 con 0x7fd9f807eed0 2024-04-26T23:34:05.846 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:05.022+0000 7fd9f5600700 1 -- 172.21.15.72:0/2890329620 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd9ec01a430 con 0x7fd9f807eed0 2024-04-26T23:34:05.846 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:05.022+0000 7fd9f5600700 1 -- 172.21.15.72:0/2890329620 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd9ec019550 con 0x7fd9f807eed0 2024-04-26T23:34:05.846 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:05.022+0000 7fd9ffc00700 1 -- 172.21.15.72:0/2890329620 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9f819f6f0 con 0x7fd9f807eed0 2024-04-26T23:34:05.846 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:05.023+0000 7fd9ffc00700 1 -- 172.21.15.72:0/2890329620 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9f81a23e0 con 0x7fd9f807eed0 2024-04-26T23:34:05.846 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:05.023+0000 7fd9f5600700 1 -- 172.21.15.72:0/2890329620 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd9ec019b90 con 0x7fd9f807eed0 2024-04-26T23:34:05.846 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:05.023+0000 7fd9f5600700 1 -- 172.21.15.72:0/2890329620 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd9ec024730 con 0x7fd9f807eed0 2024-04-26T23:34:05.846 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:05.024+0000 7fd9ffc00700 1 -- 172.21.15.72:0/2890329620 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9f8064a80 con 0x7fd9f807eed0 2024-04-26T23:34:05.846 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:05.026+0000 7fd9f5600700 1 -- 172.21.15.72:0/2890329620 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fd9ec043ec0 con 0x7fd9f807eed0 2024-04-26T23:34:05.846 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:05.105+0000 7fd9ffc00700 1 -- 172.21.15.72:0/2890329620 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fd9f81a2920 con 0x7fd9f807eed0 2024-04-26T23:34:05.846 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:05.106+0000 7fd9f5600700 1 -- 172.21.15.72:0/2890329620 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fd9ec032020 con 0x7fd9f807eed0 2024-04-26T23:34:05.846 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:05.107+0000 7fd9ffc00700 1 -- 172.21.15.72:0/2890329620 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9f807eed0 msgr2=0x7fd9f819ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:05.846 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:05.107+0000 7fd9ffc00700 1 --2- 172.21.15.72:0/2890329620 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9f807eed0 0x7fd9f819ef20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec009fb0 tx=0x7fd9ec011940 comp rx=0 tx=0).stop 2024-04-26T23:34:05.846 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:05.107+0000 7fd9ffc00700 1 -- 172.21.15.72:0/2890329620 shutdown_connections 2024-04-26T23:34:05.847 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:05.107+0000 7fd9ffc00700 1 --2- 172.21.15.72:0/2890329620 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9f807eed0 0x7fd9f819ef20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:05.847 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:05.107+0000 7fd9ffc00700 1 -- 172.21.15.72:0/2890329620 >> 172.21.15.72:0/2890329620 conn(0x7fd9f8104330 msgr2=0x7fd9f8104eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:34:05.847 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:05.108+0000 7fd9ffc00700 1 -- 172.21.15.72:0/2890329620 shutdown_connections 2024-04-26T23:34:05.847 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:05.108+0000 7fd9ffc00700 1 -- 172.21.15.72:0/2890329620 wait complete. 2024-04-26T23:34:05.847 INFO:teuthology.orchestra.run.smithi072.stdout:mgr not available, waiting (2/15)... 2024-04-26T23:34:06.292 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:05 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:05.844+0000 7f19c4042200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T23:34:06.780 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:06 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:06.386+0000 7f19c4042200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T23:34:06.781 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:06 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:06.513+0000 7f19c4042200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T23:34:07.043 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:06 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:06.777+0000 7f19c4042200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T23:34:07.793 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:07 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:07.431+0000 7f19c4042200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T23:34:07.793 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:07 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:07.553+0000 7f19c4042200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T23:34:08.151 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:07 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:07.878+0000 7f19c4042200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T23:34:08.151 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:08 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:08.065+0000 7f19c4042200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T23:34:08.402 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:08 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:08.198+0000 7f19c4042200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T23:34:08.403 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:08 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:08.349+0000 7f19c4042200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T23:34:08.731 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:08 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/3110977003' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T23:34:09.011 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:08 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:08.727+0000 7f19c4042200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T23:34:09.218 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 2024-04-26T23:34:09.218 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout { 2024-04-26T23:34:09.219 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "fsid": "3c8453ae-0425-11ef-bc93-c7b262605968", 2024-04-26T23:34:09.219 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T23:34:09.219 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T23:34:09.219 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T23:34:09.219 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T23:34:09.219 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:34:09.219 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T23:34:09.219 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T23:34:09.219 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 0 2024-04-26T23:34:09.219 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout ], 2024-04-26T23:34:09.219 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T23:34:09.219 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T23:34:09.219 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout ], 2024-04-26T23:34:09.219 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-04-26T23:34:09.219 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T23:34:09.220 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T23:34:09.220 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T23:34:09.220 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T23:34:09.220 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:34:09.220 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T23:34:09.220 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T23:34:09.220 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T23:34:09.220 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T23:34:09.228 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T23:34:09.228 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T23:34:09.228 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T23:34:09.228 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T23:34:09.228 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:34:09.228 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T23:34:09.229 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T23:34:09.229 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T23:34:09.229 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T23:34:09.229 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T23:34:09.229 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T23:34:09.229 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T23:34:09.229 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T23:34:09.229 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T23:34:09.229 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:34:09.229 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T23:34:09.229 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T23:34:09.229 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T23:34:09.229 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T23:34:09.229 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:34:09.229 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T23:34:09.230 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-26T23:34:09.230 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T23:34:09.230 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T23:34:09.230 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T23:34:09.230 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T23:34:09.230 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T23:34:09.230 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout ], 2024-04-26T23:34:09.230 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T23:34:09.230 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:34:09.230 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T23:34:09.230 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T23:34:09.230 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T23:33:48.088871+0000", 2024-04-26T23:34:09.230 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T23:34:09.230 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:34:09.231 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T23:34:09.231 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout } 2024-04-26T23:34:09.231 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:08.399+0000 7f50e0e00700 1 Processor -- start 2024-04-26T23:34:09.231 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:08.399+0000 7f50e0e00700 1 -- start start 2024-04-26T23:34:09.231 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:08.399+0000 7f50e0e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50dc07ee90 0x7f50dc07d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:09.231 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:08.399+0000 7f50e0e00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f50dc07d8c0 con 0x7f50dc07ee90 2024-04-26T23:34:09.231 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:08.400+0000 7f50d9e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50dc07ee90 0x7f50dc07d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:09.231 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:08.400+0000 7f50d9e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50dc07ee90 0x7f50dc07d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37968/0 (socket says 172.21.15.72:37968) 2024-04-26T23:34:09.231 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:08.400+0000 7f50d9e00700 1 -- 172.21.15.72:0/2590922699 learned_addr learned my addr 172.21.15.72:0/2590922699 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:34:09.231 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:08.400+0000 7f50d9e00700 1 -- 172.21.15.72:0/2590922699 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50dc07da40 con 0x7f50dc07ee90 2024-04-26T23:34:09.231 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:08.401+0000 7f50d9e00700 1 --2- 172.21.15.72:0/2590922699 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50dc07ee90 0x7f50dc07d2f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f50cc0098a0 tx=0x7f50cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=376be18d399b578d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:09.231 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:08.401+0000 7f50d8a00700 1 -- 172.21.15.72:0/2590922699 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f50cc004160 con 0x7f50dc07ee90 2024-04-26T23:34:09.231 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:08.401+0000 7f50d8a00700 1 -- 172.21.15.72:0/2590922699 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f50cc004300 con 0x7f50dc07ee90 2024-04-26T23:34:09.231 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:08.401+0000 7f50d8a00700 1 -- 172.21.15.72:0/2590922699 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f50cc011af0 con 0x7f50dc07ee90 2024-04-26T23:34:09.232 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:08.402+0000 7f50e0e00700 1 -- 172.21.15.72:0/2590922699 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50dc07ee90 msgr2=0x7f50dc07d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:09.232 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:08.402+0000 7f50e0e00700 1 --2- 172.21.15.72:0/2590922699 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50dc07ee90 0x7f50dc07d2f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f50cc0098a0 tx=0x7f50cc009bb0 comp rx=0 tx=0).stop 2024-04-26T23:34:09.232 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:08.402+0000 7f50e0e00700 1 -- 172.21.15.72:0/2590922699 shutdown_connections 2024-04-26T23:34:09.232 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:08.402+0000 7f50e0e00700 1 --2- 172.21.15.72:0/2590922699 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50dc07ee90 0x7f50dc07d2f0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:09.232 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:08.402+0000 7f50e0e00700 1 -- 172.21.15.72:0/2590922699 >> 172.21.15.72:0/2590922699 conn(0x7f50dc1042d0 msgr2=0x7f50dc106750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:34:09.232 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:08.402+0000 7f50e0e00700 1 -- 172.21.15.72:0/2590922699 shutdown_connections 2024-04-26T23:34:09.232 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:08.402+0000 7f50e0e00700 1 -- 172.21.15.72:0/2590922699 wait complete. 2024-04-26T23:34:09.232 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:08.403+0000 7f50e0e00700 1 Processor -- start 2024-04-26T23:34:09.232 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:08.403+0000 7f50e0e00700 1 -- start start 2024-04-26T23:34:09.232 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:08.403+0000 7f50e0e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50dc07ee90 0x7f50dc19ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:09.232 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:08.403+0000 7f50e0e00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f50dc10f320 con 0x7f50dc07ee90 2024-04-26T23:34:09.232 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:08.403+0000 7f50d9e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50dc07ee90 0x7f50dc19ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:09.232 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:08.403+0000 7f50d9e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50dc07ee90 0x7f50dc19ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37980/0 (socket says 172.21.15.72:37980) 2024-04-26T23:34:09.232 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:08.403+0000 7f50d9e00700 1 -- 172.21.15.72:0/3110977003 learned_addr learned my addr 172.21.15.72:0/3110977003 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:34:09.232 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:08.404+0000 7f50d9e00700 1 -- 172.21.15.72:0/3110977003 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50dc19f470 con 0x7f50dc07ee90 2024-04-26T23:34:09.233 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:08.404+0000 7f50d9e00700 1 --2- 172.21.15.72:0/3110977003 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50dc07ee90 0x7f50dc19ef30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f50cc0117b0 tx=0x7f50cc011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:09.233 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:08.404+0000 7f50d2a00700 1 -- 172.21.15.72:0/3110977003 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f50cc003af0 con 0x7f50dc07ee90 2024-04-26T23:34:09.233 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:08.404+0000 7f50e0e00700 1 -- 172.21.15.72:0/3110977003 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50dc19f700 con 0x7f50dc07ee90 2024-04-26T23:34:09.233 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:08.404+0000 7f50d2a00700 1 -- 172.21.15.72:0/3110977003 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f50cc020070 con 0x7f50dc07ee90 2024-04-26T23:34:09.233 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:08.404+0000 7f50e0e00700 1 -- 172.21.15.72:0/3110977003 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50dc1a23f0 con 0x7f50dc07ee90 2024-04-26T23:34:09.233 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:08.405+0000 7f50d2a00700 1 -- 172.21.15.72:0/3110977003 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f50cc02a060 con 0x7f50dc07ee90 2024-04-26T23:34:09.233 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:08.405+0000 7f50d2a00700 1 -- 172.21.15.72:0/3110977003 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f50cc019a10 con 0x7f50dc07ee90 2024-04-26T23:34:09.233 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:08.405+0000 7f50d2a00700 1 -- 172.21.15.72:0/3110977003 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f50cc02e6e0 con 0x7f50dc07ee90 2024-04-26T23:34:09.233 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:08.406+0000 7f50e0e00700 1 -- 172.21.15.72:0/3110977003 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50dc064a80 con 0x7f50dc07ee90 2024-04-26T23:34:09.233 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:08.410+0000 7f50d2a00700 1 -- 172.21.15.72:0/3110977003 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f50cc025dc0 con 0x7f50dc07ee90 2024-04-26T23:34:09.233 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:08.490+0000 7f50e0e00700 1 -- 172.21.15.72:0/3110977003 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f50dc1a25c0 con 0x7f50dc07ee90 2024-04-26T23:34:09.233 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:08.491+0000 7f50d2a00700 1 -- 172.21.15.72:0/3110977003 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1240 (secure 0 0 0) 0x7f50cc01a9c0 con 0x7f50dc07ee90 2024-04-26T23:34:09.233 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:08.492+0000 7f50e0e00700 1 -- 172.21.15.72:0/3110977003 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50dc07ee90 msgr2=0x7f50dc19ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:09.233 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:08.492+0000 7f50e0e00700 1 --2- 172.21.15.72:0/3110977003 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50dc07ee90 0x7f50dc19ef30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f50cc0117b0 tx=0x7f50cc011b70 comp rx=0 tx=0).stop 2024-04-26T23:34:09.234 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:08.492+0000 7f50e0e00700 1 -- 172.21.15.72:0/3110977003 shutdown_connections 2024-04-26T23:34:09.234 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:08.492+0000 7f50e0e00700 1 --2- 172.21.15.72:0/3110977003 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50dc07ee90 0x7f50dc19ef30 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:09.234 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:08.492+0000 7f50e0e00700 1 -- 172.21.15.72:0/3110977003 >> 172.21.15.72:0/3110977003 conn(0x7f50dc1042d0 msgr2=0x7f50dc104ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:34:09.234 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:08.492+0000 7f50e0e00700 1 -- 172.21.15.72:0/3110977003 shutdown_connections 2024-04-26T23:34:09.234 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:08.492+0000 7f50e0e00700 1 -- 172.21.15.72:0/3110977003 wait complete. 2024-04-26T23:34:09.234 INFO:teuthology.orchestra.run.smithi072.stdout:mgr not available, waiting (3/15)... 2024-04-26T23:34:09.768 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:09 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:09.486+0000 7f19c4042200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T23:34:10.043 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:09 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:09.764+0000 7f19c4042200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T23:34:10.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:09 smithi072 ceph-mon[28184]: Activating manager daemon x 2024-04-26T23:34:10.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:09 smithi072 ceph-mon[28184]: mgrmap e2: x(active, starting, since 0.0127172s) 2024-04-26T23:34:10.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:09 smithi072 ceph-mon[28184]: from='mgr.14100 172.21.15.72:0/763668554' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T23:34:10.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:09 smithi072 ceph-mon[28184]: from='mgr.14100 172.21.15.72:0/763668554' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T23:34:10.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:09 smithi072 ceph-mon[28184]: from='mgr.14100 172.21.15.72:0/763668554' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T23:34:10.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:09 smithi072 ceph-mon[28184]: from='mgr.14100 172.21.15.72:0/763668554' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T23:34:10.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:09 smithi072 ceph-mon[28184]: from='mgr.14100 172.21.15.72:0/763668554' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T23:34:10.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:09 smithi072 ceph-mon[28184]: Manager daemon x is now available 2024-04-26T23:34:10.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:09 smithi072 ceph-mon[28184]: from='mgr.14100 172.21.15.72:0/763668554' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T23:34:10.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:09 smithi072 ceph-mon[28184]: from='mgr.14100 172.21.15.72:0/763668554' entity='mgr.x' 2024-04-26T23:34:10.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:09 smithi072 ceph-mon[28184]: from='mgr.14100 172.21.15.72:0/763668554' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T23:34:11.292 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:10 smithi072 ceph-mon[28184]: from='mgr.14100 172.21.15.72:0/763668554' entity='mgr.x' 2024-04-26T23:34:11.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:10 smithi072 ceph-mon[28184]: from='mgr.14100 172.21.15.72:0/763668554' entity='mgr.x' 2024-04-26T23:34:11.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:10 smithi072 ceph-mon[28184]: mgrmap e3: x(active, since 1.02269s) 2024-04-26T23:34:12.755 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 2024-04-26T23:34:12.756 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout { 2024-04-26T23:34:12.756 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "fsid": "3c8453ae-0425-11ef-bc93-c7b262605968", 2024-04-26T23:34:12.756 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T23:34:12.756 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T23:34:12.756 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T23:34:12.756 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T23:34:12.756 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:34:12.756 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T23:34:12.757 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T23:34:12.759 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 0 2024-04-26T23:34:12.759 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout ], 2024-04-26T23:34:12.759 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T23:34:12.759 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T23:34:12.759 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout ], 2024-04-26T23:34:12.759 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-04-26T23:34:12.759 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T23:34:12.759 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T23:34:12.759 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T23:34:12.759 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T23:34:12.759 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:34:12.759 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T23:34:12.760 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T23:34:12.760 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T23:34:12.760 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T23:34:12.760 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T23:34:12.769 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T23:34:12.769 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T23:34:12.770 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T23:34:12.770 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:34:12.770 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T23:34:12.770 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T23:34:12.770 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T23:34:12.770 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T23:34:12.770 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T23:34:12.770 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T23:34:12.770 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T23:34:12.770 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T23:34:12.770 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T23:34:12.771 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:34:12.771 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T23:34:12.771 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T23:34:12.771 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T23:34:12.771 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T23:34:12.771 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:34:12.771 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T23:34:12.771 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-26T23:34:12.771 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T23:34:12.771 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T23:34:12.771 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T23:34:12.771 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T23:34:12.771 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T23:34:12.772 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout ], 2024-04-26T23:34:12.772 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T23:34:12.772 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:34:12.772 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T23:34:12.772 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T23:34:12.772 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T23:33:48.088871+0000", 2024-04-26T23:34:12.772 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T23:34:12.772 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-04-26T23:34:12.772 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T23:34:12.772 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout } 2024-04-26T23:34:12.772 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:11.808+0000 7ff13a400700 1 Processor -- start 2024-04-26T23:34:12.772 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:11.809+0000 7ff13a400700 1 -- start start 2024-04-26T23:34:12.772 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:11.809+0000 7ff13a400700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff13407f760 0x7ff13407dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:12.772 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:11.809+0000 7ff13a400700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7ff13407e190 con 0x7ff13407f760 2024-04-26T23:34:12.773 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:11.809+0000 7ff133400700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff13407f760 0x7ff13407dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:12.773 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:11.809+0000 7ff133400700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff13407f760 0x7ff13407dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38054/0 (socket says 172.21.15.72:38054) 2024-04-26T23:34:12.773 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:11.809+0000 7ff133400700 1 -- 172.21.15.72:0/3428643795 learned_addr learned my addr 172.21.15.72:0/3428643795 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:34:12.773 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:11.810+0000 7ff133400700 1 -- 172.21.15.72:0/3428643795 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff13407e310 con 0x7ff13407f760 2024-04-26T23:34:12.773 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:11.810+0000 7ff133400700 1 --2- 172.21.15.72:0/3428643795 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff13407f760 0x7ff13407dbc0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff1240098a0 tx=0x7ff124009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54ac061380d71996 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:12.773 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:11.810+0000 7ff132000700 1 -- 172.21.15.72:0/3428643795 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff124004160 con 0x7ff13407f760 2024-04-26T23:34:12.773 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:11.810+0000 7ff132000700 1 -- 172.21.15.72:0/3428643795 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7ff124004300 con 0x7ff13407f760 2024-04-26T23:34:12.773 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:11.810+0000 7ff132000700 1 -- 172.21.15.72:0/3428643795 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff124011af0 con 0x7ff13407f760 2024-04-26T23:34:12.773 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:11.811+0000 7ff13a400700 1 -- 172.21.15.72:0/3428643795 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff13407f760 msgr2=0x7ff13407dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:12.773 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:11.811+0000 7ff13a400700 1 --2- 172.21.15.72:0/3428643795 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff13407f760 0x7ff13407dbc0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff1240098a0 tx=0x7ff124009bb0 comp rx=0 tx=0).stop 2024-04-26T23:34:12.773 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:11.811+0000 7ff13a400700 1 -- 172.21.15.72:0/3428643795 shutdown_connections 2024-04-26T23:34:12.773 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:11.811+0000 7ff13a400700 1 --2- 172.21.15.72:0/3428643795 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff13407f760 0x7ff13407dbc0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:12.774 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:11.811+0000 7ff13a400700 1 -- 172.21.15.72:0/3428643795 >> 172.21.15.72:0/3428643795 conn(0x7ff134104310 msgr2=0x7ff134106750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:34:12.774 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:11.811+0000 7ff13a400700 1 -- 172.21.15.72:0/3428643795 shutdown_connections 2024-04-26T23:34:12.774 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:11.811+0000 7ff13a400700 1 -- 172.21.15.72:0/3428643795 wait complete. 2024-04-26T23:34:12.774 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:11.812+0000 7ff13a400700 1 Processor -- start 2024-04-26T23:34:12.774 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:11.812+0000 7ff13a400700 1 -- start start 2024-04-26T23:34:12.774 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:11.813+0000 7ff13a400700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff13407f760 0x7ff13419ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:12.774 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:11.813+0000 7ff13a400700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7ff13410f330 con 0x7ff13407f760 2024-04-26T23:34:12.774 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:11.813+0000 7ff133400700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff13407f760 0x7ff13419ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:12.774 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:11.813+0000 7ff133400700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff13407f760 0x7ff13419ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38062/0 (socket says 172.21.15.72:38062) 2024-04-26T23:34:12.774 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:11.813+0000 7ff133400700 1 -- 172.21.15.72:0/2365062208 learned_addr learned my addr 172.21.15.72:0/2365062208 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:34:12.774 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:11.813+0000 7ff133400700 1 -- 172.21.15.72:0/2365062208 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff13419f450 con 0x7ff13407f760 2024-04-26T23:34:12.774 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:11.814+0000 7ff133400700 1 --2- 172.21.15.72:0/2365062208 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff13407f760 0x7ff13419ef10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff1240117b0 tx=0x7ff124011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:12.775 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:11.814+0000 7ff12be00700 1 -- 172.21.15.72:0/2365062208 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff124003af0 con 0x7ff13407f760 2024-04-26T23:34:12.775 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:11.814+0000 7ff12be00700 1 -- 172.21.15.72:0/2365062208 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7ff124020070 con 0x7ff13407f760 2024-04-26T23:34:12.775 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:11.814+0000 7ff12be00700 1 -- 172.21.15.72:0/2365062208 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff124019630 con 0x7ff13407f760 2024-04-26T23:34:12.775 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:11.814+0000 7ff13a400700 1 -- 172.21.15.72:0/2365062208 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff13419f6e0 con 0x7ff13407f760 2024-04-26T23:34:12.775 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:11.814+0000 7ff13a400700 1 -- 172.21.15.72:0/2365062208 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1341a23d0 con 0x7ff13407f760 2024-04-26T23:34:12.775 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:11.815+0000 7ff12be00700 1 -- 172.21.15.72:0/2365062208 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 3) ==== 46441+0+0 (secure 0 0 0) 0x7ff12401a430 con 0x7ff13407f760 2024-04-26T23:34:12.775 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:11.815+0000 7ff13a400700 1 -- 172.21.15.72:0/2365062208 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff134064a80 con 0x7ff13407f760 2024-04-26T23:34:12.775 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:11.815+0000 7ff12be00700 1 --2- 172.21.15.72:0/2365062208 >> [v2:172.21.15.72:6800/4175896290,v1:172.21.15.72:6801/4175896290] conn(0x7ff118039490 0x7ff11803b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:12.775 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:11.816+0000 7ff12be00700 1 -- 172.21.15.72:0/2365062208 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff12404d880 con 0x7ff13407f760 2024-04-26T23:34:12.775 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:11.816+0000 7ff132a00700 1 --2- 172.21.15.72:0/2365062208 >> [v2:172.21.15.72:6800/4175896290,v1:172.21.15.72:6801/4175896290] conn(0x7ff118039490 0x7ff11803b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:12.776 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:11.817+0000 7ff132a00700 1 --2- 172.21.15.72:0/2365062208 >> [v2:172.21.15.72:6800/4175896290,v1:172.21.15.72:6801/4175896290] conn(0x7ff118039490 0x7ff11803b950 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff120006fd0 tx=0x7ff120006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:12.776 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:11.822+0000 7ff12be00700 1 -- 172.21.15.72:0/2365062208 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff1240042a0 con 0x7ff13407f760 2024-04-26T23:34:12.776 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:11.828+0000 7ff12be00700 1 -- 172.21.15.72:0/2365062208 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7ff124019850 con 0x7ff13407f760 2024-04-26T23:34:12.776 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:12.111+0000 7ff13a400700 1 -- 172.21.15.72:0/2365062208 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7ff1341a25a0 con 0x7ff13407f760 2024-04-26T23:34:12.776 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:12.112+0000 7ff12be00700 1 -- 172.21.15.72:0/2365062208 <== mon.0 v2:172.21.15.72:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7ff1341a25a0 con 0x7ff13407f760 2024-04-26T23:34:12.776 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:12.115+0000 7ff13a400700 1 -- 172.21.15.72:0/2365062208 >> [v2:172.21.15.72:6800/4175896290,v1:172.21.15.72:6801/4175896290] conn(0x7ff118039490 msgr2=0x7ff11803b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:12.776 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:12.115+0000 7ff13a400700 1 --2- 172.21.15.72:0/2365062208 >> [v2:172.21.15.72:6800/4175896290,v1:172.21.15.72:6801/4175896290] conn(0x7ff118039490 0x7ff11803b950 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff120006fd0 tx=0x7ff120006e40 comp rx=0 tx=0).stop 2024-04-26T23:34:12.776 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:12.115+0000 7ff13a400700 1 -- 172.21.15.72:0/2365062208 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff13407f760 msgr2=0x7ff13419ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:12.776 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:12.115+0000 7ff13a400700 1 --2- 172.21.15.72:0/2365062208 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff13407f760 0x7ff13419ef10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff1240117b0 tx=0x7ff124011b70 comp rx=0 tx=0).stop 2024-04-26T23:34:12.776 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:12.115+0000 7ff13a400700 1 -- 172.21.15.72:0/2365062208 shutdown_connections 2024-04-26T23:34:12.776 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:12.115+0000 7ff13a400700 1 --2- 172.21.15.72:0/2365062208 >> [v2:172.21.15.72:6800/4175896290,v1:172.21.15.72:6801/4175896290] conn(0x7ff118039490 0x7ff11803b950 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:12.776 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:12.115+0000 7ff13a400700 1 --2- 172.21.15.72:0/2365062208 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff13407f760 0x7ff13419ef10 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:12.777 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:12.115+0000 7ff13a400700 1 -- 172.21.15.72:0/2365062208 >> 172.21.15.72:0/2365062208 conn(0x7ff134104310 msgr2=0x7ff134104f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:34:12.777 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:12.116+0000 7ff13a400700 1 -- 172.21.15.72:0/2365062208 shutdown_connections 2024-04-26T23:34:12.777 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:12.116+0000 7ff13a400700 1 -- 172.21.15.72:0/2365062208 wait complete. 2024-04-26T23:34:12.777 INFO:teuthology.orchestra.run.smithi072.stdout:mgr is available 2024-04-26T23:34:13.042 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:12 smithi072 ceph-mon[28184]: mgrmap e4: x(active, since 2s) 2024-04-26T23:34:13.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:12 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/2365062208' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T23:34:13.962 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:13 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/382805678' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-26T23:34:14.277 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 2024-04-26T23:34:14.277 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout [global] 2024-04-26T23:34:14.277 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout fsid = 3c8453ae-0425-11ef-bc93-c7b262605968 2024-04-26T23:34:14.277 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-26T23:34:14.277 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-26T23:34:14.277 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-26T23:34:14.277 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-26T23:34:14.277 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-26T23:34:14.277 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 2024-04-26T23:34:14.277 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-26T23:34:14.277 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-26T23:34:14.277 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 2024-04-26T23:34:14.277 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout [osd] 2024-04-26T23:34:14.278 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-26T23:34:14.278 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-26T23:34:14.278 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-26T23:34:14.278 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-26T23:34:14.278 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-26T23:34:14.278 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.363+0000 7f2989400700 1 Processor -- start 2024-04-26T23:34:14.278 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.363+0000 7f2989400700 1 -- start start 2024-04-26T23:34:14.278 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.363+0000 7f2989400700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2984109800 0x7f298410bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:14.278 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.363+0000 7f2989400700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f298407cea0 con 0x7f2984109800 2024-04-26T23:34:14.278 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.364+0000 7f2982800700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2984109800 0x7f298410bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:14.278 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.364+0000 7f2982800700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2984109800 0x7f298410bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38066/0 (socket says 172.21.15.72:38066) 2024-04-26T23:34:14.278 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.364+0000 7f2982800700 1 -- 172.21.15.72:0/3057487310 learned_addr learned my addr 172.21.15.72:0/3057487310 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:34:14.279 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.364+0000 7f2982800700 1 -- 172.21.15.72:0/3057487310 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f298410c130 con 0x7f2984109800 2024-04-26T23:34:14.279 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.365+0000 7f2982800700 1 --2- 172.21.15.72:0/3057487310 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2984109800 0x7f298410bbf0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f29740098a0 tx=0x7f2974009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af414c949312e162 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:14.279 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.365+0000 7f2981400700 1 -- 172.21.15.72:0/3057487310 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2974004160 con 0x7f2984109800 2024-04-26T23:34:14.279 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.365+0000 7f2981400700 1 -- 172.21.15.72:0/3057487310 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2974004300 con 0x7f2984109800 2024-04-26T23:34:14.279 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.365+0000 7f2981400700 1 -- 172.21.15.72:0/3057487310 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2974011af0 con 0x7f2984109800 2024-04-26T23:34:14.279 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.366+0000 7f2989400700 1 -- 172.21.15.72:0/3057487310 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2984109800 msgr2=0x7f298410bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:14.279 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.366+0000 7f2989400700 1 --2- 172.21.15.72:0/3057487310 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2984109800 0x7f298410bbf0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f29740098a0 tx=0x7f2974009bb0 comp rx=0 tx=0).stop 2024-04-26T23:34:14.279 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.366+0000 7f2989400700 1 -- 172.21.15.72:0/3057487310 shutdown_connections 2024-04-26T23:34:14.279 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.366+0000 7f2989400700 1 --2- 172.21.15.72:0/3057487310 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2984109800 0x7f298410bbf0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:14.279 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.366+0000 7f2989400700 1 -- 172.21.15.72:0/3057487310 >> 172.21.15.72:0/3057487310 conn(0x7f29841031f0 msgr2=0x7f2984105670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:34:14.279 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.366+0000 7f2989400700 1 -- 172.21.15.72:0/3057487310 shutdown_connections 2024-04-26T23:34:14.279 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.366+0000 7f2989400700 1 -- 172.21.15.72:0/3057487310 wait complete. 2024-04-26T23:34:14.279 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.367+0000 7f2989400700 1 Processor -- start 2024-04-26T23:34:14.279 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.367+0000 7f2989400700 1 -- start start 2024-04-26T23:34:14.279 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.367+0000 7f2989400700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2984109800 0x7f2984109080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:14.279 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.367+0000 7f2989400700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f298410d100 con 0x7f2984109800 2024-04-26T23:34:14.280 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.368+0000 7f2982800700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2984109800 0x7f2984109080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:14.280 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.368+0000 7f2982800700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2984109800 0x7f2984109080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38072/0 (socket says 172.21.15.72:38072) 2024-04-26T23:34:14.280 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.368+0000 7f2982800700 1 -- 172.21.15.72:0/382805678 learned_addr learned my addr 172.21.15.72:0/382805678 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:34:14.280 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.368+0000 7f2982800700 1 -- 172.21.15.72:0/382805678 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29841095c0 con 0x7f2984109800 2024-04-26T23:34:14.280 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.368+0000 7f2982800700 1 --2- 172.21.15.72:0/382805678 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2984109800 0x7f2984109080 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2974011ac0 tx=0x7f2974011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:14.280 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.369+0000 7f297b400700 1 -- 172.21.15.72:0/382805678 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2974003970 con 0x7f2984109800 2024-04-26T23:34:14.280 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.369+0000 7f297b400700 1 -- 172.21.15.72:0/382805678 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2974003b10 con 0x7f2984109800 2024-04-26T23:34:14.280 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.369+0000 7f297b400700 1 -- 172.21.15.72:0/382805678 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2974024610 con 0x7f2984109800 2024-04-26T23:34:14.280 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.369+0000 7f2989400700 1 -- 172.21.15.72:0/382805678 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29841077b0 con 0x7f2984109800 2024-04-26T23:34:14.280 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.369+0000 7f2989400700 1 -- 172.21.15.72:0/382805678 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2984107cf0 con 0x7f2984109800 2024-04-26T23:34:14.280 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.370+0000 7f297b400700 1 -- 172.21.15.72:0/382805678 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f297401a430 con 0x7f2984109800 2024-04-26T23:34:14.280 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.370+0000 7f2989400700 1 -- 172.21.15.72:0/382805678 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2984064a80 con 0x7f2984109800 2024-04-26T23:34:14.280 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.370+0000 7f297b400700 1 --2- 172.21.15.72:0/382805678 >> [v2:172.21.15.72:6800/4175896290,v1:172.21.15.72:6801/4175896290] conn(0x7f29680395b0 0x7f296803ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:14.280 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.370+0000 7f297b400700 1 -- 172.21.15.72:0/382805678 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f297404d880 con 0x7f2984109800 2024-04-26T23:34:14.280 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.371+0000 7f2981e00700 1 --2- 172.21.15.72:0/382805678 >> [v2:172.21.15.72:6800/4175896290,v1:172.21.15.72:6801/4175896290] conn(0x7f29680395b0 0x7f296803ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:14.281 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.371+0000 7f2981e00700 1 --2- 172.21.15.72:0/382805678 >> [v2:172.21.15.72:6800/4175896290,v1:172.21.15.72:6801/4175896290] conn(0x7f29680395b0 0x7f296803ba70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2970006fd0 tx=0x7f2970006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:14.281 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.376+0000 7f297b400700 1 -- 172.21.15.72:0/382805678 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f2974016260 con 0x7f2984109800 2024-04-26T23:34:14.281 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.600+0000 7f2989400700 1 -- 172.21.15.72:0/382805678 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f29841084e0 con 0x7f2984109800 2024-04-26T23:34:14.281 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.602+0000 7f297b400700 1 -- 172.21.15.72:0/382805678 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+409 (secure 0 0 0) 0x7f297404d2a0 con 0x7f2984109800 2024-04-26T23:34:14.281 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.605+0000 7f2989400700 1 -- 172.21.15.72:0/382805678 >> [v2:172.21.15.72:6800/4175896290,v1:172.21.15.72:6801/4175896290] conn(0x7f29680395b0 msgr2=0x7f296803ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:14.281 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.605+0000 7f2989400700 1 --2- 172.21.15.72:0/382805678 >> [v2:172.21.15.72:6800/4175896290,v1:172.21.15.72:6801/4175896290] conn(0x7f29680395b0 0x7f296803ba70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2970006fd0 tx=0x7f2970006e40 comp rx=0 tx=0).stop 2024-04-26T23:34:14.281 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.605+0000 7f2989400700 1 -- 172.21.15.72:0/382805678 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2984109800 msgr2=0x7f2984109080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:14.282 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.605+0000 7f2989400700 1 --2- 172.21.15.72:0/382805678 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2984109800 0x7f2984109080 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2974011ac0 tx=0x7f2974011ba0 comp rx=0 tx=0).stop 2024-04-26T23:34:14.282 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.605+0000 7f2989400700 1 -- 172.21.15.72:0/382805678 shutdown_connections 2024-04-26T23:34:14.282 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.605+0000 7f2989400700 1 --2- 172.21.15.72:0/382805678 >> [v2:172.21.15.72:6800/4175896290,v1:172.21.15.72:6801/4175896290] conn(0x7f29680395b0 0x7f296803ba70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:14.282 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.605+0000 7f2989400700 1 --2- 172.21.15.72:0/382805678 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2984109800 0x7f2984109080 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:14.282 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.605+0000 7f2989400700 1 -- 172.21.15.72:0/382805678 >> 172.21.15.72:0/382805678 conn(0x7f29841031f0 msgr2=0x7f2984103d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:34:14.282 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.605+0000 7f2989400700 1 -- 172.21.15.72:0/382805678 shutdown_connections 2024-04-26T23:34:14.282 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:13.605+0000 7f2989400700 1 -- 172.21.15.72:0/382805678 wait complete. 2024-04-26T23:34:14.282 INFO:teuthology.orchestra.run.smithi072.stdout:Enabling cephadm module... 2024-04-26T23:34:15.539 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:15 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/1637930324' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-26T23:34:15.539 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:15 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: ignoring --setuser ceph since I am not root 2024-04-26T23:34:15.539 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:15 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: ignoring --setgroup ceph since I am not root 2024-04-26T23:34:15.539 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:15 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:15.536+0000 7f797f3be200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T23:34:15.793 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:15 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:15.783+0000 7f797f3be200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T23:34:15.945 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:14.900+0000 7f7d72600700 1 Processor -- start 2024-04-26T23:34:15.946 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:14.900+0000 7f7d72600700 1 -- start start 2024-04-26T23:34:15.946 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:14.900+0000 7f7d72600700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d6c108bb0 0x7f7d6c108fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:15.946 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:14.901+0000 7f7d72600700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f7d6c1095a0 con 0x7f7d6c108bb0 2024-04-26T23:34:15.951 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:14.901+0000 7f7d71200700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d6c108bb0 0x7f7d6c108fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:15.952 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:14.901+0000 7f7d71200700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d6c108bb0 0x7f7d6c108fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38088/0 (socket says 172.21.15.72:38088) 2024-04-26T23:34:15.952 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:14.901+0000 7f7d71200700 1 -- 172.21.15.72:0/523914580 learned_addr learned my addr 172.21.15.72:0/523914580 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:34:15.952 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:14.902+0000 7f7d71200700 1 -- 172.21.15.72:0/523914580 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d6c109db0 con 0x7f7d6c108bb0 2024-04-26T23:34:15.952 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:14.902+0000 7f7d71200700 1 --2- 172.21.15.72:0/523914580 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d6c108bb0 0x7f7d6c108fd0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7d600098a0 tx=0x7f7d60009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb497ee23f2f6246 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:15.952 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:14.902+0000 7f7d6b400700 1 -- 172.21.15.72:0/523914580 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d60004160 con 0x7f7d6c108bb0 2024-04-26T23:34:15.952 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:14.902+0000 7f7d6b400700 1 -- 172.21.15.72:0/523914580 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7d60004300 con 0x7f7d6c108bb0 2024-04-26T23:34:15.952 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:14.902+0000 7f7d6b400700 1 -- 172.21.15.72:0/523914580 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d60011af0 con 0x7f7d6c108bb0 2024-04-26T23:34:15.952 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:14.903+0000 7f7d72600700 1 -- 172.21.15.72:0/523914580 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d6c108bb0 msgr2=0x7f7d6c108fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:15.952 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:14.903+0000 7f7d72600700 1 --2- 172.21.15.72:0/523914580 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d6c108bb0 0x7f7d6c108fd0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7d600098a0 tx=0x7f7d60009bb0 comp rx=0 tx=0).stop 2024-04-26T23:34:15.952 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:14.903+0000 7f7d72600700 1 -- 172.21.15.72:0/523914580 shutdown_connections 2024-04-26T23:34:15.952 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:14.903+0000 7f7d72600700 1 --2- 172.21.15.72:0/523914580 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d6c108bb0 0x7f7d6c108fd0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:15.952 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:14.903+0000 7f7d72600700 1 -- 172.21.15.72:0/523914580 >> 172.21.15.72:0/523914580 conn(0x7f7d6c1041d0 msgr2=0x7f7d6c1065f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:34:15.953 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:14.903+0000 7f7d72600700 1 -- 172.21.15.72:0/523914580 shutdown_connections 2024-04-26T23:34:15.953 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:14.903+0000 7f7d72600700 1 -- 172.21.15.72:0/523914580 wait complete. 2024-04-26T23:34:15.953 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:14.904+0000 7f7d72600700 1 Processor -- start 2024-04-26T23:34:15.953 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:14.904+0000 7f7d72600700 1 -- start start 2024-04-26T23:34:15.953 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:14.905+0000 7f7d72600700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d6c108bb0 0x7f7d6c19cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:15.953 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:14.905+0000 7f7d72600700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f7d6c10d160 con 0x7f7d6c108bb0 2024-04-26T23:34:15.953 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:14.905+0000 7f7d71200700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d6c108bb0 0x7f7d6c19cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:15.953 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:14.905+0000 7f7d71200700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d6c108bb0 0x7f7d6c19cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38098/0 (socket says 172.21.15.72:38098) 2024-04-26T23:34:15.953 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:14.905+0000 7f7d71200700 1 -- 172.21.15.72:0/1637930324 learned_addr learned my addr 172.21.15.72:0/1637930324 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:34:15.953 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:14.905+0000 7f7d71200700 1 -- 172.21.15.72:0/1637930324 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d6c19d040 con 0x7f7d6c108bb0 2024-04-26T23:34:15.953 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:14.906+0000 7f7d71200700 1 --2- 172.21.15.72:0/1637930324 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d6c108bb0 0x7f7d6c19cb00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7d60003f40 tx=0x7f7d60011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:15.953 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:14.906+0000 7f7d69600700 1 -- 172.21.15.72:0/1637930324 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d600038b0 con 0x7f7d6c108bb0 2024-04-26T23:34:15.953 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:14.906+0000 7f7d69600700 1 -- 172.21.15.72:0/1637930324 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7d60003a50 con 0x7f7d6c108bb0 2024-04-26T23:34:15.953 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:14.906+0000 7f7d69600700 1 -- 172.21.15.72:0/1637930324 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d6001a5c0 con 0x7f7d6c108bb0 2024-04-26T23:34:15.953 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:14.906+0000 7f7d72600700 1 -- 172.21.15.72:0/1637930324 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d6c19d2d0 con 0x7f7d6c108bb0 2024-04-26T23:34:15.954 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:14.906+0000 7f7d72600700 1 -- 172.21.15.72:0/1637930324 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d6c19ffc0 con 0x7f7d6c108bb0 2024-04-26T23:34:15.954 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:14.908+0000 7f7d69600700 1 -- 172.21.15.72:0/1637930324 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f7d60024430 con 0x7f7d6c108bb0 2024-04-26T23:34:15.954 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:14.908+0000 7f7d72600700 1 -- 172.21.15.72:0/1637930324 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d6c06a540 con 0x7f7d6c108bb0 2024-04-26T23:34:15.954 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:14.908+0000 7f7d69600700 1 --2- 172.21.15.72:0/1637930324 >> [v2:172.21.15.72:6800/4175896290,v1:172.21.15.72:6801/4175896290] conn(0x7f7d54039510 0x7f7d5403b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:15.954 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:14.908+0000 7f7d69600700 1 -- 172.21.15.72:0/1637930324 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7d6004da50 con 0x7f7d6c108bb0 2024-04-26T23:34:15.954 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:14.908+0000 7f7d6be00700 1 --2- 172.21.15.72:0/1637930324 >> [v2:172.21.15.72:6800/4175896290,v1:172.21.15.72:6801/4175896290] conn(0x7f7d54039510 0x7f7d5403b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:15.954 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:14.909+0000 7f7d6be00700 1 --2- 172.21.15.72:0/1637930324 >> [v2:172.21.15.72:6800/4175896290,v1:172.21.15.72:6801/4175896290] conn(0x7f7d54039510 0x7f7d5403b9d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7d58006fd0 tx=0x7f7d58006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:15.954 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:14.914+0000 7f7d69600700 1 -- 172.21.15.72:0/1637930324 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7d6004d470 con 0x7f7d6c108bb0 2024-04-26T23:34:15.954 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:15.197+0000 7f7d72600700 1 -- 172.21.15.72:0/1637930324 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f7d6c1a0190 con 0x7f7d6c108bb0 2024-04-26T23:34:15.954 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:15.250+0000 7f7d69600700 1 -- 172.21.15.72:0/1637930324 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f7d6c1a0190 con 0x7f7d6c108bb0 2024-04-26T23:34:15.954 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:15.254+0000 7f7d69600700 1 -- 172.21.15.72:0/1637930324 <== mon.0 v2:172.21.15.72:3300/0 8 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f7d6002d910 con 0x7f7d6c108bb0 2024-04-26T23:34:15.954 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:15.254+0000 7f7d72600700 1 -- 172.21.15.72:0/1637930324 >> [v2:172.21.15.72:6800/4175896290,v1:172.21.15.72:6801/4175896290] conn(0x7f7d54039510 msgr2=0x7f7d5403b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:15.955 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:15.254+0000 7f7d72600700 1 --2- 172.21.15.72:0/1637930324 >> [v2:172.21.15.72:6800/4175896290,v1:172.21.15.72:6801/4175896290] conn(0x7f7d54039510 0x7f7d5403b9d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7d58006fd0 tx=0x7f7d58006e40 comp rx=0 tx=0).stop 2024-04-26T23:34:15.955 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:15.254+0000 7f7d72600700 1 -- 172.21.15.72:0/1637930324 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d6c108bb0 msgr2=0x7f7d6c19cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:15.955 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:15.254+0000 7f7d72600700 1 --2- 172.21.15.72:0/1637930324 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d6c108bb0 0x7f7d6c19cb00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7d60003f40 tx=0x7f7d60011890 comp rx=0 tx=0).stop 2024-04-26T23:34:15.955 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:15.254+0000 7f7d72600700 1 -- 172.21.15.72:0/1637930324 shutdown_connections 2024-04-26T23:34:15.955 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:15.254+0000 7f7d72600700 1 --2- 172.21.15.72:0/1637930324 >> [v2:172.21.15.72:6800/4175896290,v1:172.21.15.72:6801/4175896290] conn(0x7f7d54039510 0x7f7d5403b9d0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:15.955 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:15.254+0000 7f7d72600700 1 --2- 172.21.15.72:0/1637930324 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d6c108bb0 0x7f7d6c19cb00 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:15.955 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:15.254+0000 7f7d72600700 1 -- 172.21.15.72:0/1637930324 >> 172.21.15.72:0/1637930324 conn(0x7f7d6c1041d0 msgr2=0x7f7d6c104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:34:15.955 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:15.255+0000 7f7d72600700 1 -- 172.21.15.72:0/1637930324 shutdown_connections 2024-04-26T23:34:15.955 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:15.255+0000 7f7d72600700 1 -- 172.21.15.72:0/1637930324 wait complete. 2024-04-26T23:34:16.265 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:15 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:15.915+0000 7f797f3be200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T23:34:16.542 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:16 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:16.331+0000 7f797f3be200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T23:34:16.543 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:16 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:16.445+0000 7f797f3be200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T23:34:16.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:16 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/1637930324' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-26T23:34:16.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:16 smithi072 ceph-mon[28184]: mgrmap e5: x(active, since 5s) 2024-04-26T23:34:17.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:17 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/1107882737' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-26T23:34:17.604 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout { 2024-04-26T23:34:17.610 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-26T23:34:17.610 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-26T23:34:17.610 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-26T23:34:17.610 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-26T23:34:17.610 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout } 2024-04-26T23:34:17.610 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.569+0000 7fc63ec00700 1 Processor -- start 2024-04-26T23:34:17.610 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.569+0000 7fc63ec00700 1 -- start start 2024-04-26T23:34:17.610 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.569+0000 7fc63ec00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc63807f760 0x7fc63807dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:17.610 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.569+0000 7fc63ec00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fc63807e190 con 0x7fc63807f760 2024-04-26T23:34:17.611 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.570+0000 7fc637e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc63807f760 0x7fc63807dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:17.611 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.570+0000 7fc637e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc63807f760 0x7fc63807dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38126/0 (socket says 172.21.15.72:38126) 2024-04-26T23:34:17.611 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.570+0000 7fc637e00700 1 -- 172.21.15.72:0/2549251783 learned_addr learned my addr 172.21.15.72:0/2549251783 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:34:17.611 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.570+0000 7fc637e00700 1 -- 172.21.15.72:0/2549251783 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc63807e310 con 0x7fc63807f760 2024-04-26T23:34:17.611 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.571+0000 7fc637e00700 1 --2- 172.21.15.72:0/2549251783 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc63807f760 0x7fc63807dbc0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc6280098a0 tx=0x7fc628009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a30e358767953c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:17.611 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.571+0000 7fc636a00700 1 -- 172.21.15.72:0/2549251783 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc628004160 con 0x7fc63807f760 2024-04-26T23:34:17.611 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.571+0000 7fc636a00700 1 -- 172.21.15.72:0/2549251783 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc628004300 con 0x7fc63807f760 2024-04-26T23:34:17.611 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.571+0000 7fc636a00700 1 -- 172.21.15.72:0/2549251783 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc628011af0 con 0x7fc63807f760 2024-04-26T23:34:17.611 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.572+0000 7fc63ec00700 1 -- 172.21.15.72:0/2549251783 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc63807f760 msgr2=0x7fc63807dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:17.611 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.572+0000 7fc63ec00700 1 --2- 172.21.15.72:0/2549251783 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc63807f760 0x7fc63807dbc0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc6280098a0 tx=0x7fc628009bb0 comp rx=0 tx=0).stop 2024-04-26T23:34:17.611 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.572+0000 7fc63ec00700 1 -- 172.21.15.72:0/2549251783 shutdown_connections 2024-04-26T23:34:17.611 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.572+0000 7fc63ec00700 1 --2- 172.21.15.72:0/2549251783 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc63807f760 0x7fc63807dbc0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:17.611 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.572+0000 7fc63ec00700 1 -- 172.21.15.72:0/2549251783 >> 172.21.15.72:0/2549251783 conn(0x7fc638104310 msgr2=0x7fc638106770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:34:17.611 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.572+0000 7fc63ec00700 1 -- 172.21.15.72:0/2549251783 shutdown_connections 2024-04-26T23:34:17.612 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.572+0000 7fc63ec00700 1 -- 172.21.15.72:0/2549251783 wait complete. 2024-04-26T23:34:17.612 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.573+0000 7fc63ec00700 1 Processor -- start 2024-04-26T23:34:17.612 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.573+0000 7fc63ec00700 1 -- start start 2024-04-26T23:34:17.612 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.574+0000 7fc63ec00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc63807f760 0x7fc63819ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:17.612 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.574+0000 7fc63ec00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fc63810f350 con 0x7fc63807f760 2024-04-26T23:34:17.612 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.574+0000 7fc637e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc63807f760 0x7fc63819ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:17.612 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.574+0000 7fc637e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc63807f760 0x7fc63819ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38142/0 (socket says 172.21.15.72:38142) 2024-04-26T23:34:17.612 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.574+0000 7fc637e00700 1 -- 172.21.15.72:0/1107882737 learned_addr learned my addr 172.21.15.72:0/1107882737 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:34:17.612 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.574+0000 7fc637e00700 1 -- 172.21.15.72:0/1107882737 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc63819f440 con 0x7fc63807f760 2024-04-26T23:34:17.612 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.575+0000 7fc637e00700 1 --2- 172.21.15.72:0/1107882737 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc63807f760 0x7fc63819ef00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc6280117b0 tx=0x7fc628011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:17.612 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.575+0000 7fc634c00700 1 -- 172.21.15.72:0/1107882737 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc628003af0 con 0x7fc63807f760 2024-04-26T23:34:17.612 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.575+0000 7fc634c00700 1 -- 172.21.15.72:0/1107882737 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc628020070 con 0x7fc63807f760 2024-04-26T23:34:17.612 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.575+0000 7fc63ec00700 1 -- 172.21.15.72:0/1107882737 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc63819f6d0 con 0x7fc63807f760 2024-04-26T23:34:17.612 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.575+0000 7fc634c00700 1 -- 172.21.15.72:0/1107882737 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc628019630 con 0x7fc63807f760 2024-04-26T23:34:17.612 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.575+0000 7fc63ec00700 1 -- 172.21.15.72:0/1107882737 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6381a23c0 con 0x7fc63807f760 2024-04-26T23:34:17.612 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.576+0000 7fc634c00700 1 -- 172.21.15.72:0/1107882737 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7fc62801a430 con 0x7fc63807f760 2024-04-26T23:34:17.613 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.576+0000 7fc63ec00700 1 -- 172.21.15.72:0/1107882737 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc638064a80 con 0x7fc63807f760 2024-04-26T23:34:17.613 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.577+0000 7fc634c00700 1 --2- 172.21.15.72:0/1107882737 >> [v2:172.21.15.72:6800/4175896290,v1:172.21.15.72:6801/4175896290] conn(0x7fc61c039600 0x7fc61c03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:17.613 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.577+0000 7fc634c00700 1 -- 172.21.15.72:0/1107882737 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc62804dac0 con 0x7fc63807f760 2024-04-26T23:34:17.613 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.577+0000 7fc637400700 1 -- 172.21.15.72:0/1107882737 >> [v2:172.21.15.72:6800/4175896290,v1:172.21.15.72:6801/4175896290] conn(0x7fc61c039600 msgr2=0x7fc61c03bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/4175896290 2024-04-26T23:34:17.613 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.577+0000 7fc637400700 1 --2- 172.21.15.72:0/1107882737 >> [v2:172.21.15.72:6800/4175896290,v1:172.21.15.72:6801/4175896290] conn(0x7fc61c039600 0x7fc61c03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T23:34:17.613 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.582+0000 7fc634c00700 1 -- 172.21.15.72:0/1107882737 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc62800e260 con 0x7fc63807f760 2024-04-26T23:34:17.613 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.778+0000 7fc637400700 1 -- 172.21.15.72:0/1107882737 >> [v2:172.21.15.72:6800/4175896290,v1:172.21.15.72:6801/4175896290] conn(0x7fc61c039600 msgr2=0x7fc61c03bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/4175896290 2024-04-26T23:34:17.613 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.778+0000 7fc637400700 1 --2- 172.21.15.72:0/1107882737 >> [v2:172.21.15.72:6800/4175896290,v1:172.21.15.72:6801/4175896290] conn(0x7fc61c039600 0x7fc61c03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T23:34:17.613 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.891+0000 7fc63ec00700 1 -- 172.21.15.72:0/1107882737 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fc6381a2590 con 0x7fc63807f760 2024-04-26T23:34:17.613 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.891+0000 7fc634c00700 1 -- 172.21.15.72:0/1107882737 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7fc6381a2590 con 0x7fc63807f760 2024-04-26T23:34:17.613 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.896+0000 7fc63ec00700 1 -- 172.21.15.72:0/1107882737 >> [v2:172.21.15.72:6800/4175896290,v1:172.21.15.72:6801/4175896290] conn(0x7fc61c039600 msgr2=0x7fc61c03bac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T23:34:17.613 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.896+0000 7fc63ec00700 1 --2- 172.21.15.72:0/1107882737 >> [v2:172.21.15.72:6800/4175896290,v1:172.21.15.72:6801/4175896290] conn(0x7fc61c039600 0x7fc61c03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:17.613 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.896+0000 7fc63ec00700 1 -- 172.21.15.72:0/1107882737 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc63807f760 msgr2=0x7fc63819ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:17.613 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.896+0000 7fc63ec00700 1 --2- 172.21.15.72:0/1107882737 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc63807f760 0x7fc63819ef00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc6280117b0 tx=0x7fc628011b70 comp rx=0 tx=0).stop 2024-04-26T23:34:17.614 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.896+0000 7fc63ec00700 1 -- 172.21.15.72:0/1107882737 shutdown_connections 2024-04-26T23:34:17.614 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.896+0000 7fc63ec00700 1 --2- 172.21.15.72:0/1107882737 >> [v2:172.21.15.72:6800/4175896290,v1:172.21.15.72:6801/4175896290] conn(0x7fc61c039600 0x7fc61c03bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:17.614 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.896+0000 7fc63ec00700 1 --2- 172.21.15.72:0/1107882737 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc63807f760 0x7fc63819ef00 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:17.614 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.896+0000 7fc63ec00700 1 -- 172.21.15.72:0/1107882737 >> 172.21.15.72:0/1107882737 conn(0x7fc638104310 msgr2=0x7fc638104ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:34:17.614 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.896+0000 7fc63ec00700 1 -- 172.21.15.72:0/1107882737 shutdown_connections 2024-04-26T23:34:17.614 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:16.896+0000 7fc63ec00700 1 -- 172.21.15.72:0/1107882737 wait complete. 2024-04-26T23:34:17.614 INFO:teuthology.orchestra.run.smithi072.stdout:Waiting for the mgr to restart... 2024-04-26T23:34:17.614 INFO:teuthology.orchestra.run.smithi072.stdout:Waiting for mgr epoch 5... 2024-04-26T23:34:18.171 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:18 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:18.086+0000 7f797f3be200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T23:34:18.425 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:18 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:18.315+0000 7f797f3be200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T23:34:18.793 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:18 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:18.530+0000 7f797f3be200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T23:34:19.286 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:18 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:18.907+0000 7f797f3be200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T23:34:19.287 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:19 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:19.030+0000 7f797f3be200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T23:34:19.540 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:19 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:19.283+0000 7f797f3be200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T23:34:19.541 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:19 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:19.537+0000 7f797f3be200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T23:34:20.695 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:20 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:20.318+0000 7f797f3be200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T23:34:20.997 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:20 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:20.863+0000 7f797f3be200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T23:34:20.997 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:20 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:20.994+0000 7f797f3be200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T23:34:21.543 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:21 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:21.256+0000 7f797f3be200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T23:34:22.031 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:21 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:21.901+0000 7f797f3be200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T23:34:22.031 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:22 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:22.028+0000 7f797f3be200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T23:34:22.674 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:22 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:22.358+0000 7f797f3be200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T23:34:22.674 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:22 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:22.541+0000 7f797f3be200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T23:34:22.675 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:22 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:22.671+0000 7f797f3be200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T23:34:23.043 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:22 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:22.824+0000 7f797f3be200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T23:34:23.543 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:23 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:23.201+0000 7f797f3be200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T23:34:24.232 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:23 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:23.949+0000 7f797f3be200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T23:34:24.543 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:24 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:24.228+0000 7f797f3be200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T23:34:24.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:24 smithi072 ceph-mon[28184]: Active manager daemon x restarted 2024-04-26T23:34:24.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:24 smithi072 ceph-mon[28184]: Activating manager daemon x 2024-04-26T23:34:24.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:24 smithi072 ceph-mon[28184]: osdmap e2: 0 total, 0 up, 0 in 2024-04-26T23:34:24.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:24 smithi072 ceph-mon[28184]: mgrmap e6: x(active, starting, since 0.0053014s) 2024-04-26T23:34:24.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:24 smithi072 ceph-mon[28184]: from='mgr.14120 172.21.15.72:0/3482651254' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T23:34:24.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:24 smithi072 ceph-mon[28184]: from='mgr.14120 172.21.15.72:0/3482651254' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T23:34:24.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:24 smithi072 ceph-mon[28184]: from='mgr.14120 172.21.15.72:0/3482651254' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T23:34:24.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:24 smithi072 ceph-mon[28184]: from='mgr.14120 172.21.15.72:0/3482651254' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T23:34:24.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:24 smithi072 ceph-mon[28184]: from='mgr.14120 172.21.15.72:0/3482651254' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T23:34:24.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:24 smithi072 ceph-mon[28184]: Manager daemon x is now available 2024-04-26T23:34:24.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:24 smithi072 ceph-mon[28184]: from='mgr.14120 172.21.15.72:0/3482651254' entity='mgr.x' 2024-04-26T23:34:24.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:24 smithi072 ceph-mon[28184]: from='mgr.14120 172.21.15.72:0/3482651254' entity='mgr.x' 2024-04-26T23:34:25.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:25 smithi072 ceph-mon[28184]: Found migration_current of "None". Setting to last migration. 2024-04-26T23:34:25.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:25 smithi072 ceph-mon[28184]: from='mgr.14120 172.21.15.72:0/3482651254' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:34:25.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:25 smithi072 ceph-mon[28184]: from='mgr.14120 172.21.15.72:0/3482651254' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:34:25.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:25 smithi072 ceph-mon[28184]: from='mgr.14120 172.21.15.72:0/3482651254' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T23:34:25.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:25 smithi072 ceph-mon[28184]: from='mgr.14120 172.21.15.72:0/3482651254' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T23:34:25.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:25 smithi072 ceph-mon[28184]: from='mgr.14120 172.21.15.72:0/3482651254' entity='mgr.x' 2024-04-26T23:34:25.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:25 smithi072 ceph-mon[28184]: from='mgr.14120 172.21.15.72:0/3482651254' entity='mgr.x' 2024-04-26T23:34:25.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:25 smithi072 ceph-mon[28184]: from='mgr.14120 172.21.15.72:0/3482651254' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:34:25.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:25 smithi072 ceph-mon[28184]: mgrmap e7: x(active, since 1.01731s) 2024-04-26T23:34:25.983 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout { 2024-04-26T23:34:25.983 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-26T23:34:25.983 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-26T23:34:25.983 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout } 2024-04-26T23:34:25.984 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:18.214+0000 7f729fa00700 1 Processor -- start 2024-04-26T23:34:25.984 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:18.214+0000 7f729fa00700 1 -- start start 2024-04-26T23:34:25.984 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:18.215+0000 7f729fa00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7298107070 0x7f7298107490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:25.984 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:18.215+0000 7f729fa00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f7298107a60 con 0x7f7298107070 2024-04-26T23:34:25.984 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:18.215+0000 7f729d000700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7298107070 0x7f7298107490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:25.984 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:18.215+0000 7f729d000700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7298107070 0x7f7298107490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:53210/0 (socket says 172.21.15.72:53210) 2024-04-26T23:34:25.984 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:18.215+0000 7f729d000700 1 -- 172.21.15.72:0/2160475746 learned_addr learned my addr 172.21.15.72:0/2160475746 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:34:25.984 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:18.216+0000 7f729d000700 1 -- 172.21.15.72:0/2160475746 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7298107be0 con 0x7f7298107070 2024-04-26T23:34:25.984 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:18.216+0000 7f729d000700 1 --2- 172.21.15.72:0/2160475746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7298107070 0x7f7298107490 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f728c0098a0 tx=0x7f728c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=611e12e6f65f15fd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:25.984 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:18.216+0000 7f7297400700 1 -- 172.21.15.72:0/2160475746 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f728c004160 con 0x7f7298107070 2024-04-26T23:34:25.984 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:18.216+0000 7f7297400700 1 -- 172.21.15.72:0/2160475746 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f728c004300 con 0x7f7298107070 2024-04-26T23:34:25.984 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:18.216+0000 7f7297400700 1 -- 172.21.15.72:0/2160475746 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f728c011af0 con 0x7f7298107070 2024-04-26T23:34:25.984 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:18.217+0000 7f729fa00700 1 -- 172.21.15.72:0/2160475746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7298107070 msgr2=0x7f7298107490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:25.984 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:18.217+0000 7f729fa00700 1 --2- 172.21.15.72:0/2160475746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7298107070 0x7f7298107490 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f728c0098a0 tx=0x7f728c009bb0 comp rx=0 tx=0).stop 2024-04-26T23:34:25.985 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:18.217+0000 7f729fa00700 1 -- 172.21.15.72:0/2160475746 shutdown_connections 2024-04-26T23:34:25.985 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:18.217+0000 7f729fa00700 1 --2- 172.21.15.72:0/2160475746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7298107070 0x7f7298107490 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:25.985 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:18.217+0000 7f729fa00700 1 -- 172.21.15.72:0/2160475746 >> 172.21.15.72:0/2160475746 conn(0x7f7298102970 msgr2=0x7f7298104df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:34:25.985 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:18.217+0000 7f729fa00700 1 -- 172.21.15.72:0/2160475746 shutdown_connections 2024-04-26T23:34:25.985 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:18.218+0000 7f729fa00700 1 -- 172.21.15.72:0/2160475746 wait complete. 2024-04-26T23:34:25.985 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:18.219+0000 7f729fa00700 1 Processor -- start 2024-04-26T23:34:25.985 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:18.219+0000 7f729fa00700 1 -- start start 2024-04-26T23:34:25.985 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:18.219+0000 7f729fa00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7298107070 0x7f729819ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:25.985 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:18.219+0000 7f729fa00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f729810aed0 con 0x7f7298107070 2024-04-26T23:34:25.985 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:18.219+0000 7f729d000700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7298107070 0x7f729819ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:25.985 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:18.219+0000 7f729d000700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7298107070 0x7f729819ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:53214/0 (socket says 172.21.15.72:53214) 2024-04-26T23:34:25.985 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:18.219+0000 7f729d000700 1 -- 172.21.15.72:0/1277391030 learned_addr learned my addr 172.21.15.72:0/1277391030 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:34:25.985 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:18.220+0000 7f729d000700 1 -- 172.21.15.72:0/1277391030 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f729819d210 con 0x7f7298107070 2024-04-26T23:34:25.985 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:18.220+0000 7f729d000700 1 --2- 172.21.15.72:0/1277391030 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7298107070 0x7f729819ccd0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f728c0117b0 tx=0x7f728c011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:25.986 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:18.220+0000 7f7295600700 1 -- 172.21.15.72:0/1277391030 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f728c003af0 con 0x7f7298107070 2024-04-26T23:34:25.986 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:18.220+0000 7f729fa00700 1 -- 172.21.15.72:0/1277391030 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f729819d500 con 0x7f7298107070 2024-04-26T23:34:25.986 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:18.220+0000 7f7295600700 1 -- 172.21.15.72:0/1277391030 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f728c020070 con 0x7f7298107070 2024-04-26T23:34:25.986 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:18.221+0000 7f729fa00700 1 -- 172.21.15.72:0/1277391030 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72981a0180 con 0x7f7298107070 2024-04-26T23:34:25.986 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:18.221+0000 7f7295600700 1 -- 172.21.15.72:0/1277391030 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f728c02a060 con 0x7f7298107070 2024-04-26T23:34:25.986 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:18.222+0000 7f7295600700 1 -- 172.21.15.72:0/1277391030 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f728c01a430 con 0x7f7298107070 2024-04-26T23:34:25.986 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:18.222+0000 7f7295600700 1 --2- 172.21.15.72:0/1277391030 >> [v2:172.21.15.72:6800/4175896290,v1:172.21.15.72:6801/4175896290] conn(0x7f72800395b0 0x7f728003ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:25.986 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:18.222+0000 7f7295600700 1 -- 172.21.15.72:0/1277391030 --> [v2:172.21.15.72:6800/4175896290,v1:172.21.15.72:6801/4175896290] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f728003c180 con 0x7f72800395b0 2024-04-26T23:34:25.986 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:18.222+0000 7f7295600700 1 -- 172.21.15.72:0/1277391030 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f728c04deb0 con 0x7f7298107070 2024-04-26T23:34:25.986 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:18.222+0000 7f7297e00700 1 -- 172.21.15.72:0/1277391030 >> [v2:172.21.15.72:6800/4175896290,v1:172.21.15.72:6801/4175896290] conn(0x7f72800395b0 msgr2=0x7f728003ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/4175896290 2024-04-26T23:34:25.986 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:18.222+0000 7f7297e00700 1 --2- 172.21.15.72:0/1277391030 >> [v2:172.21.15.72:6800/4175896290,v1:172.21.15.72:6801/4175896290] conn(0x7f72800395b0 0x7f728003ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T23:34:25.986 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:18.423+0000 7f7297e00700 1 -- 172.21.15.72:0/1277391030 >> [v2:172.21.15.72:6800/4175896290,v1:172.21.15.72:6801/4175896290] conn(0x7f72800395b0 msgr2=0x7f728003ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/4175896290 2024-04-26T23:34:25.986 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:18.423+0000 7f7297e00700 1 --2- 172.21.15.72:0/1277391030 >> [v2:172.21.15.72:6800/4175896290,v1:172.21.15.72:6801/4175896290] conn(0x7f72800395b0 0x7f728003ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T23:34:25.987 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:18.824+0000 7f7297e00700 1 -- 172.21.15.72:0/1277391030 >> [v2:172.21.15.72:6800/4175896290,v1:172.21.15.72:6801/4175896290] conn(0x7f72800395b0 msgr2=0x7f728003ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/4175896290 2024-04-26T23:34:25.987 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:18.824+0000 7f7297e00700 1 --2- 172.21.15.72:0/1277391030 >> [v2:172.21.15.72:6800/4175896290,v1:172.21.15.72:6801/4175896290] conn(0x7f72800395b0 0x7f728003ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T23:34:25.987 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:19.625+0000 7f7297e00700 1 -- 172.21.15.72:0/1277391030 >> [v2:172.21.15.72:6800/4175896290,v1:172.21.15.72:6801/4175896290] conn(0x7f72800395b0 msgr2=0x7f728003ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/4175896290 2024-04-26T23:34:25.987 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:19.625+0000 7f7297e00700 1 --2- 172.21.15.72:0/1277391030 >> [v2:172.21.15.72:6800/4175896290,v1:172.21.15.72:6801/4175896290] conn(0x7f72800395b0 0x7f728003ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T23:34:25.987 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:21.227+0000 7f7297e00700 1 -- 172.21.15.72:0/1277391030 >> [v2:172.21.15.72:6800/4175896290,v1:172.21.15.72:6801/4175896290] conn(0x7f72800395b0 msgr2=0x7f728003ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/4175896290 2024-04-26T23:34:25.987 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:21.227+0000 7f7297e00700 1 --2- 172.21.15.72:0/1277391030 >> [v2:172.21.15.72:6800/4175896290,v1:172.21.15.72:6801/4175896290] conn(0x7f72800395b0 0x7f728003ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-26T23:34:25.987 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:24.237+0000 7f7295600700 1 -- 172.21.15.72:0/1277391030 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mgrmap(e 6) ==== 46325+0+0 (secure 0 0 0) 0x7f728c01a9c0 con 0x7f7298107070 2024-04-26T23:34:25.987 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:24.238+0000 7f7295600700 1 -- 172.21.15.72:0/1277391030 >> [v2:172.21.15.72:6800/4175896290,v1:172.21.15.72:6801/4175896290] conn(0x7f72800395b0 msgr2=0x7f728003ba70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T23:34:25.987 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:24.238+0000 7f7295600700 1 --2- 172.21.15.72:0/1277391030 >> [v2:172.21.15.72:6800/4175896290,v1:172.21.15.72:6801/4175896290] conn(0x7f72800395b0 0x7f728003ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:25.987 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:25.242+0000 7f7295600700 1 -- 172.21.15.72:0/1277391030 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7f728c04f930 con 0x7f7298107070 2024-04-26T23:34:25.987 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:25.243+0000 7f7295600700 1 --2- 172.21.15.72:0/1277391030 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7f728003cea0 0x7f728003f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:25.987 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:25.243+0000 7f7295600700 1 -- 172.21.15.72:0/1277391030 --> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f728003c180 con 0x7f728003cea0 2024-04-26T23:34:25.987 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:25.245+0000 7f7297e00700 1 --2- 172.21.15.72:0/1277391030 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7f728003cea0 0x7f728003f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:25.987 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:25.248+0000 7f7297e00700 1 --2- 172.21.15.72:0/1277391030 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7f728003cea0 0x7f728003f290 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f7284003a10 tx=0x7f7284009290 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:25.988 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:25.251+0000 7f7295600700 1 -- 172.21.15.72:0/1277391030 <== mgr.14120 v2:172.21.15.72:6800/2903979134 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f728003c180 con 0x7f728003cea0 2024-04-26T23:34:25.988 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:25.261+0000 7f728aa00700 1 -- 172.21.15.72:0/1277391030 --> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f7274002600 con 0x7f728003cea0 2024-04-26T23:34:25.988 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:25.262+0000 7f7295600700 1 -- 172.21.15.72:0/1277391030 <== mgr.14120 v2:172.21.15.72:6800/2903979134 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f7274002600 con 0x7f728003cea0 2024-04-26T23:34:25.988 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:25.262+0000 7f728aa00700 1 -- 172.21.15.72:0/1277391030 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7f728003cea0 msgr2=0x7f728003f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:25.988 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:25.262+0000 7f728aa00700 1 --2- 172.21.15.72:0/1277391030 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7f728003cea0 0x7f728003f290 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f7284003a10 tx=0x7f7284009290 comp rx=0 tx=0).stop 2024-04-26T23:34:25.988 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:25.262+0000 7f728aa00700 1 -- 172.21.15.72:0/1277391030 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7298107070 msgr2=0x7f729819ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:25.988 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:25.262+0000 7f728aa00700 1 --2- 172.21.15.72:0/1277391030 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7298107070 0x7f729819ccd0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f728c0117b0 tx=0x7f728c011b70 comp rx=0 tx=0).stop 2024-04-26T23:34:25.988 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:25.262+0000 7f728aa00700 1 -- 172.21.15.72:0/1277391030 shutdown_connections 2024-04-26T23:34:25.988 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:25.262+0000 7f728aa00700 1 --2- 172.21.15.72:0/1277391030 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7298107070 0x7f729819ccd0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:25.988 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:25.262+0000 7f728aa00700 1 --2- 172.21.15.72:0/1277391030 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7f728003cea0 0x7f728003f290 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:25.988 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:25.262+0000 7f728aa00700 1 --2- 172.21.15.72:0/1277391030 >> [v2:172.21.15.72:6800/4175896290,v1:172.21.15.72:6801/4175896290] conn(0x7f72800395b0 0x7f728003ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:25.988 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:25.262+0000 7f728aa00700 1 -- 172.21.15.72:0/1277391030 >> 172.21.15.72:0/1277391030 conn(0x7f7298102970 msgr2=0x7f7298103560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:34:25.988 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:25.263+0000 7f728aa00700 1 -- 172.21.15.72:0/1277391030 shutdown_connections 2024-04-26T23:34:25.988 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:25.263+0000 7f728aa00700 1 -- 172.21.15.72:0/1277391030 wait complete. 2024-04-26T23:34:25.988 INFO:teuthology.orchestra.run.smithi072.stdout:mgr epoch 5 is available 2024-04-26T23:34:25.989 INFO:teuthology.orchestra.run.smithi072.stdout:Setting orchestrator backend to cephadm... 2024-04-26T23:34:26.586 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:26 smithi072 ceph-mon[28184]: [26/Apr/2024:23:34:24] ENGINE Bus STARTING 2024-04-26T23:34:26.586 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:26 smithi072 ceph-mon[28184]: [26/Apr/2024:23:34:24] ENGINE Serving on https://172.21.15.72:7150 2024-04-26T23:34:26.586 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:26 smithi072 ceph-mon[28184]: [26/Apr/2024:23:34:24] ENGINE Serving on http://172.21.15.72:8765 2024-04-26T23:34:26.586 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:26 smithi072 ceph-mon[28184]: [26/Apr/2024:23:34:24] ENGINE Bus STARTED 2024-04-26T23:34:26.586 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:26 smithi072 ceph-mon[28184]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-26T23:34:26.586 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:26 smithi072 ceph-mon[28184]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-26T23:34:27.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:27 smithi072 ceph-mon[28184]: mgrmap e8: x(active, since 2s) 2024-04-26T23:34:27.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:27 smithi072 ceph-mon[28184]: from='mgr.14120 172.21.15.72:0/3482651254' entity='mgr.x' 2024-04-26T23:34:27.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:27 smithi072 ceph-mon[28184]: from='mgr.14120 172.21.15.72:0/3482651254' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:34:27.605 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.652+0000 7f522fc00700 1 Processor -- start 2024-04-26T23:34:27.605 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.652+0000 7f522fc00700 1 -- start start 2024-04-26T23:34:27.605 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.653+0000 7f522fc00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5228082dd0 0x7f52280831f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:27.606 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.653+0000 7f522fc00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f52280837c0 con 0x7f5228082dd0 2024-04-26T23:34:27.606 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.653+0000 7f522d200700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5228082dd0 0x7f52280831f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:27.606 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.654+0000 7f522d200700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5228082dd0 0x7f52280831f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:53280/0 (socket says 172.21.15.72:53280) 2024-04-26T23:34:27.606 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.654+0000 7f522d200700 1 -- 172.21.15.72:0/3555806558 learned_addr learned my addr 172.21.15.72:0/3555806558 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:34:27.606 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.654+0000 7f522d200700 1 -- 172.21.15.72:0/3555806558 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5228110550 con 0x7f5228082dd0 2024-04-26T23:34:27.606 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.654+0000 7f522d200700 1 --2- 172.21.15.72:0/3555806558 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5228082dd0 0x7f52280831f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f521c0098a0 tx=0x7f521c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caf59a44d8f3479f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:27.606 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.654+0000 7f5227400700 1 -- 172.21.15.72:0/3555806558 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f521c004160 con 0x7f5228082dd0 2024-04-26T23:34:27.606 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.655+0000 7f5227400700 1 -- 172.21.15.72:0/3555806558 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f521c004300 con 0x7f5228082dd0 2024-04-26T23:34:27.606 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.655+0000 7f5227400700 1 -- 172.21.15.72:0/3555806558 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f521c011af0 con 0x7f5228082dd0 2024-04-26T23:34:27.606 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.655+0000 7f522fc00700 1 -- 172.21.15.72:0/3555806558 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5228082dd0 msgr2=0x7f52280831f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:27.606 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.655+0000 7f522fc00700 1 --2- 172.21.15.72:0/3555806558 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5228082dd0 0x7f52280831f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f521c0098a0 tx=0x7f521c009bb0 comp rx=0 tx=0).stop 2024-04-26T23:34:27.606 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.655+0000 7f522fc00700 1 -- 172.21.15.72:0/3555806558 shutdown_connections 2024-04-26T23:34:27.606 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.655+0000 7f522fc00700 1 --2- 172.21.15.72:0/3555806558 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5228082dd0 0x7f52280831f0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:27.607 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.655+0000 7f522fc00700 1 -- 172.21.15.72:0/3555806558 >> 172.21.15.72:0/3555806558 conn(0x7f522807e390 msgr2=0x7f52280807b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:34:27.607 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.656+0000 7f522fc00700 1 -- 172.21.15.72:0/3555806558 shutdown_connections 2024-04-26T23:34:27.607 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.656+0000 7f522fc00700 1 -- 172.21.15.72:0/3555806558 wait complete. 2024-04-26T23:34:27.607 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.657+0000 7f522fc00700 1 Processor -- start 2024-04-26T23:34:27.607 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.657+0000 7f522fc00700 1 -- start start 2024-04-26T23:34:27.607 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.657+0000 7f522fc00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5228082dd0 0x7f52281a3300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:27.607 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.657+0000 7f522fc00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f5228113900 con 0x7f5228082dd0 2024-04-26T23:34:27.607 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.657+0000 7f522d200700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5228082dd0 0x7f52281a3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:27.607 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.657+0000 7f522d200700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5228082dd0 0x7f52281a3300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:53292/0 (socket says 172.21.15.72:53292) 2024-04-26T23:34:27.607 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.657+0000 7f522d200700 1 -- 172.21.15.72:0/783292262 learned_addr learned my addr 172.21.15.72:0/783292262 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:34:27.607 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.658+0000 7f522d200700 1 -- 172.21.15.72:0/783292262 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52281a3840 con 0x7f5228082dd0 2024-04-26T23:34:27.607 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.658+0000 7f522d200700 1 --2- 172.21.15.72:0/783292262 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5228082dd0 0x7f52281a3300 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f521c0117b0 tx=0x7f521c011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:27.607 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.658+0000 7f5225600700 1 -- 172.21.15.72:0/783292262 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f521c003af0 con 0x7f5228082dd0 2024-04-26T23:34:27.607 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.658+0000 7f5225600700 1 -- 172.21.15.72:0/783292262 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f521c020070 con 0x7f5228082dd0 2024-04-26T23:34:27.608 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.658+0000 7f522fc00700 1 -- 172.21.15.72:0/783292262 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52281a3ad0 con 0x7f5228082dd0 2024-04-26T23:34:27.608 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.658+0000 7f5225600700 1 -- 172.21.15.72:0/783292262 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f521c019630 con 0x7f5228082dd0 2024-04-26T23:34:27.608 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.659+0000 7f522fc00700 1 -- 172.21.15.72:0/783292262 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52281a67c0 con 0x7f5228082dd0 2024-04-26T23:34:27.608 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.660+0000 7f5225600700 1 -- 172.21.15.72:0/783292262 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f521c01a9c0 con 0x7f5228082dd0 2024-04-26T23:34:27.608 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.660+0000 7f522fc00700 1 -- 172.21.15.72:0/783292262 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5228064a80 con 0x7f5228082dd0 2024-04-26T23:34:27.608 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.660+0000 7f5225600700 1 --2- 172.21.15.72:0/783292262 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7f5210039600 0x7f521003bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:27.608 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.660+0000 7f5225600700 1 -- 172.21.15.72:0/783292262 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f521c04dbf0 con 0x7f5228082dd0 2024-04-26T23:34:27.608 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.660+0000 7f5227e00700 1 --2- 172.21.15.72:0/783292262 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7f5210039600 0x7f521003bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:27.608 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.661+0000 7f5227e00700 1 --2- 172.21.15.72:0/783292262 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7f5210039600 0x7f521003bac0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5214006fd0 tx=0x7f5214006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:27.608 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.666+0000 7f5225600700 1 -- 172.21.15.72:0/783292262 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f521c0042a0 con 0x7f5228082dd0 2024-04-26T23:34:27.608 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.910+0000 7f522fc00700 1 -- 172.21.15.72:0/783292262 --> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f522807f650 con 0x7f5210039600 2024-04-26T23:34:27.608 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.922+0000 7f5225600700 1 -- 172.21.15.72:0/783292262 <== mgr.14120 v2:172.21.15.72:6800/2903979134 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f522807f650 con 0x7f5210039600 2024-04-26T23:34:27.608 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.925+0000 7f522fc00700 1 -- 172.21.15.72:0/783292262 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7f5210039600 msgr2=0x7f521003bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:27.608 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.925+0000 7f522fc00700 1 --2- 172.21.15.72:0/783292262 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7f5210039600 0x7f521003bac0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5214006fd0 tx=0x7f5214006e40 comp rx=0 tx=0).stop 2024-04-26T23:34:27.609 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.925+0000 7f522fc00700 1 -- 172.21.15.72:0/783292262 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5228082dd0 msgr2=0x7f52281a3300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:27.609 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.925+0000 7f522fc00700 1 --2- 172.21.15.72:0/783292262 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5228082dd0 0x7f52281a3300 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f521c0117b0 tx=0x7f521c011b70 comp rx=0 tx=0).stop 2024-04-26T23:34:27.609 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.926+0000 7f522fc00700 1 -- 172.21.15.72:0/783292262 shutdown_connections 2024-04-26T23:34:27.609 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.926+0000 7f522fc00700 1 --2- 172.21.15.72:0/783292262 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7f5210039600 0x7f521003bac0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:27.609 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.926+0000 7f522fc00700 1 --2- 172.21.15.72:0/783292262 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5228082dd0 0x7f52281a3300 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:27.609 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.926+0000 7f522fc00700 1 -- 172.21.15.72:0/783292262 >> 172.21.15.72:0/783292262 conn(0x7f522807e390 msgr2=0x7f522807ef40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:34:27.609 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.926+0000 7f522fc00700 1 -- 172.21.15.72:0/783292262 shutdown_connections 2024-04-26T23:34:27.609 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:26.926+0000 7f522fc00700 1 -- 172.21.15.72:0/783292262 wait complete. 2024-04-26T23:34:28.505 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:28 smithi072 ceph-mon[28184]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:34:29.339 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-26T23:34:29.340 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.235+0000 7f6306600700 1 Processor -- start 2024-04-26T23:34:29.340 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.236+0000 7f6306600700 1 -- start start 2024-04-26T23:34:29.340 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.236+0000 7f6306600700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f630010b0f0 0x7f630010b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:29.340 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.236+0000 7f6306600700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f630010ba50 con 0x7f630010b0f0 2024-04-26T23:34:29.340 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.236+0000 7f62ff400700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f630010b0f0 0x7f630010b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:29.340 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.237+0000 7f62ff400700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f630010b0f0 0x7f630010b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34270/0 (socket says 172.21.15.72:34270) 2024-04-26T23:34:29.340 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.237+0000 7f62ff400700 1 -- 172.21.15.72:0/3294448337 learned_addr learned my addr 172.21.15.72:0/3294448337 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:34:29.340 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.237+0000 7f62ff400700 1 -- 172.21.15.72:0/3294448337 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f630010c0d0 con 0x7f630010b0f0 2024-04-26T23:34:29.340 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.237+0000 7f62ff400700 1 --2- 172.21.15.72:0/3294448337 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f630010b0f0 0x7f630010b510 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f62f00098a0 tx=0x7f62f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75ccc0dfd3e421d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:29.340 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.238+0000 7f62fe000700 1 -- 172.21.15.72:0/3294448337 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f62f0004160 con 0x7f630010b0f0 2024-04-26T23:34:29.340 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.238+0000 7f62fe000700 1 -- 172.21.15.72:0/3294448337 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f62f0004300 con 0x7f630010b0f0 2024-04-26T23:34:29.340 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.238+0000 7f62fe000700 1 -- 172.21.15.72:0/3294448337 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f62f0011af0 con 0x7f630010b0f0 2024-04-26T23:34:29.340 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.238+0000 7f6306600700 1 -- 172.21.15.72:0/3294448337 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f630010b0f0 msgr2=0x7f630010b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:29.341 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.238+0000 7f6306600700 1 --2- 172.21.15.72:0/3294448337 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f630010b0f0 0x7f630010b510 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f62f00098a0 tx=0x7f62f0009bb0 comp rx=0 tx=0).stop 2024-04-26T23:34:29.341 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.238+0000 7f6306600700 1 -- 172.21.15.72:0/3294448337 shutdown_connections 2024-04-26T23:34:29.341 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.238+0000 7f6306600700 1 --2- 172.21.15.72:0/3294448337 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f630010b0f0 0x7f630010b510 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:29.341 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.238+0000 7f6306600700 1 -- 172.21.15.72:0/3294448337 >> 172.21.15.72:0/3294448337 conn(0x7f630007e390 msgr2=0x7f630007e7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:34:29.341 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.239+0000 7f6306600700 1 -- 172.21.15.72:0/3294448337 shutdown_connections 2024-04-26T23:34:29.341 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.239+0000 7f6306600700 1 -- 172.21.15.72:0/3294448337 wait complete. 2024-04-26T23:34:29.341 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.240+0000 7f6306600700 1 Processor -- start 2024-04-26T23:34:29.341 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.240+0000 7f6306600700 1 -- start start 2024-04-26T23:34:29.341 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.240+0000 7f6306600700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f630010b0f0 0x7f630019ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:29.341 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.240+0000 7f6306600700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f630010f390 con 0x7f630010b0f0 2024-04-26T23:34:29.341 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.240+0000 7f62ff400700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f630010b0f0 0x7f630019ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:29.341 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.240+0000 7f62ff400700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f630010b0f0 0x7f630019ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34272/0 (socket says 172.21.15.72:34272) 2024-04-26T23:34:29.341 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.240+0000 7f62ff400700 1 -- 172.21.15.72:0/1508184992 learned_addr learned my addr 172.21.15.72:0/1508184992 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:34:29.341 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.241+0000 7f62ff400700 1 -- 172.21.15.72:0/1508184992 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f630019f480 con 0x7f630010b0f0 2024-04-26T23:34:29.341 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.241+0000 7f62ff400700 1 --2- 172.21.15.72:0/1508184992 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f630010b0f0 0x7f630019ef40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f62f0009fb0 tx=0x7f62f0011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:29.342 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.241+0000 7f62f7e00700 1 -- 172.21.15.72:0/1508184992 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f62f00036a0 con 0x7f630010b0f0 2024-04-26T23:34:29.342 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.241+0000 7f62f7e00700 1 -- 172.21.15.72:0/1508184992 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f62f001a430 con 0x7f630010b0f0 2024-04-26T23:34:29.342 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.241+0000 7f6306600700 1 -- 172.21.15.72:0/1508184992 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f630019f710 con 0x7f630010b0f0 2024-04-26T23:34:29.342 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.241+0000 7f62f7e00700 1 -- 172.21.15.72:0/1508184992 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f62f0019550 con 0x7f630010b0f0 2024-04-26T23:34:29.342 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.241+0000 7f6306600700 1 -- 172.21.15.72:0/1508184992 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63001a2400 con 0x7f630010b0f0 2024-04-26T23:34:29.342 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.242+0000 7f6306600700 1 -- 172.21.15.72:0/1508184992 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f630010f510 con 0x7f630010b0f0 2024-04-26T23:34:29.342 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.242+0000 7f62f7e00700 1 -- 172.21.15.72:0/1508184992 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f62f001a8e0 con 0x7f630010b0f0 2024-04-26T23:34:29.342 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.243+0000 7f62f7e00700 1 --2- 172.21.15.72:0/1508184992 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7f62e4039560 0x7f62e403ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:29.342 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.243+0000 7f62f7e00700 1 -- 172.21.15.72:0/1508184992 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f62f004d9c0 con 0x7f630010b0f0 2024-04-26T23:34:29.342 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.244+0000 7f62fea00700 1 --2- 172.21.15.72:0/1508184992 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7f62e4039560 0x7f62e403ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:29.342 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.244+0000 7f62fea00700 1 --2- 172.21.15.72:0/1508184992 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7f62e4039560 0x7f62e403ba20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f62ec006fd0 tx=0x7f62ec006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:29.342 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.249+0000 7f62f7e00700 1 -- 172.21.15.72:0/1508184992 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f62f001a690 con 0x7f630010b0f0 2024-04-26T23:34:29.342 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.479+0000 7f6306600700 1 -- 172.21.15.72:0/1508184992 --> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f63001094c0 con 0x7f62e4039560 2024-04-26T23:34:29.343 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.480+0000 7f62f7e00700 1 -- 172.21.15.72:0/1508184992 <== mgr.14120 v2:172.21.15.72:6800/2903979134 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f63001094c0 con 0x7f62e4039560 2024-04-26T23:34:29.343 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.483+0000 7f6306600700 1 -- 172.21.15.72:0/1508184992 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7f62e4039560 msgr2=0x7f62e403ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:29.343 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.483+0000 7f6306600700 1 --2- 172.21.15.72:0/1508184992 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7f62e4039560 0x7f62e403ba20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f62ec006fd0 tx=0x7f62ec006e40 comp rx=0 tx=0).stop 2024-04-26T23:34:29.343 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.483+0000 7f6306600700 1 -- 172.21.15.72:0/1508184992 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f630010b0f0 msgr2=0x7f630019ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:29.343 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.483+0000 7f6306600700 1 --2- 172.21.15.72:0/1508184992 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f630010b0f0 0x7f630019ef40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f62f0009fb0 tx=0x7f62f0011940 comp rx=0 tx=0).stop 2024-04-26T23:34:29.343 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.483+0000 7f6306600700 1 -- 172.21.15.72:0/1508184992 shutdown_connections 2024-04-26T23:34:29.343 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.483+0000 7f6306600700 1 --2- 172.21.15.72:0/1508184992 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7f62e4039560 0x7f62e403ba20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:29.343 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.483+0000 7f6306600700 1 --2- 172.21.15.72:0/1508184992 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f630010b0f0 0x7f630019ef40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:29.343 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.483+0000 7f6306600700 1 -- 172.21.15.72:0/1508184992 >> 172.21.15.72:0/1508184992 conn(0x7f630007e390 msgr2=0x7f6300108db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:34:29.343 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.483+0000 7f6306600700 1 -- 172.21.15.72:0/1508184992 shutdown_connections 2024-04-26T23:34:29.343 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:28.483+0000 7f6306600700 1 -- 172.21.15.72:0/1508184992 wait complete. 2024-04-26T23:34:29.343 INFO:teuthology.orchestra.run.smithi072.stdout:Generating ssh key... 2024-04-26T23:34:29.672 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:29 smithi072 ceph-mon[28184]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:34:30.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:30 smithi072 ceph-mon[28184]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:34:30.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:30 smithi072 ceph-mon[28184]: Generating ssh key... 2024-04-26T23:34:30.794 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:30 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: Generating public/private rsa key pair. 2024-04-26T23:34:30.794 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:30 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: Your identification has been saved in /tmp/tmpe79y66ve/key. 2024-04-26T23:34:30.794 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:30 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: Your public key has been saved in /tmp/tmpe79y66ve/key.pub. 2024-04-26T23:34:30.794 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:30 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: The key fingerprint is: 2024-04-26T23:34:30.794 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:30 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: SHA256:laCFEAYO2o6DTmvu0nIOFjem1467uMQY1PQSz/g0e/E ceph-3c8453ae-0425-11ef-bc93-c7b262605968 2024-04-26T23:34:30.794 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:30 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: The key's randomart image is: 2024-04-26T23:34:30.794 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:30 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: +---[RSA 3072]----+ 2024-04-26T23:34:30.794 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:30 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: |. .++o .o | 2024-04-26T23:34:30.794 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:30 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: |.+o.* .o . . | 2024-04-26T23:34:30.794 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:30 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: |..o+ *.. o | 2024-04-26T23:34:30.795 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:30 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: |oo + o o. | 2024-04-26T23:34:30.795 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:30 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: |=o.+ o .SE | 2024-04-26T23:34:30.795 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:30 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: |=o* o . | 2024-04-26T23:34:30.795 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:30 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: |oX . . | 2024-04-26T23:34:30.795 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:30 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: |B.= o | 2024-04-26T23:34:30.795 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:30 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: |oOo+o. | 2024-04-26T23:34:30.795 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:30 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: +----[SHA256]-----+ 2024-04-26T23:34:31.222 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:29.975+0000 7efd99600700 1 Processor -- start 2024-04-26T23:34:31.222 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:29.975+0000 7efd99600700 1 -- start start 2024-04-26T23:34:31.222 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:29.976+0000 7efd99600700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd9407ee70 0x7efd9407d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:31.222 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:29.976+0000 7efd99600700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7efd9407d8a0 con 0x7efd9407ee70 2024-04-26T23:34:31.222 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:29.976+0000 7efd92a00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd9407ee70 0x7efd9407d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:31.227 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:29.976+0000 7efd92a00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd9407ee70 0x7efd9407d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34280/0 (socket says 172.21.15.72:34280) 2024-04-26T23:34:31.228 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:29.976+0000 7efd92a00700 1 -- 172.21.15.72:0/2301573035 learned_addr learned my addr 172.21.15.72:0/2301573035 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:34:31.228 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:29.977+0000 7efd92a00700 1 -- 172.21.15.72:0/2301573035 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd9407da20 con 0x7efd9407ee70 2024-04-26T23:34:31.228 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:29.977+0000 7efd92a00700 1 --2- 172.21.15.72:0/2301573035 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd9407ee70 0x7efd9407d2d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7efd840098a0 tx=0x7efd84009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8370b80cacece61 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:31.228 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:29.977+0000 7efd91600700 1 -- 172.21.15.72:0/2301573035 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efd84004160 con 0x7efd9407ee70 2024-04-26T23:34:31.228 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:29.977+0000 7efd91600700 1 -- 172.21.15.72:0/2301573035 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7efd84004300 con 0x7efd9407ee70 2024-04-26T23:34:31.228 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:29.977+0000 7efd91600700 1 -- 172.21.15.72:0/2301573035 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efd84011af0 con 0x7efd9407ee70 2024-04-26T23:34:31.228 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:29.978+0000 7efd99600700 1 -- 172.21.15.72:0/2301573035 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd9407ee70 msgr2=0x7efd9407d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:31.228 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:29.978+0000 7efd99600700 1 --2- 172.21.15.72:0/2301573035 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd9407ee70 0x7efd9407d2d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7efd840098a0 tx=0x7efd84009bb0 comp rx=0 tx=0).stop 2024-04-26T23:34:31.228 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:29.978+0000 7efd99600700 1 -- 172.21.15.72:0/2301573035 shutdown_connections 2024-04-26T23:34:31.228 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:29.978+0000 7efd99600700 1 --2- 172.21.15.72:0/2301573035 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd9407ee70 0x7efd9407d2d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:31.228 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:29.978+0000 7efd99600700 1 -- 172.21.15.72:0/2301573035 >> 172.21.15.72:0/2301573035 conn(0x7efd94104330 msgr2=0x7efd94106770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:34:31.228 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:29.978+0000 7efd99600700 1 -- 172.21.15.72:0/2301573035 shutdown_connections 2024-04-26T23:34:31.229 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:29.978+0000 7efd99600700 1 -- 172.21.15.72:0/2301573035 wait complete. 2024-04-26T23:34:31.229 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:29.979+0000 7efd99600700 1 Processor -- start 2024-04-26T23:34:31.229 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:29.979+0000 7efd99600700 1 -- start start 2024-04-26T23:34:31.229 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:29.980+0000 7efd99600700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd9407ee70 0x7efd9419eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:31.229 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:29.980+0000 7efd99600700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7efd9410f360 con 0x7efd9407ee70 2024-04-26T23:34:31.229 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:29.980+0000 7efd92a00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd9407ee70 0x7efd9419eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:31.229 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:29.980+0000 7efd92a00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd9407ee70 0x7efd9419eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34292/0 (socket says 172.21.15.72:34292) 2024-04-26T23:34:31.229 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:29.980+0000 7efd92a00700 1 -- 172.21.15.72:0/2481127119 learned_addr learned my addr 172.21.15.72:0/2481127119 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:34:31.229 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:29.980+0000 7efd92a00700 1 -- 172.21.15.72:0/2481127119 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd9419f410 con 0x7efd9407ee70 2024-04-26T23:34:31.229 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:29.981+0000 7efd92a00700 1 --2- 172.21.15.72:0/2481127119 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd9407ee70 0x7efd9419eed0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7efd84011ac0 tx=0x7efd84011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:31.229 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:29.981+0000 7efd8b400700 1 -- 172.21.15.72:0/2481127119 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efd84003970 con 0x7efd9407ee70 2024-04-26T23:34:31.229 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:29.981+0000 7efd8b400700 1 -- 172.21.15.72:0/2481127119 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7efd84003b10 con 0x7efd9407ee70 2024-04-26T23:34:31.229 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:29.981+0000 7efd8b400700 1 -- 172.21.15.72:0/2481127119 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efd84024610 con 0x7efd9407ee70 2024-04-26T23:34:31.230 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:29.981+0000 7efd99600700 1 -- 172.21.15.72:0/2481127119 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd9419f6a0 con 0x7efd9407ee70 2024-04-26T23:34:31.230 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:29.981+0000 7efd99600700 1 -- 172.21.15.72:0/2481127119 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd941a2390 con 0x7efd9407ee70 2024-04-26T23:34:31.230 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:29.982+0000 7efd8b400700 1 -- 172.21.15.72:0/2481127119 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7efd8401a9e0 con 0x7efd9407ee70 2024-04-26T23:34:31.230 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:29.982+0000 7efd99600700 1 -- 172.21.15.72:0/2481127119 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd94064a80 con 0x7efd9407ee70 2024-04-26T23:34:31.230 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:29.983+0000 7efd8b400700 1 --2- 172.21.15.72:0/2481127119 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7efd78039600 0x7efd7803bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:31.230 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:29.983+0000 7efd8b400700 1 -- 172.21.15.72:0/2481127119 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7efd8404d980 con 0x7efd9407ee70 2024-04-26T23:34:31.230 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:29.983+0000 7efd92000700 1 --2- 172.21.15.72:0/2481127119 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7efd78039600 0x7efd7803bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:31.230 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:29.984+0000 7efd92000700 1 --2- 172.21.15.72:0/2481127119 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7efd78039600 0x7efd7803bac0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7efd80006fd0 tx=0x7efd80006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:31.230 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:29.988+0000 7efd8b400700 1 -- 172.21.15.72:0/2481127119 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7efd8400b3e0 con 0x7efd9407ee70 2024-04-26T23:34:31.230 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:30.213+0000 7efd99600700 1 -- 172.21.15.72:0/2481127119 --> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7efd94105600 con 0x7efd78039600 2024-04-26T23:34:31.230 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:30.496+0000 7efd8b400700 1 -- 172.21.15.72:0/2481127119 <== mgr.14120 v2:172.21.15.72:6800/2903979134 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7efd94105600 con 0x7efd78039600 2024-04-26T23:34:31.230 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:30.499+0000 7efd99600700 1 -- 172.21.15.72:0/2481127119 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7efd78039600 msgr2=0x7efd7803bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:31.230 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:30.499+0000 7efd99600700 1 --2- 172.21.15.72:0/2481127119 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7efd78039600 0x7efd7803bac0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7efd80006fd0 tx=0x7efd80006e40 comp rx=0 tx=0).stop 2024-04-26T23:34:31.231 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:30.499+0000 7efd99600700 1 -- 172.21.15.72:0/2481127119 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd9407ee70 msgr2=0x7efd9419eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:31.231 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:30.499+0000 7efd99600700 1 --2- 172.21.15.72:0/2481127119 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd9407ee70 0x7efd9419eed0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7efd84011ac0 tx=0x7efd84011ba0 comp rx=0 tx=0).stop 2024-04-26T23:34:31.231 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:30.500+0000 7efd99600700 1 -- 172.21.15.72:0/2481127119 shutdown_connections 2024-04-26T23:34:31.231 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:30.500+0000 7efd99600700 1 --2- 172.21.15.72:0/2481127119 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7efd78039600 0x7efd7803bac0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:31.231 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:30.500+0000 7efd99600700 1 --2- 172.21.15.72:0/2481127119 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd9407ee70 0x7efd9419eed0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:31.231 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:30.500+0000 7efd99600700 1 -- 172.21.15.72:0/2481127119 >> 172.21.15.72:0/2481127119 conn(0x7efd94104330 msgr2=0x7efd94104ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:34:31.231 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:30.500+0000 7efd99600700 1 -- 172.21.15.72:0/2481127119 shutdown_connections 2024-04-26T23:34:31.231 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:30.500+0000 7efd99600700 1 -- 172.21.15.72:0/2481127119 wait complete. 2024-04-26T23:34:31.753 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:31 smithi072 ceph-mon[28184]: from='mgr.14120 172.21.15.72:0/3482651254' entity='mgr.x' 2024-04-26T23:34:31.754 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:31 smithi072 ceph-mon[28184]: from='mgr.14120 172.21.15.72:0/3482651254' entity='mgr.x' 2024-04-26T23:34:32.627 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:32 smithi072 ceph-mon[28184]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:34:32.641 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQ8x+3x4cLVQ0JokUHrmSSlrt+iv2TrdIBotlq7h6PSXdQuM04AokPyvhQ0c+lcsrmeRA+epFiFOrD6Ms5mS/jJugqZFnFkakiHBdWXVO+zD/Zls8m/VPZF4MU9tFU4SJRG6KvxaMeO9/qncAUSiB2IIbOfM4TknJ0A73UU7AohshgENnurjWTif4PX2OVMqa1j0jkSsg1iupCZ7QKg/+6soF+uAHhFq9QqeM2G/k8Q4nzWmG0Woonkn561tB7rsAiJ4MyTvBUeKSdPtO+E+yFRFwQgPyZXjsAcDMLB7/vQA9JEmFCuyjl0OLefxmy+1KyTDSPvDOxNdFrTEHtz1/K9i661U/tujnDENADNZma+UCQGiB9OE9arzmlZWjb3EZkXTJn247mo2fKWPbiQcld+fUmvgi4PaYmIbHWwePlVOhFMeFlh9lN+4WIto15WVYstQqAsJ3ZB4s0pU2M61k873GMX5OO5sHEzUh+nWxZoO5Zflp/NBPYhIVa7ma+onM= ceph-3c8453ae-0425-11ef-bc93-c7b262605968 2024-04-26T23:34:32.641 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:31.795+0000 7fd6f5a00700 1 Processor -- start 2024-04-26T23:34:32.641 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:31.795+0000 7fd6f5a00700 1 -- start start 2024-04-26T23:34:32.641 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:31.796+0000 7fd6f5a00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd6f0108d70 0x7fd6f0109190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:32.641 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:31.796+0000 7fd6f5a00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f0109760 con 0x7fd6f0108d70 2024-04-26T23:34:32.641 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:31.796+0000 7fd6eea00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd6f0108d70 0x7fd6f0109190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:32.641 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:31.796+0000 7fd6eea00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd6f0108d70 0x7fd6f0109190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34296/0 (socket says 172.21.15.72:34296) 2024-04-26T23:34:32.641 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:31.796+0000 7fd6eea00700 1 -- 172.21.15.72:0/692450850 learned_addr learned my addr 172.21.15.72:0/692450850 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:34:32.641 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:31.797+0000 7fd6eea00700 1 -- 172.21.15.72:0/692450850 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6f0109f70 con 0x7fd6f0108d70 2024-04-26T23:34:32.641 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:31.797+0000 7fd6eea00700 1 --2- 172.21.15.72:0/692450850 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd6f0108d70 0x7fd6f0109190 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd6e00098a0 tx=0x7fd6e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=567669b19651c118 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:32.641 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:31.797+0000 7fd6ee000700 1 -- 172.21.15.72:0/692450850 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd6e0004160 con 0x7fd6f0108d70 2024-04-26T23:34:32.641 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:31.797+0000 7fd6ee000700 1 -- 172.21.15.72:0/692450850 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd6e0004300 con 0x7fd6f0108d70 2024-04-26T23:34:32.642 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:31.797+0000 7fd6ee000700 1 -- 172.21.15.72:0/692450850 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd6e0011af0 con 0x7fd6f0108d70 2024-04-26T23:34:32.642 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:31.798+0000 7fd6f5a00700 1 -- 172.21.15.72:0/692450850 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd6f0108d70 msgr2=0x7fd6f0109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:32.642 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:31.798+0000 7fd6f5a00700 1 --2- 172.21.15.72:0/692450850 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd6f0108d70 0x7fd6f0109190 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd6e00098a0 tx=0x7fd6e0009bb0 comp rx=0 tx=0).stop 2024-04-26T23:34:32.642 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:31.798+0000 7fd6f5a00700 1 -- 172.21.15.72:0/692450850 shutdown_connections 2024-04-26T23:34:32.642 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:31.798+0000 7fd6f5a00700 1 --2- 172.21.15.72:0/692450850 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd6f0108d70 0x7fd6f0109190 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:32.642 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:31.798+0000 7fd6f5a00700 1 -- 172.21.15.72:0/692450850 >> 172.21.15.72:0/692450850 conn(0x7fd6f01042d0 msgr2=0x7fd6f0106750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:34:32.642 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:31.798+0000 7fd6f5a00700 1 -- 172.21.15.72:0/692450850 shutdown_connections 2024-04-26T23:34:32.642 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:31.798+0000 7fd6f5a00700 1 -- 172.21.15.72:0/692450850 wait complete. 2024-04-26T23:34:32.642 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:31.799+0000 7fd6f5a00700 1 Processor -- start 2024-04-26T23:34:32.642 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:31.800+0000 7fd6f5a00700 1 -- start start 2024-04-26T23:34:32.642 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:31.800+0000 7fd6f5a00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd6f0108d70 0x7fd6f007eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:32.642 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:31.800+0000 7fd6f5a00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f010d320 con 0x7fd6f0108d70 2024-04-26T23:34:32.642 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:31.800+0000 7fd6eea00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd6f0108d70 0x7fd6f007eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:32.642 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:31.800+0000 7fd6eea00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd6f0108d70 0x7fd6f007eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34304/0 (socket says 172.21.15.72:34304) 2024-04-26T23:34:32.643 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:31.800+0000 7fd6eea00700 1 -- 172.21.15.72:0/2854515522 learned_addr learned my addr 172.21.15.72:0/2854515522 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:34:32.643 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:31.800+0000 7fd6eea00700 1 -- 172.21.15.72:0/2854515522 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6f007f0a0 con 0x7fd6f0108d70 2024-04-26T23:34:32.643 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:31.801+0000 7fd6eea00700 1 --2- 172.21.15.72:0/2854515522 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd6f0108d70 0x7fd6f007eb60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd6e00117b0 tx=0x7fd6e0011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:32.643 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:31.801+0000 7fd6e7400700 1 -- 172.21.15.72:0/2854515522 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd6e0003af0 con 0x7fd6f0108d70 2024-04-26T23:34:32.643 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:31.801+0000 7fd6e7400700 1 -- 172.21.15.72:0/2854515522 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd6e0020070 con 0x7fd6f0108d70 2024-04-26T23:34:32.643 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:31.801+0000 7fd6e7400700 1 -- 172.21.15.72:0/2854515522 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd6e0019630 con 0x7fd6f0108d70 2024-04-26T23:34:32.643 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:31.801+0000 7fd6f5a00700 1 -- 172.21.15.72:0/2854515522 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6f007d290 con 0x7fd6f0108d70 2024-04-26T23:34:32.643 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:31.801+0000 7fd6f5a00700 1 -- 172.21.15.72:0/2854515522 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6f007d7d0 con 0x7fd6f0108d70 2024-04-26T23:34:32.643 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:31.802+0000 7fd6e7400700 1 -- 172.21.15.72:0/2854515522 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fd6e001a9c0 con 0x7fd6f0108d70 2024-04-26T23:34:32.643 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:31.802+0000 7fd6f5a00700 1 -- 172.21.15.72:0/2854515522 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6f0064a80 con 0x7fd6f0108d70 2024-04-26T23:34:32.643 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:31.803+0000 7fd6e7400700 1 --2- 172.21.15.72:0/2854515522 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7fd6d4039600 0x7fd6d403bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:32.643 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:31.803+0000 7fd6e7400700 1 -- 172.21.15.72:0/2854515522 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fd6e004dbf0 con 0x7fd6f0108d70 2024-04-26T23:34:32.643 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:31.803+0000 7fd6e7e00700 1 --2- 172.21.15.72:0/2854515522 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7fd6d4039600 0x7fd6d403bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:32.643 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:31.804+0000 7fd6e7e00700 1 --2- 172.21.15.72:0/2854515522 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7fd6d4039600 0x7fd6d403bac0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd6d8006fd0 tx=0x7fd6d8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:32.643 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:31.808+0000 7fd6e7400700 1 -- 172.21.15.72:0/2854515522 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd6e00042a0 con 0x7fd6f0108d70 2024-04-26T23:34:32.644 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:32.036+0000 7fd6f5a00700 1 -- 172.21.15.72:0/2854515522 --> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7fd6f01055a0 con 0x7fd6d4039600 2024-04-26T23:34:32.644 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:32.037+0000 7fd6e7400700 1 -- 172.21.15.72:0/2854515522 <== mgr.14120 v2:172.21.15.72:6800/2903979134 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7fd6f01055a0 con 0x7fd6d4039600 2024-04-26T23:34:32.644 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:32.040+0000 7fd6f5a00700 1 -- 172.21.15.72:0/2854515522 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7fd6d4039600 msgr2=0x7fd6d403bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:32.644 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:32.040+0000 7fd6f5a00700 1 --2- 172.21.15.72:0/2854515522 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7fd6d4039600 0x7fd6d403bac0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd6d8006fd0 tx=0x7fd6d8006e40 comp rx=0 tx=0).stop 2024-04-26T23:34:32.644 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:32.040+0000 7fd6f5a00700 1 -- 172.21.15.72:0/2854515522 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd6f0108d70 msgr2=0x7fd6f007eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:32.644 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:32.040+0000 7fd6f5a00700 1 --2- 172.21.15.72:0/2854515522 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd6f0108d70 0x7fd6f007eb60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd6e00117b0 tx=0x7fd6e0011b70 comp rx=0 tx=0).stop 2024-04-26T23:34:32.644 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:32.040+0000 7fd6f5a00700 1 -- 172.21.15.72:0/2854515522 shutdown_connections 2024-04-26T23:34:32.644 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:32.040+0000 7fd6f5a00700 1 --2- 172.21.15.72:0/2854515522 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7fd6d4039600 0x7fd6d403bac0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:32.644 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:32.040+0000 7fd6f5a00700 1 --2- 172.21.15.72:0/2854515522 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd6f0108d70 0x7fd6f007eb60 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:32.644 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:32.040+0000 7fd6f5a00700 1 -- 172.21.15.72:0/2854515522 >> 172.21.15.72:0/2854515522 conn(0x7fd6f01042d0 msgr2=0x7fd6f0104e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:34:32.644 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:32.041+0000 7fd6f5a00700 1 -- 172.21.15.72:0/2854515522 shutdown_connections 2024-04-26T23:34:32.644 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:32.041+0000 7fd6f5a00700 1 -- 172.21.15.72:0/2854515522 wait complete. 2024-04-26T23:34:32.644 INFO:teuthology.orchestra.run.smithi072.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-26T23:34:32.644 INFO:teuthology.orchestra.run.smithi072.stdout:Adding key to root@localhost authorized_keys... 2024-04-26T23:34:32.649 INFO:teuthology.orchestra.run.smithi072.stdout:Adding host smithi072... 2024-04-26T23:34:34.514 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:34 smithi072 ceph-mon[28184]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi072", "addr": "172.21.15.72", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:34:35.662 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:35 smithi072 ceph-mon[28184]: Deploying cephadm binary to smithi072 2024-04-26T23:34:40.301 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout Added host 'smithi072' with addr '172.21.15.72' 2024-04-26T23:34:40.301 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:33.281+0000 7fc211a00700 1 Processor -- start 2024-04-26T23:34:40.302 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:33.281+0000 7fc211a00700 1 -- start start 2024-04-26T23:34:40.302 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:33.281+0000 7fc211a00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc20c07f710 0x7fc20c07dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:40.302 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:33.281+0000 7fc211a00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fc20c07e190 con 0x7fc20c07f710 2024-04-26T23:34:40.302 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:33.282+0000 7fc20aa00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc20c07f710 0x7fc20c07dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:40.302 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:33.282+0000 7fc20aa00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc20c07f710 0x7fc20c07dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34316/0 (socket says 172.21.15.72:34316) 2024-04-26T23:34:40.302 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:33.282+0000 7fc20aa00700 1 -- 172.21.15.72:0/1127026567 learned_addr learned my addr 172.21.15.72:0/1127026567 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:34:40.302 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:33.282+0000 7fc20aa00700 1 -- 172.21.15.72:0/1127026567 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc20c07e310 con 0x7fc20c07f710 2024-04-26T23:34:40.302 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:33.282+0000 7fc20aa00700 1 --2- 172.21.15.72:0/1127026567 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc20c07f710 0x7fc20c07dbc0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc0098a0 tx=0x7fc1fc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa14d5ccf37186ea server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:40.302 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:33.283+0000 7fc209600700 1 -- 172.21.15.72:0/1127026567 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc1fc004160 con 0x7fc20c07f710 2024-04-26T23:34:40.302 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:33.283+0000 7fc209600700 1 -- 172.21.15.72:0/1127026567 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc1fc004300 con 0x7fc20c07f710 2024-04-26T23:34:40.302 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:33.283+0000 7fc209600700 1 -- 172.21.15.72:0/1127026567 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc1fc011af0 con 0x7fc20c07f710 2024-04-26T23:34:40.302 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:33.283+0000 7fc211a00700 1 -- 172.21.15.72:0/1127026567 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc20c07f710 msgr2=0x7fc20c07dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:40.303 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:33.283+0000 7fc211a00700 1 --2- 172.21.15.72:0/1127026567 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc20c07f710 0x7fc20c07dbc0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc0098a0 tx=0x7fc1fc009bb0 comp rx=0 tx=0).stop 2024-04-26T23:34:40.303 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:33.284+0000 7fc211a00700 1 -- 172.21.15.72:0/1127026567 shutdown_connections 2024-04-26T23:34:40.303 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:33.284+0000 7fc211a00700 1 --2- 172.21.15.72:0/1127026567 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc20c07f710 0x7fc20c07dbc0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:40.303 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:33.284+0000 7fc211a00700 1 -- 172.21.15.72:0/1127026567 >> 172.21.15.72:0/1127026567 conn(0x7fc20c104370 msgr2=0x7fc20c1067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:34:40.303 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:33.284+0000 7fc211a00700 1 -- 172.21.15.72:0/1127026567 shutdown_connections 2024-04-26T23:34:40.303 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:33.284+0000 7fc211a00700 1 -- 172.21.15.72:0/1127026567 wait complete. 2024-04-26T23:34:40.303 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:33.285+0000 7fc211a00700 1 Processor -- start 2024-04-26T23:34:40.303 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:33.285+0000 7fc211a00700 1 -- start start 2024-04-26T23:34:40.303 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:33.285+0000 7fc211a00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc20c07f710 0x7fc20c1a32c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:40.303 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:33.285+0000 7fc211a00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fc20c10af30 con 0x7fc20c07f710 2024-04-26T23:34:40.303 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:33.285+0000 7fc20aa00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc20c07f710 0x7fc20c1a32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:40.303 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:33.285+0000 7fc20aa00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc20c07f710 0x7fc20c1a32c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34326/0 (socket says 172.21.15.72:34326) 2024-04-26T23:34:40.303 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:33.285+0000 7fc20aa00700 1 -- 172.21.15.72:0/1603666210 learned_addr learned my addr 172.21.15.72:0/1603666210 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:34:40.303 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:33.286+0000 7fc20aa00700 1 -- 172.21.15.72:0/1603666210 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc20c1a3800 con 0x7fc20c07f710 2024-04-26T23:34:40.304 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:33.286+0000 7fc20aa00700 1 --2- 172.21.15.72:0/1603666210 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc20c07f710 0x7fc20c1a32c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc011ac0 tx=0x7fc1fc011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:40.304 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:33.286+0000 7fc203400700 1 -- 172.21.15.72:0/1603666210 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc1fc003970 con 0x7fc20c07f710 2024-04-26T23:34:40.304 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:33.286+0000 7fc203400700 1 -- 172.21.15.72:0/1603666210 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc1fc003b10 con 0x7fc20c07f710 2024-04-26T23:34:40.304 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:33.286+0000 7fc211a00700 1 -- 172.21.15.72:0/1603666210 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc20c1a3a90 con 0x7fc20c07f710 2024-04-26T23:34:40.304 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:33.286+0000 7fc203400700 1 -- 172.21.15.72:0/1603666210 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc1fc024610 con 0x7fc20c07f710 2024-04-26T23:34:40.304 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:33.286+0000 7fc211a00700 1 -- 172.21.15.72:0/1603666210 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc20c1a6780 con 0x7fc20c07f710 2024-04-26T23:34:40.304 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:33.288+0000 7fc203400700 1 -- 172.21.15.72:0/1603666210 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fc1fc024840 con 0x7fc20c07f710 2024-04-26T23:34:40.304 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:33.288+0000 7fc211a00700 1 -- 172.21.15.72:0/1603666210 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc20c064a80 con 0x7fc20c07f710 2024-04-26T23:34:40.304 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:33.288+0000 7fc203400700 1 --2- 172.21.15.72:0/1603666210 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7fc1f0039600 0x7fc1f003bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:40.304 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:33.288+0000 7fc203400700 1 -- 172.21.15.72:0/1603666210 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc1fc04d970 con 0x7fc20c07f710 2024-04-26T23:34:40.304 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:33.288+0000 7fc20a000700 1 --2- 172.21.15.72:0/1603666210 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7fc1f0039600 0x7fc1f003bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:40.304 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:33.289+0000 7fc20a000700 1 --2- 172.21.15.72:0/1603666210 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7fc1f0039600 0x7fc1f003bac0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc1f8006fd0 tx=0x7fc1f8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:40.304 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:33.294+0000 7fc203400700 1 -- 172.21.15.72:0/1603666210 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc1fc0172b0 con 0x7fc20c07f710 2024-04-26T23:34:40.304 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:33.521+0000 7fc211a00700 1 -- 172.21.15.72:0/1603666210 --> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi072", "addr": "172.21.15.72", "target": ["mon-mgr", ""]}) -- 0x7fc20c0008d0 con 0x7fc1f0039600 2024-04-26T23:34:40.305 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:39.563+0000 7fc203400700 1 -- 172.21.15.72:0/1603666210 <== mgr.14120 v2:172.21.15.72:6800/2903979134 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7fc20c0008d0 con 0x7fc1f0039600 2024-04-26T23:34:40.305 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:39.567+0000 7fc211a00700 1 -- 172.21.15.72:0/1603666210 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7fc1f0039600 msgr2=0x7fc1f003bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:40.305 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:39.567+0000 7fc211a00700 1 --2- 172.21.15.72:0/1603666210 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7fc1f0039600 0x7fc1f003bac0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc1f8006fd0 tx=0x7fc1f8006e40 comp rx=0 tx=0).stop 2024-04-26T23:34:40.305 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:39.567+0000 7fc211a00700 1 -- 172.21.15.72:0/1603666210 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc20c07f710 msgr2=0x7fc20c1a32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:40.305 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:39.567+0000 7fc211a00700 1 --2- 172.21.15.72:0/1603666210 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc20c07f710 0x7fc20c1a32c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc011ac0 tx=0x7fc1fc011ba0 comp rx=0 tx=0).stop 2024-04-26T23:34:40.305 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:39.567+0000 7fc211a00700 1 -- 172.21.15.72:0/1603666210 shutdown_connections 2024-04-26T23:34:40.305 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:39.567+0000 7fc211a00700 1 --2- 172.21.15.72:0/1603666210 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7fc1f0039600 0x7fc1f003bac0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:40.305 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:39.567+0000 7fc211a00700 1 --2- 172.21.15.72:0/1603666210 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc20c07f710 0x7fc20c1a32c0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:40.305 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:39.567+0000 7fc211a00700 1 -- 172.21.15.72:0/1603666210 >> 172.21.15.72:0/1603666210 conn(0x7fc20c104370 msgr2=0x7fc20c1067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:34:40.305 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:39.567+0000 7fc211a00700 1 -- 172.21.15.72:0/1603666210 shutdown_connections 2024-04-26T23:34:40.305 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:39.567+0000 7fc211a00700 1 -- 172.21.15.72:0/1603666210 wait complete. 2024-04-26T23:34:40.305 INFO:teuthology.orchestra.run.smithi072.stdout:Deploying unmanaged mon service... 2024-04-26T23:34:40.828 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:40 smithi072 ceph-mon[28184]: from='mgr.14120 172.21.15.72:0/3482651254' entity='mgr.x' 2024-04-26T23:34:40.828 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:40 smithi072 ceph-mon[28184]: Added host smithi072 2024-04-26T23:34:40.828 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:40 smithi072 ceph-mon[28184]: from='mgr.14120 172.21.15.72:0/3482651254' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:34:41.788 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-26T23:34:41.788 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:40.879+0000 7f64f5e00700 1 Processor -- start 2024-04-26T23:34:41.788 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:40.879+0000 7f64f5e00700 1 -- start start 2024-04-26T23:34:41.788 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:40.880+0000 7f64f5e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64f010afd0 0x7f64f010b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:41.788 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:40.880+0000 7f64f5e00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f64f010b9c0 con 0x7f64f010afd0 2024-04-26T23:34:41.788 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:40.880+0000 7f64eea00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64f010afd0 0x7f64f010b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:41.788 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:40.880+0000 7f64eea00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64f010afd0 0x7f64f010b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56298/0 (socket says 172.21.15.72:56298) 2024-04-26T23:34:41.789 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:40.880+0000 7f64eea00700 1 -- 172.21.15.72:0/1263494034 learned_addr learned my addr 172.21.15.72:0/1263494034 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:34:41.789 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:40.881+0000 7f64eea00700 1 -- 172.21.15.72:0/1263494034 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64f010c180 con 0x7f64f010afd0 2024-04-26T23:34:41.789 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:40.881+0000 7f64eea00700 1 --2- 172.21.15.72:0/1263494034 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64f010afd0 0x7f64f010b3f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f64e00098a0 tx=0x7f64e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d83e3c795af315ec server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:41.789 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:40.881+0000 7f64ee000700 1 -- 172.21.15.72:0/1263494034 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f64e0004160 con 0x7f64f010afd0 2024-04-26T23:34:41.789 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:40.881+0000 7f64ee000700 1 -- 172.21.15.72:0/1263494034 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f64e0004300 con 0x7f64f010afd0 2024-04-26T23:34:41.789 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:40.881+0000 7f64ee000700 1 -- 172.21.15.72:0/1263494034 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f64e0011af0 con 0x7f64f010afd0 2024-04-26T23:34:41.789 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:40.882+0000 7f64f5e00700 1 -- 172.21.15.72:0/1263494034 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64f010afd0 msgr2=0x7f64f010b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:41.789 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:40.882+0000 7f64f5e00700 1 --2- 172.21.15.72:0/1263494034 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64f010afd0 0x7f64f010b3f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f64e00098a0 tx=0x7f64e0009bb0 comp rx=0 tx=0).stop 2024-04-26T23:34:41.789 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:40.882+0000 7f64f5e00700 1 -- 172.21.15.72:0/1263494034 shutdown_connections 2024-04-26T23:34:41.789 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:40.882+0000 7f64f5e00700 1 --2- 172.21.15.72:0/1263494034 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64f010afd0 0x7f64f010b3f0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:41.789 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:40.882+0000 7f64f5e00700 1 -- 172.21.15.72:0/1263494034 >> 172.21.15.72:0/1263494034 conn(0x7f64f007e6b0 msgr2=0x7f64f007eac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:34:41.789 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:40.882+0000 7f64f5e00700 1 -- 172.21.15.72:0/1263494034 shutdown_connections 2024-04-26T23:34:41.789 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:40.882+0000 7f64f5e00700 1 -- 172.21.15.72:0/1263494034 wait complete. 2024-04-26T23:34:41.790 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:40.883+0000 7f64f5e00700 1 Processor -- start 2024-04-26T23:34:41.790 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:40.884+0000 7f64f5e00700 1 -- start start 2024-04-26T23:34:41.790 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:40.884+0000 7f64f5e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64f010afd0 0x7f64f019ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:41.790 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:40.884+0000 7f64f5e00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f64f010f530 con 0x7f64f010afd0 2024-04-26T23:34:41.790 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:40.884+0000 7f64eea00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64f010afd0 0x7f64f019ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:41.790 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:40.884+0000 7f64eea00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64f010afd0 0x7f64f019ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56310/0 (socket says 172.21.15.72:56310) 2024-04-26T23:34:41.790 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:40.884+0000 7f64eea00700 1 -- 172.21.15.72:0/254348690 learned_addr learned my addr 172.21.15.72:0/254348690 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:34:41.790 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:40.884+0000 7f64eea00700 1 -- 172.21.15.72:0/254348690 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64f019f450 con 0x7f64f010afd0 2024-04-26T23:34:41.790 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:40.885+0000 7f64eea00700 1 --2- 172.21.15.72:0/254348690 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64f010afd0 0x7f64f019ef10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f64e00117b0 tx=0x7f64e0011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:41.790 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:40.885+0000 7f64f4a00700 1 -- 172.21.15.72:0/254348690 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f64e0003af0 con 0x7f64f010afd0 2024-04-26T23:34:41.790 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:40.885+0000 7f64f4a00700 1 -- 172.21.15.72:0/254348690 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f64e0020070 con 0x7f64f010afd0 2024-04-26T23:34:41.790 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:40.885+0000 7f64f5e00700 1 -- 172.21.15.72:0/254348690 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64f019f6e0 con 0x7f64f010afd0 2024-04-26T23:34:41.790 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:40.885+0000 7f64f4a00700 1 -- 172.21.15.72:0/254348690 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f64e0019630 con 0x7f64f010afd0 2024-04-26T23:34:41.790 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:40.885+0000 7f64f5e00700 1 -- 172.21.15.72:0/254348690 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64f01a23d0 con 0x7f64f010afd0 2024-04-26T23:34:41.791 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:40.886+0000 7f64f4a00700 1 -- 172.21.15.72:0/254348690 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f64e001a9c0 con 0x7f64f010afd0 2024-04-26T23:34:41.791 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:40.886+0000 7f64f5e00700 1 -- 172.21.15.72:0/254348690 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64f0064a80 con 0x7f64f010afd0 2024-04-26T23:34:41.791 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:40.887+0000 7f64f4a00700 1 --2- 172.21.15.72:0/254348690 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7f64d40395b0 0x7f64d403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:41.791 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:40.887+0000 7f64f4a00700 1 -- 172.21.15.72:0/254348690 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f64e004dc00 con 0x7f64f010afd0 2024-04-26T23:34:41.791 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:40.887+0000 7f64e6000700 1 --2- 172.21.15.72:0/254348690 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7f64d40395b0 0x7f64d403ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:41.791 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:40.888+0000 7f64e6000700 1 --2- 172.21.15.72:0/254348690 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7f64d40395b0 0x7f64d403ba70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f64d8006fd0 tx=0x7f64d8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:41.791 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:40.892+0000 7f64f4a00700 1 -- 172.21.15.72:0/254348690 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f64e000e260 con 0x7f64f010afd0 2024-04-26T23:34:41.791 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:41.130+0000 7f64f5e00700 1 -- 172.21.15.72:0/254348690 --> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f64f0108c10 con 0x7f64d40395b0 2024-04-26T23:34:41.791 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:41.140+0000 7f64f4a00700 1 -- 172.21.15.72:0/254348690 <== mgr.14120 v2:172.21.15.72:6800/2903979134 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f64f0108c10 con 0x7f64d40395b0 2024-04-26T23:34:41.791 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:41.143+0000 7f64f5e00700 1 -- 172.21.15.72:0/254348690 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7f64d40395b0 msgr2=0x7f64d403ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:41.791 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:41.143+0000 7f64f5e00700 1 --2- 172.21.15.72:0/254348690 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7f64d40395b0 0x7f64d403ba70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f64d8006fd0 tx=0x7f64d8006e40 comp rx=0 tx=0).stop 2024-04-26T23:34:41.791 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:41.143+0000 7f64f5e00700 1 -- 172.21.15.72:0/254348690 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64f010afd0 msgr2=0x7f64f019ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:41.791 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:41.143+0000 7f64f5e00700 1 --2- 172.21.15.72:0/254348690 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64f010afd0 0x7f64f019ef10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f64e00117b0 tx=0x7f64e0011b70 comp rx=0 tx=0).stop 2024-04-26T23:34:41.792 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:41.143+0000 7f64f5e00700 1 -- 172.21.15.72:0/254348690 shutdown_connections 2024-04-26T23:34:41.792 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:41.143+0000 7f64f5e00700 1 --2- 172.21.15.72:0/254348690 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7f64d40395b0 0x7f64d403ba70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:41.792 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:41.143+0000 7f64f5e00700 1 --2- 172.21.15.72:0/254348690 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64f010afd0 0x7f64f019ef10 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:41.792 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:41.143+0000 7f64f5e00700 1 -- 172.21.15.72:0/254348690 >> 172.21.15.72:0/254348690 conn(0x7f64f007e6b0 msgr2=0x7f64f0108500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:34:41.792 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:41.143+0000 7f64f5e00700 1 -- 172.21.15.72:0/254348690 shutdown_connections 2024-04-26T23:34:41.792 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:41.143+0000 7f64f5e00700 1 -- 172.21.15.72:0/254348690 wait complete. 2024-04-26T23:34:41.792 INFO:teuthology.orchestra.run.smithi072.stdout:Deploying unmanaged mgr service... 2024-04-26T23:34:42.378 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:42 smithi072 ceph-mon[28184]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:34:42.379 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:42 smithi072 ceph-mon[28184]: Saving service mon spec with placement count:5 2024-04-26T23:34:42.379 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:42 smithi072 ceph-mon[28184]: from='mgr.14120 172.21.15.72:0/3482651254' entity='mgr.x' 2024-04-26T23:34:43.994 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:43 smithi072 ceph-mon[28184]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:34:43.994 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:43 smithi072 ceph-mon[28184]: Saving service mgr spec with placement count:2 2024-04-26T23:34:43.994 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:43 smithi072 ceph-mon[28184]: from='mgr.14120 172.21.15.72:0/3482651254' entity='mgr.x' 2024-04-26T23:34:43.994 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:43 smithi072 ceph-mon[28184]: from='mgr.14120 172.21.15.72:0/3482651254' entity='mgr.x' 2024-04-26T23:34:44.008 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-26T23:34:44.008 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.582+0000 7fdcb4000700 1 Processor -- start 2024-04-26T23:34:44.008 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.582+0000 7fdcb4000700 1 -- start start 2024-04-26T23:34:44.008 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.582+0000 7fdcb4000700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcac109800 0x7fdcac10bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:44.008 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.582+0000 7fdcb4000700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fdcac07cea0 con 0x7fdcac109800 2024-04-26T23:34:44.009 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.583+0000 7fdcb1600700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcac109800 0x7fdcac10bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:44.009 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.583+0000 7fdcb1600700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcac109800 0x7fdcac10bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56318/0 (socket says 172.21.15.72:56318) 2024-04-26T23:34:44.009 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.583+0000 7fdcb1600700 1 -- 172.21.15.72:0/3015423577 learned_addr learned my addr 172.21.15.72:0/3015423577 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:34:44.009 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.583+0000 7fdcb1600700 1 -- 172.21.15.72:0/3015423577 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcac10c130 con 0x7fdcac109800 2024-04-26T23:34:44.009 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.584+0000 7fdcb1600700 1 --2- 172.21.15.72:0/3015423577 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcac109800 0x7fdcac10bbf0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdca00098a0 tx=0x7fdca0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8adfa37d9550d4c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:44.009 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.584+0000 7fdcabe00700 1 -- 172.21.15.72:0/3015423577 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdca0004160 con 0x7fdcac109800 2024-04-26T23:34:44.009 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.584+0000 7fdcabe00700 1 -- 172.21.15.72:0/3015423577 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fdca0004300 con 0x7fdcac109800 2024-04-26T23:34:44.009 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.584+0000 7fdcabe00700 1 -- 172.21.15.72:0/3015423577 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdca0011af0 con 0x7fdcac109800 2024-04-26T23:34:44.009 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.584+0000 7fdcb4000700 1 -- 172.21.15.72:0/3015423577 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcac109800 msgr2=0x7fdcac10bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:44.009 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.585+0000 7fdcb4000700 1 --2- 172.21.15.72:0/3015423577 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcac109800 0x7fdcac10bbf0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdca00098a0 tx=0x7fdca0009bb0 comp rx=0 tx=0).stop 2024-04-26T23:34:44.009 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.585+0000 7fdcb4000700 1 -- 172.21.15.72:0/3015423577 shutdown_connections 2024-04-26T23:34:44.009 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.585+0000 7fdcb4000700 1 --2- 172.21.15.72:0/3015423577 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcac109800 0x7fdcac10bbf0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:44.009 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.585+0000 7fdcb4000700 1 -- 172.21.15.72:0/3015423577 >> 172.21.15.72:0/3015423577 conn(0x7fdcac1031f0 msgr2=0x7fdcac105670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:34:44.009 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.585+0000 7fdcb4000700 1 -- 172.21.15.72:0/3015423577 shutdown_connections 2024-04-26T23:34:44.009 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.585+0000 7fdcb4000700 1 -- 172.21.15.72:0/3015423577 wait complete. 2024-04-26T23:34:44.010 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.586+0000 7fdcb4000700 1 Processor -- start 2024-04-26T23:34:44.010 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.586+0000 7fdcb4000700 1 -- start start 2024-04-26T23:34:44.010 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.587+0000 7fdcb4000700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcac109800 0x7fdcac109080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:44.010 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.587+0000 7fdcb4000700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fdcac10d100 con 0x7fdcac109800 2024-04-26T23:34:44.010 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.587+0000 7fdcb1600700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcac109800 0x7fdcac109080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:44.010 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.587+0000 7fdcb1600700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcac109800 0x7fdcac109080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56330/0 (socket says 172.21.15.72:56330) 2024-04-26T23:34:44.010 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.587+0000 7fdcb1600700 1 -- 172.21.15.72:0/4074917589 learned_addr learned my addr 172.21.15.72:0/4074917589 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:34:44.010 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.587+0000 7fdcb1600700 1 -- 172.21.15.72:0/4074917589 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcac1095c0 con 0x7fdcac109800 2024-04-26T23:34:44.010 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.587+0000 7fdcb1600700 1 --2- 172.21.15.72:0/4074917589 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcac109800 0x7fdcac109080 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdca0011ac0 tx=0x7fdca0011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:44.010 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.588+0000 7fdcaa000700 1 -- 172.21.15.72:0/4074917589 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdca0003970 con 0x7fdcac109800 2024-04-26T23:34:44.010 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.588+0000 7fdcaa000700 1 -- 172.21.15.72:0/4074917589 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fdca0003b10 con 0x7fdcac109800 2024-04-26T23:34:44.010 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.588+0000 7fdcb4000700 1 -- 172.21.15.72:0/4074917589 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcac1077b0 con 0x7fdcac109800 2024-04-26T23:34:44.010 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.588+0000 7fdcaa000700 1 -- 172.21.15.72:0/4074917589 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdca0024610 con 0x7fdcac109800 2024-04-26T23:34:44.010 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.588+0000 7fdcb4000700 1 -- 172.21.15.72:0/4074917589 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcac107cf0 con 0x7fdcac109800 2024-04-26T23:34:44.011 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.589+0000 7fdcaa000700 1 -- 172.21.15.72:0/4074917589 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fdca001a430 con 0x7fdcac109800 2024-04-26T23:34:44.011 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.589+0000 7fdcb4000700 1 -- 172.21.15.72:0/4074917589 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcac064a80 con 0x7fdcac109800 2024-04-26T23:34:44.011 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.590+0000 7fdcaa000700 1 --2- 172.21.15.72:0/4074917589 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7fdc94039600 0x7fdc9403bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:44.011 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.590+0000 7fdcaa000700 1 -- 172.21.15.72:0/4074917589 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fdca004d8d0 con 0x7fdcac109800 2024-04-26T23:34:44.011 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.590+0000 7fdcb0c00700 1 --2- 172.21.15.72:0/4074917589 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7fdc94039600 0x7fdc9403bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:44.011 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.591+0000 7fdcb0c00700 1 --2- 172.21.15.72:0/4074917589 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7fdc94039600 0x7fdc9403bac0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fdc98006fd0 tx=0x7fdc98006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:44.011 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.595+0000 7fdcaa000700 1 -- 172.21.15.72:0/4074917589 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fdca0016260 con 0x7fdcac109800 2024-04-26T23:34:44.011 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.847+0000 7fdcb4000700 1 -- 172.21.15.72:0/4074917589 --> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fdcac104480 con 0x7fdc94039600 2024-04-26T23:34:44.011 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.856+0000 7fdcaa000700 1 -- 172.21.15.72:0/4074917589 <== mgr.14120 v2:172.21.15.72:6800/2903979134 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fdcac104480 con 0x7fdc94039600 2024-04-26T23:34:44.011 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.858+0000 7fdcb4000700 1 -- 172.21.15.72:0/4074917589 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7fdc94039600 msgr2=0x7fdc9403bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:44.011 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.859+0000 7fdcb4000700 1 --2- 172.21.15.72:0/4074917589 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7fdc94039600 0x7fdc9403bac0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fdc98006fd0 tx=0x7fdc98006e40 comp rx=0 tx=0).stop 2024-04-26T23:34:44.011 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.859+0000 7fdcb4000700 1 -- 172.21.15.72:0/4074917589 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcac109800 msgr2=0x7fdcac109080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:44.011 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.859+0000 7fdcb4000700 1 --2- 172.21.15.72:0/4074917589 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcac109800 0x7fdcac109080 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdca0011ac0 tx=0x7fdca0011ba0 comp rx=0 tx=0).stop 2024-04-26T23:34:44.011 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.859+0000 7fdcb4000700 1 -- 172.21.15.72:0/4074917589 shutdown_connections 2024-04-26T23:34:44.011 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.859+0000 7fdcb4000700 1 --2- 172.21.15.72:0/4074917589 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7fdc94039600 0x7fdc9403bac0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:44.012 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.859+0000 7fdcb4000700 1 --2- 172.21.15.72:0/4074917589 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcac109800 0x7fdcac109080 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:44.012 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.859+0000 7fdcb4000700 1 -- 172.21.15.72:0/4074917589 >> 172.21.15.72:0/4074917589 conn(0x7fdcac1031f0 msgr2=0x7fdcac103d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:34:44.012 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.859+0000 7fdcb4000700 1 -- 172.21.15.72:0/4074917589 shutdown_connections 2024-04-26T23:34:44.012 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:42.859+0000 7fdcb4000700 1 -- 172.21.15.72:0/4074917589 wait complete. 2024-04-26T23:34:45.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:45 smithi072 ceph-mon[28184]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:34:45.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:45 smithi072 ceph-mon[28184]: from='mgr.14120 172.21.15.72:0/3482651254' entity='mgr.x' 2024-04-26T23:34:45.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:45 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/1234384402' entity='client.admin' 2024-04-26T23:34:45.564 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.632+0000 7fb51aa00700 1 Processor -- start 2024-04-26T23:34:45.565 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.632+0000 7fb51aa00700 1 -- start start 2024-04-26T23:34:45.565 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.633+0000 7fb51aa00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb514108d70 0x7fb514109190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:45.565 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.633+0000 7fb51aa00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fb514109760 con 0x7fb514108d70 2024-04-26T23:34:45.565 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.633+0000 7fb513e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb514108d70 0x7fb514109190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:45.565 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.633+0000 7fb513e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb514108d70 0x7fb514109190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56338/0 (socket says 172.21.15.72:56338) 2024-04-26T23:34:45.565 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.633+0000 7fb513e00700 1 -- 172.21.15.72:0/3997132290 learned_addr learned my addr 172.21.15.72:0/3997132290 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:34:45.570 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.633+0000 7fb513e00700 1 -- 172.21.15.72:0/3997132290 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb514109f70 con 0x7fb514108d70 2024-04-26T23:34:45.570 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.634+0000 7fb513e00700 1 --2- 172.21.15.72:0/3997132290 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb514108d70 0x7fb514109190 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb5040098a0 tx=0x7fb504009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c416decae2a15fa9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:45.570 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.634+0000 7fb512a00700 1 -- 172.21.15.72:0/3997132290 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb504004160 con 0x7fb514108d70 2024-04-26T23:34:45.570 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.634+0000 7fb512a00700 1 -- 172.21.15.72:0/3997132290 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb504004300 con 0x7fb514108d70 2024-04-26T23:34:45.570 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.634+0000 7fb512a00700 1 -- 172.21.15.72:0/3997132290 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb504011a20 con 0x7fb514108d70 2024-04-26T23:34:45.570 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.635+0000 7fb51aa00700 1 -- 172.21.15.72:0/3997132290 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb514108d70 msgr2=0x7fb514109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:45.570 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.635+0000 7fb51aa00700 1 --2- 172.21.15.72:0/3997132290 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb514108d70 0x7fb514109190 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb5040098a0 tx=0x7fb504009bb0 comp rx=0 tx=0).stop 2024-04-26T23:34:45.570 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.635+0000 7fb51aa00700 1 -- 172.21.15.72:0/3997132290 shutdown_connections 2024-04-26T23:34:45.570 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.635+0000 7fb51aa00700 1 --2- 172.21.15.72:0/3997132290 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb514108d70 0x7fb514109190 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:45.571 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.635+0000 7fb51aa00700 1 -- 172.21.15.72:0/3997132290 >> 172.21.15.72:0/3997132290 conn(0x7fb5141042d0 msgr2=0x7fb514106750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:34:45.571 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.635+0000 7fb51aa00700 1 -- 172.21.15.72:0/3997132290 shutdown_connections 2024-04-26T23:34:45.571 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.635+0000 7fb51aa00700 1 -- 172.21.15.72:0/3997132290 wait complete. 2024-04-26T23:34:45.571 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.637+0000 7fb51aa00700 1 Processor -- start 2024-04-26T23:34:45.571 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.637+0000 7fb51aa00700 1 -- start start 2024-04-26T23:34:45.571 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.637+0000 7fb51aa00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb514108d70 0x7fb51407eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:45.571 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.637+0000 7fb51aa00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fb51410d320 con 0x7fb514108d70 2024-04-26T23:34:45.571 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.638+0000 7fb513e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb514108d70 0x7fb51407eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:45.571 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.638+0000 7fb513e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb514108d70 0x7fb51407eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56348/0 (socket says 172.21.15.72:56348) 2024-04-26T23:34:45.571 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.638+0000 7fb513e00700 1 -- 172.21.15.72:0/1234384402 learned_addr learned my addr 172.21.15.72:0/1234384402 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:34:45.571 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.638+0000 7fb513e00700 1 -- 172.21.15.72:0/1234384402 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb51407f0a0 con 0x7fb514108d70 2024-04-26T23:34:45.571 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.638+0000 7fb513e00700 1 --2- 172.21.15.72:0/1234384402 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb514108d70 0x7fb51407eb60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb504000c00 tx=0x7fb504011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:45.571 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.639+0000 7fb510c00700 1 -- 172.21.15.72:0/1234384402 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb504003aa0 con 0x7fb514108d70 2024-04-26T23:34:45.571 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.639+0000 7fb510c00700 1 -- 172.21.15.72:0/1234384402 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb50401b8f0 con 0x7fb514108d70 2024-04-26T23:34:45.572 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.639+0000 7fb51aa00700 1 -- 172.21.15.72:0/1234384402 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb51407d290 con 0x7fb514108d70 2024-04-26T23:34:45.572 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.639+0000 7fb510c00700 1 -- 172.21.15.72:0/1234384402 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb504023890 con 0x7fb514108d70 2024-04-26T23:34:45.572 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.639+0000 7fb51aa00700 1 -- 172.21.15.72:0/1234384402 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb51407d7d0 con 0x7fb514108d70 2024-04-26T23:34:45.572 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.640+0000 7fb51aa00700 1 -- 172.21.15.72:0/1234384402 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb514064a80 con 0x7fb514108d70 2024-04-26T23:34:45.572 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.640+0000 7fb510c00700 1 -- 172.21.15.72:0/1234384402 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fb50401b440 con 0x7fb514108d70 2024-04-26T23:34:45.572 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.641+0000 7fb510c00700 1 --2- 172.21.15.72:0/1234384402 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7fb4f8039600 0x7fb4f803bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:45.572 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.641+0000 7fb510c00700 1 -- 172.21.15.72:0/1234384402 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fb50404ddf0 con 0x7fb514108d70 2024-04-26T23:34:45.572 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.641+0000 7fb513400700 1 --2- 172.21.15.72:0/1234384402 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7fb4f8039600 0x7fb4f803bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:45.572 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.642+0000 7fb513400700 1 --2- 172.21.15.72:0/1234384402 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7fb4f8039600 0x7fb4f803bac0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb500006fd0 tx=0x7fb500006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:45.572 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.647+0000 7fb510c00700 1 -- 172.21.15.72:0/1234384402 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb50401f030 con 0x7fb514108d70 2024-04-26T23:34:45.572 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.878+0000 7fb51aa00700 1 -- 172.21.15.72:0/1234384402 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7fb51407df70 con 0x7fb514108d70 2024-04-26T23:34:45.572 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.885+0000 7fb510c00700 1 -- 172.21.15.72:0/1234384402 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7fb5040202b0 con 0x7fb514108d70 2024-04-26T23:34:45.572 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.888+0000 7fb51aa00700 1 -- 172.21.15.72:0/1234384402 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7fb4f8039600 msgr2=0x7fb4f803bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:45.572 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.888+0000 7fb51aa00700 1 --2- 172.21.15.72:0/1234384402 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7fb4f8039600 0x7fb4f803bac0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb500006fd0 tx=0x7fb500006e40 comp rx=0 tx=0).stop 2024-04-26T23:34:45.572 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.888+0000 7fb51aa00700 1 -- 172.21.15.72:0/1234384402 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb514108d70 msgr2=0x7fb51407eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:45.572 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.888+0000 7fb51aa00700 1 --2- 172.21.15.72:0/1234384402 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb514108d70 0x7fb51407eb60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb504000c00 tx=0x7fb504011aa0 comp rx=0 tx=0).stop 2024-04-26T23:34:45.573 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.888+0000 7fb51aa00700 1 -- 172.21.15.72:0/1234384402 shutdown_connections 2024-04-26T23:34:45.573 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.888+0000 7fb51aa00700 1 --2- 172.21.15.72:0/1234384402 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7fb4f8039600 0x7fb4f803bac0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:45.573 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.888+0000 7fb51aa00700 1 --2- 172.21.15.72:0/1234384402 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb514108d70 0x7fb51407eb60 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:45.573 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.888+0000 7fb51aa00700 1 -- 172.21.15.72:0/1234384402 >> 172.21.15.72:0/1234384402 conn(0x7fb5141042d0 msgr2=0x7fb514104dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:34:45.573 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.888+0000 7fb51aa00700 1 -- 172.21.15.72:0/1234384402 shutdown_connections 2024-04-26T23:34:45.573 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:44.889+0000 7fb51aa00700 1 -- 172.21.15.72:0/1234384402 wait complete. 2024-04-26T23:34:47.496 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:47 smithi072 ceph-mon[28184]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:34:47.496 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:47 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/1128095610' entity='client.admin' 2024-04-26T23:34:47.508 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.139+0000 7f1894c00700 1 Processor -- start 2024-04-26T23:34:47.508 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.139+0000 7f1894c00700 1 -- start start 2024-04-26T23:34:47.508 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.140+0000 7f1894c00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1890108d50 0x7f1890109170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:47.508 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.140+0000 7f1894c00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f1890109740 con 0x7f1890108d50 2024-04-26T23:34:47.508 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.140+0000 7f188de00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1890108d50 0x7f1890109170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:47.508 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.141+0000 7f188de00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1890108d50 0x7f1890109170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56360/0 (socket says 172.21.15.72:56360) 2024-04-26T23:34:47.508 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.141+0000 7f188de00700 1 -- 172.21.15.72:0/3984207867 learned_addr learned my addr 172.21.15.72:0/3984207867 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:34:47.508 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.141+0000 7f188de00700 1 -- 172.21.15.72:0/3984207867 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1890109f50 con 0x7f1890108d50 2024-04-26T23:34:47.508 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.141+0000 7f188de00700 1 --2- 172.21.15.72:0/3984207867 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1890108d50 0x7f1890109170 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f18800098a0 tx=0x7f1880009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8536b7360193ebcc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:47.508 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.142+0000 7f188ca00700 1 -- 172.21.15.72:0/3984207867 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1880004160 con 0x7f1890108d50 2024-04-26T23:34:47.508 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.142+0000 7f188ca00700 1 -- 172.21.15.72:0/3984207867 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1880004300 con 0x7f1890108d50 2024-04-26T23:34:47.508 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.142+0000 7f188ca00700 1 -- 172.21.15.72:0/3984207867 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1880011a20 con 0x7f1890108d50 2024-04-26T23:34:47.508 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.142+0000 7f1894c00700 1 -- 172.21.15.72:0/3984207867 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1890108d50 msgr2=0x7f1890109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:47.508 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.142+0000 7f1894c00700 1 --2- 172.21.15.72:0/3984207867 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1890108d50 0x7f1890109170 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f18800098a0 tx=0x7f1880009bb0 comp rx=0 tx=0).stop 2024-04-26T23:34:47.509 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.142+0000 7f1894c00700 1 -- 172.21.15.72:0/3984207867 shutdown_connections 2024-04-26T23:34:47.509 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.142+0000 7f1894c00700 1 --2- 172.21.15.72:0/3984207867 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1890108d50 0x7f1890109170 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:47.509 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.142+0000 7f1894c00700 1 -- 172.21.15.72:0/3984207867 >> 172.21.15.72:0/3984207867 conn(0x7f18901042d0 msgr2=0x7f1890106730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:34:47.509 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.143+0000 7f1894c00700 1 -- 172.21.15.72:0/3984207867 shutdown_connections 2024-04-26T23:34:47.509 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.143+0000 7f1894c00700 1 -- 172.21.15.72:0/3984207867 wait complete. 2024-04-26T23:34:47.509 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.143+0000 7f1894c00700 1 Processor -- start 2024-04-26T23:34:47.509 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.144+0000 7f1894c00700 1 -- start start 2024-04-26T23:34:47.509 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.144+0000 7f1894c00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1890108d50 0x7f189019cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:47.509 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.144+0000 7f1894c00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f189010d300 con 0x7f1890108d50 2024-04-26T23:34:47.509 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.144+0000 7f188de00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1890108d50 0x7f189019cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:47.509 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.144+0000 7f188de00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1890108d50 0x7f189019cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56374/0 (socket says 172.21.15.72:56374) 2024-04-26T23:34:47.509 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.144+0000 7f188de00700 1 -- 172.21.15.72:0/1128095610 learned_addr learned my addr 172.21.15.72:0/1128095610 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:34:47.509 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.145+0000 7f188de00700 1 -- 172.21.15.72:0/1128095610 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f189019d1c0 con 0x7f1890108d50 2024-04-26T23:34:47.509 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.145+0000 7f188de00700 1 --2- 172.21.15.72:0/1128095610 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1890108d50 0x7f189019cc80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f18800119f0 tx=0x7f1880011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:47.509 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.145+0000 7f1886a00700 1 -- 172.21.15.72:0/1128095610 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f18800039e0 con 0x7f1890108d50 2024-04-26T23:34:47.509 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.145+0000 7f1886a00700 1 -- 172.21.15.72:0/1128095610 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f188001b8f0 con 0x7f1890108d50 2024-04-26T23:34:47.509 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.145+0000 7f1894c00700 1 -- 172.21.15.72:0/1128095610 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f189019d450 con 0x7f1890108d50 2024-04-26T23:34:47.510 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.145+0000 7f1886a00700 1 -- 172.21.15.72:0/1128095610 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f18800237a0 con 0x7f1890108d50 2024-04-26T23:34:47.510 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.145+0000 7f1894c00700 1 -- 172.21.15.72:0/1128095610 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18901a0140 con 0x7f1890108d50 2024-04-26T23:34:47.510 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.146+0000 7f1894c00700 1 -- 172.21.15.72:0/1128095610 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1890064a80 con 0x7f1890108d50 2024-04-26T23:34:47.510 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.146+0000 7f1886a00700 1 -- 172.21.15.72:0/1128095610 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f188001b440 con 0x7f1890108d50 2024-04-26T23:34:47.510 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.147+0000 7f1886a00700 1 --2- 172.21.15.72:0/1128095610 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7f18740395b0 0x7f187403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:47.510 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.147+0000 7f1886a00700 1 -- 172.21.15.72:0/1128095610 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f188004dea0 con 0x7f1890108d50 2024-04-26T23:34:47.510 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.147+0000 7f188d400700 1 --2- 172.21.15.72:0/1128095610 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7f18740395b0 0x7f187403ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:47.510 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.148+0000 7f188d400700 1 --2- 172.21.15.72:0/1128095610 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7f18740395b0 0x7f187403ba70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1878006fd0 tx=0x7f1878006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:47.510 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.152+0000 7f1886a00700 1 -- 172.21.15.72:0/1128095610 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f188001f030 con 0x7f1890108d50 2024-04-26T23:34:47.510 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.379+0000 7f1894c00700 1 -- 172.21.15.72:0/1128095610 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f18901a0760 con 0x7f1890108d50 2024-04-26T23:34:47.510 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.384+0000 7f1886a00700 1 -- 172.21.15.72:0/1128095610 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f1880023dd0 con 0x7f1890108d50 2024-04-26T23:34:47.510 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.387+0000 7f1894c00700 1 -- 172.21.15.72:0/1128095610 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7f18740395b0 msgr2=0x7f187403ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:47.510 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.387+0000 7f1894c00700 1 --2- 172.21.15.72:0/1128095610 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7f18740395b0 0x7f187403ba70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1878006fd0 tx=0x7f1878006e40 comp rx=0 tx=0).stop 2024-04-26T23:34:47.511 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.387+0000 7f1894c00700 1 -- 172.21.15.72:0/1128095610 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1890108d50 msgr2=0x7f189019cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:47.511 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.387+0000 7f1894c00700 1 --2- 172.21.15.72:0/1128095610 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1890108d50 0x7f189019cc80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f18800119f0 tx=0x7f1880011ad0 comp rx=0 tx=0).stop 2024-04-26T23:34:47.511 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.388+0000 7f1894c00700 1 -- 172.21.15.72:0/1128095610 shutdown_connections 2024-04-26T23:34:47.511 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.388+0000 7f1894c00700 1 --2- 172.21.15.72:0/1128095610 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7f18740395b0 0x7f187403ba70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:47.511 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.388+0000 7f1894c00700 1 --2- 172.21.15.72:0/1128095610 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1890108d50 0x7f189019cc80 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:47.511 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.388+0000 7f1894c00700 1 -- 172.21.15.72:0/1128095610 >> 172.21.15.72:0/1128095610 conn(0x7f18901042d0 msgr2=0x7f1890104e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:34:47.511 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.388+0000 7f1894c00700 1 -- 172.21.15.72:0/1128095610 shutdown_connections 2024-04-26T23:34:47.511 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:46.388+0000 7f1894c00700 1 -- 172.21.15.72:0/1128095610 wait complete. 2024-04-26T23:34:47.511 INFO:teuthology.orchestra.run.smithi072.stdout:Enabling the dashboard module... 2024-04-26T23:34:48.418 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:48 smithi072 ceph-mon[28184]: from='mgr.14120 172.21.15.72:0/3482651254' entity='mgr.x' 2024-04-26T23:34:49.661 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:49 smithi072 ceph-mon[28184]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:34:49.661 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:49 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/1683326374' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-26T23:34:49.661 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:49 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: ignoring --setuser ceph since I am not root 2024-04-26T23:34:49.661 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:49 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: ignoring --setgroup ceph since I am not root 2024-04-26T23:34:49.661 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:49 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:49.656+0000 7f6f5e971200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T23:34:50.039 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:49 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:49.904+0000 7f6f5e971200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T23:34:50.119 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:48.108+0000 7f60df200700 1 Processor -- start 2024-04-26T23:34:50.119 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:48.108+0000 7f60df200700 1 -- start start 2024-04-26T23:34:50.119 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:48.109+0000 7f60df200700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60d81075e0 0x7f60d8107a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:50.119 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:48.109+0000 7f60df200700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f60d807d730 con 0x7f60d81075e0 2024-04-26T23:34:50.119 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:48.109+0000 7f60d7e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60d81075e0 0x7f60d8107a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:50.119 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:48.109+0000 7f60d7e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60d81075e0 0x7f60d8107a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57936/0 (socket says 172.21.15.72:57936) 2024-04-26T23:34:50.119 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:48.109+0000 7f60d7e00700 1 -- 172.21.15.72:0/49176761 learned_addr learned my addr 172.21.15.72:0/49176761 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:34:50.120 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:48.110+0000 7f60d7e00700 1 -- 172.21.15.72:0/49176761 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60d8107f40 con 0x7f60d81075e0 2024-04-26T23:34:50.120 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:48.110+0000 7f60d7e00700 1 --2- 172.21.15.72:0/49176761 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60d81075e0 0x7f60d8107a00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f60c80098a0 tx=0x7f60c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5dbec0cfb60a0ac server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:50.120 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:48.110+0000 7f60d6a00700 1 -- 172.21.15.72:0/49176761 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f60c8004160 con 0x7f60d81075e0 2024-04-26T23:34:50.120 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:48.110+0000 7f60d6a00700 1 -- 172.21.15.72:0/49176761 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60c8004300 con 0x7f60d81075e0 2024-04-26T23:34:50.120 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:48.110+0000 7f60d6a00700 1 -- 172.21.15.72:0/49176761 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f60c8011a20 con 0x7f60d81075e0 2024-04-26T23:34:50.120 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:48.111+0000 7f60df200700 1 -- 172.21.15.72:0/49176761 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60d81075e0 msgr2=0x7f60d8107a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:50.120 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:48.111+0000 7f60df200700 1 --2- 172.21.15.72:0/49176761 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60d81075e0 0x7f60d8107a00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f60c80098a0 tx=0x7f60c8009bb0 comp rx=0 tx=0).stop 2024-04-26T23:34:50.120 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:48.111+0000 7f60df200700 1 -- 172.21.15.72:0/49176761 shutdown_connections 2024-04-26T23:34:50.120 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:48.111+0000 7f60df200700 1 --2- 172.21.15.72:0/49176761 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60d81075e0 0x7f60d8107a00 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:50.120 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:48.111+0000 7f60df200700 1 -- 172.21.15.72:0/49176761 >> 172.21.15.72:0/49176761 conn(0x7f60d8103220 msgr2=0x7f60d8105660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:34:50.120 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:48.111+0000 7f60df200700 1 -- 172.21.15.72:0/49176761 shutdown_connections 2024-04-26T23:34:50.120 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:48.111+0000 7f60df200700 1 -- 172.21.15.72:0/49176761 wait complete. 2024-04-26T23:34:50.120 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:48.112+0000 7f60df200700 1 Processor -- start 2024-04-26T23:34:50.120 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:48.112+0000 7f60df200700 1 -- start start 2024-04-26T23:34:50.120 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:48.112+0000 7f60df200700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60d81075e0 0x7f60d819eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:50.120 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:48.112+0000 7f60df200700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f60d8108fb0 con 0x7f60d81075e0 2024-04-26T23:34:50.121 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:48.113+0000 7f60d7e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60d81075e0 0x7f60d819eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:50.121 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:48.113+0000 7f60d7e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60d81075e0 0x7f60d819eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57948/0 (socket says 172.21.15.72:57948) 2024-04-26T23:34:50.121 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:48.113+0000 7f60d7e00700 1 -- 172.21.15.72:0/1683326374 learned_addr learned my addr 172.21.15.72:0/1683326374 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:34:50.121 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:48.113+0000 7f60d7e00700 1 -- 172.21.15.72:0/1683326374 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60d819f3e0 con 0x7f60d81075e0 2024-04-26T23:34:50.121 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:48.113+0000 7f60d7e00700 1 --2- 172.21.15.72:0/1683326374 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60d81075e0 0x7f60d819eea0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f60c8004660 tx=0x7f60c8011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:50.121 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:48.113+0000 7f60d4c00700 1 -- 172.21.15.72:0/1683326374 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f60c8003aa0 con 0x7f60d81075e0 2024-04-26T23:34:50.121 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:48.113+0000 7f60d4c00700 1 -- 172.21.15.72:0/1683326374 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60c801b8f0 con 0x7f60d81075e0 2024-04-26T23:34:50.121 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:48.113+0000 7f60df200700 1 -- 172.21.15.72:0/1683326374 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60d819f670 con 0x7f60d81075e0 2024-04-26T23:34:50.121 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:48.113+0000 7f60d4c00700 1 -- 172.21.15.72:0/1683326374 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f60c8023900 con 0x7f60d81075e0 2024-04-26T23:34:50.121 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:48.114+0000 7f60df200700 1 -- 172.21.15.72:0/1683326374 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60d81a2360 con 0x7f60d81075e0 2024-04-26T23:34:50.121 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:48.115+0000 7f60df200700 1 -- 172.21.15.72:0/1683326374 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60d8064a80 con 0x7f60d81075e0 2024-04-26T23:34:50.121 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:48.115+0000 7f60d4c00700 1 -- 172.21.15.72:0/1683326374 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f60c8023b70 con 0x7f60d81075e0 2024-04-26T23:34:50.121 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:48.116+0000 7f60d4c00700 1 --2- 172.21.15.72:0/1683326374 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7f60bc039600 0x7f60bc03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:50.121 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:48.116+0000 7f60d4c00700 1 -- 172.21.15.72:0/1683326374 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f60c804de90 con 0x7f60d81075e0 2024-04-26T23:34:50.121 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:48.116+0000 7f60d7400700 1 --2- 172.21.15.72:0/1683326374 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7f60bc039600 0x7f60bc03bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:50.122 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:48.116+0000 7f60d7400700 1 --2- 172.21.15.72:0/1683326374 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7f60bc039600 0x7f60bc03bac0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f60c4006fd0 tx=0x7f60c4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:50.122 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:48.121+0000 7f60d4c00700 1 -- 172.21.15.72:0/1683326374 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f60c801f030 con 0x7f60d81075e0 2024-04-26T23:34:50.122 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:48.413+0000 7f60df200700 1 -- 172.21.15.72:0/1683326374 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f60d802e700 con 0x7f60d81075e0 2024-04-26T23:34:50.122 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:49.347+0000 7f60d4c00700 1 -- 172.21.15.72:0/1683326374 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f60c80292a0 con 0x7f60d81075e0 2024-04-26T23:34:50.122 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:49.350+0000 7f60d4c00700 1 -- 172.21.15.72:0/1683326374 <== mon.0 v2:172.21.15.72:3300/0 8 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f60c802c3f0 con 0x7f60d81075e0 2024-04-26T23:34:50.122 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:49.352+0000 7f60df200700 1 -- 172.21.15.72:0/1683326374 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7f60bc039600 msgr2=0x7f60bc03bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:50.122 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:49.352+0000 7f60df200700 1 --2- 172.21.15.72:0/1683326374 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7f60bc039600 0x7f60bc03bac0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f60c4006fd0 tx=0x7f60c4006e40 comp rx=0 tx=0).stop 2024-04-26T23:34:50.122 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:49.352+0000 7f60df200700 1 -- 172.21.15.72:0/1683326374 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60d81075e0 msgr2=0x7f60d819eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:50.122 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:49.352+0000 7f60df200700 1 --2- 172.21.15.72:0/1683326374 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60d81075e0 0x7f60d819eea0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f60c8004660 tx=0x7f60c8011aa0 comp rx=0 tx=0).stop 2024-04-26T23:34:50.122 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:49.352+0000 7f60df200700 1 -- 172.21.15.72:0/1683326374 shutdown_connections 2024-04-26T23:34:50.122 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:49.352+0000 7f60df200700 1 --2- 172.21.15.72:0/1683326374 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7f60bc039600 0x7f60bc03bac0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:50.122 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:49.352+0000 7f60df200700 1 --2- 172.21.15.72:0/1683326374 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60d81075e0 0x7f60d819eea0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:50.122 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:49.352+0000 7f60df200700 1 -- 172.21.15.72:0/1683326374 >> 172.21.15.72:0/1683326374 conn(0x7f60d8103220 msgr2=0x7f60d8195aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:34:50.123 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:49.353+0000 7f60df200700 1 -- 172.21.15.72:0/1683326374 shutdown_connections 2024-04-26T23:34:50.123 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:49.353+0000 7f60df200700 1 -- 172.21.15.72:0/1683326374 wait complete. 2024-04-26T23:34:50.293 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:50 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:50.034+0000 7f6f5e971200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T23:34:50.602 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:50 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:50.442+0000 7f6f5e971200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T23:34:50.602 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:50 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:50.564+0000 7f6f5e971200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T23:34:50.603 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:50 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/1683326374' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-26T23:34:50.603 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:50 smithi072 ceph-mon[28184]: mgrmap e9: x(active, since 25s) 2024-04-26T23:34:51.677 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:51 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/450397849' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-26T23:34:51.685 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout { 2024-04-26T23:34:51.692 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-26T23:34:51.692 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-26T23:34:51.692 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-26T23:34:51.692 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-26T23:34:51.692 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout } 2024-04-26T23:34:51.692 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:50.754+0000 7fc921e00700 1 Processor -- start 2024-04-26T23:34:51.692 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:50.754+0000 7fc921e00700 1 -- start start 2024-04-26T23:34:51.692 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:50.754+0000 7fc921e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc91c07ee70 0x7fc91c07d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:51.692 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:50.754+0000 7fc921e00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fc91c07d8a0 con 0x7fc91c07ee70 2024-04-26T23:34:51.692 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:50.755+0000 7fc920a00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc91c07ee70 0x7fc91c07d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:51.693 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:50.755+0000 7fc920a00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc91c07ee70 0x7fc91c07d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57970/0 (socket says 172.21.15.72:57970) 2024-04-26T23:34:51.693 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:50.755+0000 7fc920a00700 1 -- 172.21.15.72:0/1970207579 learned_addr learned my addr 172.21.15.72:0/1970207579 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:34:51.693 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:50.755+0000 7fc920a00700 1 -- 172.21.15.72:0/1970207579 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc91c07da20 con 0x7fc91c07ee70 2024-04-26T23:34:51.693 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:50.756+0000 7fc920a00700 1 --2- 172.21.15.72:0/1970207579 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc91c07ee70 0x7fc91c07d2d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc9100098a0 tx=0x7fc910009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c4b690fa491a86e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:51.693 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:50.756+0000 7fc91b400700 1 -- 172.21.15.72:0/1970207579 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc910004160 con 0x7fc91c07ee70 2024-04-26T23:34:51.693 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:50.756+0000 7fc91b400700 1 -- 172.21.15.72:0/1970207579 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc910004300 con 0x7fc91c07ee70 2024-04-26T23:34:51.693 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:50.756+0000 7fc91b400700 1 -- 172.21.15.72:0/1970207579 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc910011a20 con 0x7fc91c07ee70 2024-04-26T23:34:51.693 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:50.757+0000 7fc921e00700 1 -- 172.21.15.72:0/1970207579 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc91c07ee70 msgr2=0x7fc91c07d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:51.693 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:50.757+0000 7fc921e00700 1 --2- 172.21.15.72:0/1970207579 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc91c07ee70 0x7fc91c07d2d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc9100098a0 tx=0x7fc910009bb0 comp rx=0 tx=0).stop 2024-04-26T23:34:51.694 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:50.757+0000 7fc921e00700 1 -- 172.21.15.72:0/1970207579 shutdown_connections 2024-04-26T23:34:51.694 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:50.757+0000 7fc921e00700 1 --2- 172.21.15.72:0/1970207579 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc91c07ee70 0x7fc91c07d2d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:51.694 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:50.757+0000 7fc921e00700 1 -- 172.21.15.72:0/1970207579 >> 172.21.15.72:0/1970207579 conn(0x7fc91c1041d0 msgr2=0x7fc91c1065f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:34:51.694 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:50.757+0000 7fc921e00700 1 -- 172.21.15.72:0/1970207579 shutdown_connections 2024-04-26T23:34:51.694 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:50.757+0000 7fc921e00700 1 -- 172.21.15.72:0/1970207579 wait complete. 2024-04-26T23:34:51.694 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:50.758+0000 7fc921e00700 1 Processor -- start 2024-04-26T23:34:51.694 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:50.758+0000 7fc921e00700 1 -- start start 2024-04-26T23:34:51.694 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:50.759+0000 7fc921e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc91c07ee70 0x7fc91c19ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:51.694 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:50.759+0000 7fc921e00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fc91c10f1d0 con 0x7fc91c07ee70 2024-04-26T23:34:51.694 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:50.759+0000 7fc920a00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc91c07ee70 0x7fc91c19ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:34:51.694 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:50.759+0000 7fc920a00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc91c07ee70 0x7fc91c19ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57986/0 (socket says 172.21.15.72:57986) 2024-04-26T23:34:51.694 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:50.759+0000 7fc920a00700 1 -- 172.21.15.72:0/450397849 learned_addr learned my addr 172.21.15.72:0/450397849 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:34:51.694 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:50.759+0000 7fc920a00700 1 -- 172.21.15.72:0/450397849 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc91c19f160 con 0x7fc91c07ee70 2024-04-26T23:34:51.695 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:50.760+0000 7fc920a00700 1 --2- 172.21.15.72:0/450397849 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc91c07ee70 0x7fc91c19ec20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc910000c00 tx=0x7fc910011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:34:51.695 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:50.760+0000 7fc919600700 1 -- 172.21.15.72:0/450397849 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc910003aa0 con 0x7fc91c07ee70 2024-04-26T23:34:51.695 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:50.760+0000 7fc919600700 1 -- 172.21.15.72:0/450397849 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc91001b8f0 con 0x7fc91c07ee70 2024-04-26T23:34:51.695 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:50.760+0000 7fc921e00700 1 -- 172.21.15.72:0/450397849 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc91c19f3f0 con 0x7fc91c07ee70 2024-04-26T23:34:51.695 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:50.760+0000 7fc919600700 1 -- 172.21.15.72:0/450397849 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc910023870 con 0x7fc91c07ee70 2024-04-26T23:34:51.695 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:50.760+0000 7fc921e00700 1 -- 172.21.15.72:0/450397849 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc91c1a20e0 con 0x7fc91c07ee70 2024-04-26T23:34:51.695 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:50.761+0000 7fc921e00700 1 -- 172.21.15.72:0/450397849 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc91c06a540 con 0x7fc91c07ee70 2024-04-26T23:34:51.695 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:50.761+0000 7fc919600700 1 -- 172.21.15.72:0/450397849 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7fc910023b70 con 0x7fc91c07ee70 2024-04-26T23:34:51.695 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:50.762+0000 7fc919600700 1 --2- 172.21.15.72:0/450397849 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7fc904039600 0x7fc90403bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:34:51.695 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:50.762+0000 7fc919600700 1 -- 172.21.15.72:0/450397849 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc91004df10 con 0x7fc91c07ee70 2024-04-26T23:34:51.695 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:50.762+0000 7fc91be00700 1 -- 172.21.15.72:0/450397849 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7fc904039600 msgr2=0x7fc90403bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/2903979134 2024-04-26T23:34:51.695 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:50.762+0000 7fc91be00700 1 --2- 172.21.15.72:0/450397849 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7fc904039600 0x7fc90403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T23:34:51.695 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:50.767+0000 7fc919600700 1 -- 172.21.15.72:0/450397849 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc91001f030 con 0x7fc91c07ee70 2024-04-26T23:34:51.695 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:50.963+0000 7fc91be00700 1 -- 172.21.15.72:0/450397849 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7fc904039600 msgr2=0x7fc90403bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/2903979134 2024-04-26T23:34:51.695 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:50.963+0000 7fc91be00700 1 --2- 172.21.15.72:0/450397849 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7fc904039600 0x7fc90403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T23:34:51.696 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:51.057+0000 7fc921e00700 1 -- 172.21.15.72:0/450397849 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fc91c1a26d0 con 0x7fc91c07ee70 2024-04-26T23:34:51.696 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:51.057+0000 7fc919600700 1 -- 172.21.15.72:0/450397849 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7fc9100292a0 con 0x7fc91c07ee70 2024-04-26T23:34:51.696 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:51.060+0000 7fc921e00700 1 -- 172.21.15.72:0/450397849 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7fc904039600 msgr2=0x7fc90403bac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T23:34:51.696 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:51.060+0000 7fc921e00700 1 --2- 172.21.15.72:0/450397849 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7fc904039600 0x7fc90403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:51.696 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:51.060+0000 7fc921e00700 1 -- 172.21.15.72:0/450397849 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc91c07ee70 msgr2=0x7fc91c19ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:34:51.696 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:51.060+0000 7fc921e00700 1 --2- 172.21.15.72:0/450397849 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc91c07ee70 0x7fc91c19ec20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc910000c00 tx=0x7fc910011aa0 comp rx=0 tx=0).stop 2024-04-26T23:34:51.696 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:51.060+0000 7fc921e00700 1 -- 172.21.15.72:0/450397849 shutdown_connections 2024-04-26T23:34:51.696 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:51.060+0000 7fc921e00700 1 --2- 172.21.15.72:0/450397849 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7fc904039600 0x7fc90403bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:51.696 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:51.060+0000 7fc921e00700 1 --2- 172.21.15.72:0/450397849 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc91c07ee70 0x7fc91c19ec20 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:34:51.696 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:51.060+0000 7fc921e00700 1 -- 172.21.15.72:0/450397849 >> 172.21.15.72:0/450397849 conn(0x7fc91c1041d0 msgr2=0x7fc91c104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:34:51.696 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:51.061+0000 7fc921e00700 1 -- 172.21.15.72:0/450397849 shutdown_connections 2024-04-26T23:34:51.696 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:51.061+0000 7fc921e00700 1 -- 172.21.15.72:0/450397849 wait complete. 2024-04-26T23:34:51.696 INFO:teuthology.orchestra.run.smithi072.stdout:Waiting for the mgr to restart... 2024-04-26T23:34:51.696 INFO:teuthology.orchestra.run.smithi072.stdout:Waiting for mgr epoch 9... 2024-04-26T23:34:52.292 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:52 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:52.176+0000 7f6f5e971200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T23:34:52.542 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:52 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:52.414+0000 7f6f5e971200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T23:34:52.962 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:52 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:52.633+0000 7f6f5e971200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T23:34:53.293 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:53 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:53.010+0000 7f6f5e971200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T23:34:53.293 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:53 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:53.130+0000 7f6f5e971200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T23:34:53.762 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:53 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:53.378+0000 7f6f5e971200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T23:34:53.763 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:53 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:53.627+0000 7f6f5e971200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T23:34:54.793 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:54 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:54.404+0000 7f6f5e971200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T23:34:55.292 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:54 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:54.947+0000 7f6f5e971200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T23:34:55.293 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:55 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:55.073+0000 7f6f5e971200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T23:34:55.793 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:55 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:55.334+0000 7f6f5e971200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T23:34:56.293 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:55 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:55.975+0000 7f6f5e971200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T23:34:56.293 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:56 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:56.097+0000 7f6f5e971200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T23:34:56.723 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:56 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:56.416+0000 7f6f5e971200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T23:34:56.723 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:56 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:56.593+0000 7f6f5e971200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T23:34:56.723 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:56 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:56.719+0000 7f6f5e971200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T23:34:57.043 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:56 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:56.868+0000 7f6f5e971200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T23:34:57.542 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:57 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:57.231+0000 7f6f5e971200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T23:34:58.266 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:57 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:57.981+0000 7f6f5e971200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T23:34:58.542 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:34:58 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:34:58.262+0000 7f6f5e971200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T23:34:58.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:58 smithi072 ceph-mon[28184]: Active manager daemon x restarted 2024-04-26T23:34:58.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:58 smithi072 ceph-mon[28184]: Activating manager daemon x 2024-04-26T23:34:58.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:58 smithi072 ceph-mon[28184]: osdmap e3: 0 total, 0 up, 0 in 2024-04-26T23:34:58.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:58 smithi072 ceph-mon[28184]: mgrmap e10: x(active, starting, since 0.00525937s) 2024-04-26T23:34:58.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:58 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T23:34:58.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:58 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T23:34:58.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:58 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T23:34:58.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:58 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T23:34:58.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:58 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T23:34:58.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:58 smithi072 ceph-mon[28184]: Manager daemon x is now available 2024-04-26T23:34:58.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:58 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:34:59.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:59 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T23:34:59.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:59 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T23:34:59.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:59 smithi072 ceph-mon[28184]: [26/Apr/2024:23:34:58] ENGINE Bus STARTING 2024-04-26T23:34:59.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:34:59 smithi072 ceph-mon[28184]: mgrmap e11: x(active, since 1.01018s) 2024-04-26T23:35:00.202 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout { 2024-04-26T23:35:00.202 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-26T23:35:00.202 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-26T23:35:00.202 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout } 2024-04-26T23:35:00.202 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:52.345+0000 7f7163400700 1 Processor -- start 2024-04-26T23:35:00.202 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:52.346+0000 7f7163400700 1 -- start start 2024-04-26T23:35:00.202 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:52.346+0000 7f7163400700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f715c07eed0 0x7f715c07d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:35:00.203 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:52.346+0000 7f7163400700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f715c07d900 con 0x7f715c07eed0 2024-04-26T23:35:00.203 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:52.346+0000 7f7160a00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f715c07eed0 0x7f715c07d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:35:00.203 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:52.346+0000 7f7160a00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f715c07eed0 0x7f715c07d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57992/0 (socket says 172.21.15.72:57992) 2024-04-26T23:35:00.203 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:52.346+0000 7f7160a00700 1 -- 172.21.15.72:0/2677011032 learned_addr learned my addr 172.21.15.72:0/2677011032 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:35:00.203 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:52.347+0000 7f7160a00700 1 -- 172.21.15.72:0/2677011032 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f715c07da80 con 0x7f715c07eed0 2024-04-26T23:35:00.203 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:52.347+0000 7f7160a00700 1 --2- 172.21.15.72:0/2677011032 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f715c07eed0 0x7f715c07d330 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f71500098a0 tx=0x7f7150009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38e6f7992377dc8a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:35:00.203 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:52.348+0000 7f715b400700 1 -- 172.21.15.72:0/2677011032 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7150004160 con 0x7f715c07eed0 2024-04-26T23:35:00.203 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:52.348+0000 7f715b400700 1 -- 172.21.15.72:0/2677011032 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7150004300 con 0x7f715c07eed0 2024-04-26T23:35:00.203 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:52.348+0000 7f715b400700 1 -- 172.21.15.72:0/2677011032 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7150011a20 con 0x7f715c07eed0 2024-04-26T23:35:00.203 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:52.349+0000 7f7163400700 1 -- 172.21.15.72:0/2677011032 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f715c07eed0 msgr2=0x7f715c07d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:35:00.203 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:52.349+0000 7f7163400700 1 --2- 172.21.15.72:0/2677011032 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f715c07eed0 0x7f715c07d330 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f71500098a0 tx=0x7f7150009bb0 comp rx=0 tx=0).stop 2024-04-26T23:35:00.203 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:52.349+0000 7f7163400700 1 -- 172.21.15.72:0/2677011032 shutdown_connections 2024-04-26T23:35:00.203 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:52.349+0000 7f7163400700 1 --2- 172.21.15.72:0/2677011032 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f715c07eed0 0x7f715c07d330 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:35:00.203 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:52.349+0000 7f7163400700 1 -- 172.21.15.72:0/2677011032 >> 172.21.15.72:0/2677011032 conn(0x7f715c1042d0 msgr2=0x7f715c106750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:35:00.204 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:52.349+0000 7f7163400700 1 -- 172.21.15.72:0/2677011032 shutdown_connections 2024-04-26T23:35:00.204 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:52.349+0000 7f7163400700 1 -- 172.21.15.72:0/2677011032 wait complete. 2024-04-26T23:35:00.204 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:52.351+0000 7f7163400700 1 Processor -- start 2024-04-26T23:35:00.204 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:52.351+0000 7f7163400700 1 -- start start 2024-04-26T23:35:00.204 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:52.351+0000 7f7163400700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f715c07eed0 0x7f715c1a32b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:35:00.204 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:52.351+0000 7f7163400700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f715c10af70 con 0x7f715c07eed0 2024-04-26T23:35:00.204 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:52.351+0000 7f7160a00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f715c07eed0 0x7f715c1a32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:35:00.204 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:52.351+0000 7f7160a00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f715c07eed0 0x7f715c1a32b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57994/0 (socket says 172.21.15.72:57994) 2024-04-26T23:35:00.204 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:52.351+0000 7f7160a00700 1 -- 172.21.15.72:0/3909250195 learned_addr learned my addr 172.21.15.72:0/3909250195 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:35:00.204 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:52.352+0000 7f7160a00700 1 -- 172.21.15.72:0/3909250195 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f715c1a37f0 con 0x7f715c07eed0 2024-04-26T23:35:00.204 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:52.352+0000 7f7160a00700 1 --2- 172.21.15.72:0/3909250195 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f715c07eed0 0x7f715c1a32b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7150009520 tx=0x7f7150011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:35:00.204 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:52.352+0000 7f7159600700 1 -- 172.21.15.72:0/3909250195 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f71500038b0 con 0x7f715c07eed0 2024-04-26T23:35:00.204 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:52.352+0000 7f7159600700 1 -- 172.21.15.72:0/3909250195 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7150003a50 con 0x7f715c07eed0 2024-04-26T23:35:00.204 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:52.353+0000 7f7159600700 1 -- 172.21.15.72:0/3909250195 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7150019690 con 0x7f715c07eed0 2024-04-26T23:35:00.205 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:52.353+0000 7f7163400700 1 -- 172.21.15.72:0/3909250195 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f715c1a3a80 con 0x7f715c07eed0 2024-04-26T23:35:00.205 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:52.353+0000 7f7163400700 1 -- 172.21.15.72:0/3909250195 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f715c1a6770 con 0x7f715c07eed0 2024-04-26T23:35:00.205 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:52.354+0000 7f7159600700 1 -- 172.21.15.72:0/3909250195 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f715001aa90 con 0x7f715c07eed0 2024-04-26T23:35:00.205 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:52.355+0000 7f7159600700 1 --2- 172.21.15.72:0/3909250195 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7f7144039650 0x7f714403bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:35:00.205 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:52.355+0000 7f7159600700 1 -- 172.21.15.72:0/3909250195 --> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f714403c220 con 0x7f7144039650 2024-04-26T23:35:00.205 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:52.355+0000 7f7159600700 1 -- 172.21.15.72:0/3909250195 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f715004dce0 con 0x7f715c07eed0 2024-04-26T23:35:00.205 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:52.355+0000 7f715be00700 1 -- 172.21.15.72:0/3909250195 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7f7144039650 msgr2=0x7f714403bb10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/2903979134 2024-04-26T23:35:00.205 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:52.355+0000 7f715be00700 1 --2- 172.21.15.72:0/3909250195 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7f7144039650 0x7f714403bb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T23:35:00.205 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:52.556+0000 7f715be00700 1 -- 172.21.15.72:0/3909250195 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7f7144039650 msgr2=0x7f714403bb10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/2903979134 2024-04-26T23:35:00.205 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:52.556+0000 7f715be00700 1 --2- 172.21.15.72:0/3909250195 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7f7144039650 0x7f714403bb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T23:35:00.205 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:52.957+0000 7f715be00700 1 -- 172.21.15.72:0/3909250195 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7f7144039650 msgr2=0x7f714403bb10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/2903979134 2024-04-26T23:35:00.205 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:52.957+0000 7f715be00700 1 --2- 172.21.15.72:0/3909250195 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7f7144039650 0x7f714403bb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T23:35:00.205 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:53.758+0000 7f715be00700 1 -- 172.21.15.72:0/3909250195 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7f7144039650 msgr2=0x7f714403bb10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/2903979134 2024-04-26T23:35:00.205 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:53.758+0000 7f715be00700 1 --2- 172.21.15.72:0/3909250195 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7f7144039650 0x7f714403bb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T23:35:00.206 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:55.360+0000 7f715be00700 1 -- 172.21.15.72:0/3909250195 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7f7144039650 msgr2=0x7f714403bb10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/2903979134 2024-04-26T23:35:00.206 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:55.360+0000 7f715be00700 1 --2- 172.21.15.72:0/3909250195 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7f7144039650 0x7f714403bb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-26T23:35:00.206 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:58.270+0000 7f7159600700 1 -- 172.21.15.72:0/3909250195 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mgrmap(e 10) ==== 46338+0+0 (secure 0 0 0) 0x7f715001a430 con 0x7f715c07eed0 2024-04-26T23:35:00.206 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:58.270+0000 7f7159600700 1 -- 172.21.15.72:0/3909250195 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7f7144039650 msgr2=0x7f714403bb10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T23:35:00.206 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:58.270+0000 7f7159600700 1 --2- 172.21.15.72:0/3909250195 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7f7144039650 0x7f714403bb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:35:00.206 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:59.272+0000 7f7159600700 1 -- 172.21.15.72:0/3909250195 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mgrmap(e 11) ==== 46465+0+0 (secure 0 0 0) 0x7f715004fb00 con 0x7f715c07eed0 2024-04-26T23:35:00.206 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:59.273+0000 7f7159600700 1 --2- 172.21.15.72:0/3909250195 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f714403cfe0 0x7f714403f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:35:00.206 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:59.273+0000 7f7159600700 1 -- 172.21.15.72:0/3909250195 --> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f715000af30 con 0x7f714403cfe0 2024-04-26T23:35:00.206 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:59.273+0000 7f715be00700 1 --2- 172.21.15.72:0/3909250195 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f714403cfe0 0x7f714403f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:35:00.206 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:59.274+0000 7f715be00700 1 --2- 172.21.15.72:0/3909250195 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f714403cfe0 0x7f714403f3d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f7148003a10 tx=0x7f7148009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:35:00.206 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:59.276+0000 7f7159600700 1 -- 172.21.15.72:0/3909250195 <== mgr.14152 v2:172.21.15.72:6800/4281177709 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f715000af30 con 0x7f714403cfe0 2024-04-26T23:35:00.206 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:59.285+0000 7f7163400700 1 -- 172.21.15.72:0/3909250195 --> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f715c06a540 con 0x7f714403cfe0 2024-04-26T23:35:00.206 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:59.285+0000 7f7159600700 1 -- 172.21.15.72:0/3909250195 <== mgr.14152 v2:172.21.15.72:6800/4281177709 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f715c06a540 con 0x7f714403cfe0 2024-04-26T23:35:00.207 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:59.286+0000 7f7163400700 1 -- 172.21.15.72:0/3909250195 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f714403cfe0 msgr2=0x7f714403f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:35:00.207 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:59.286+0000 7f7163400700 1 --2- 172.21.15.72:0/3909250195 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f714403cfe0 0x7f714403f3d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f7148003a10 tx=0x7f7148009270 comp rx=0 tx=0).stop 2024-04-26T23:35:00.207 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:59.286+0000 7f7163400700 1 -- 172.21.15.72:0/3909250195 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f715c07eed0 msgr2=0x7f715c1a32b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:35:00.207 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:59.286+0000 7f7163400700 1 --2- 172.21.15.72:0/3909250195 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f715c07eed0 0x7f715c1a32b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7150009520 tx=0x7f7150011aa0 comp rx=0 tx=0).stop 2024-04-26T23:35:00.207 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:59.286+0000 7f7163400700 1 -- 172.21.15.72:0/3909250195 shutdown_connections 2024-04-26T23:35:00.207 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:59.286+0000 7f7163400700 1 --2- 172.21.15.72:0/3909250195 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f714403cfe0 0x7f714403f3d0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:35:00.207 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:59.286+0000 7f7163400700 1 --2- 172.21.15.72:0/3909250195 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f715c07eed0 0x7f715c1a32b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:35:00.207 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:59.286+0000 7f7163400700 1 --2- 172.21.15.72:0/3909250195 >> [v2:172.21.15.72:6800/2903979134,v1:172.21.15.72:6801/2903979134] conn(0x7f7144039650 0x7f714403bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:35:00.207 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:59.286+0000 7f7163400700 1 -- 172.21.15.72:0/3909250195 >> 172.21.15.72:0/3909250195 conn(0x7f715c1042d0 msgr2=0x7f715c106750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:35:00.207 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:59.286+0000 7f7163400700 1 -- 172.21.15.72:0/3909250195 shutdown_connections 2024-04-26T23:35:00.207 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:34:59.286+0000 7f7163400700 1 -- 172.21.15.72:0/3909250195 wait complete. 2024-04-26T23:35:00.207 INFO:teuthology.orchestra.run.smithi072.stdout:mgr epoch 9 is available 2024-04-26T23:35:00.207 INFO:teuthology.orchestra.run.smithi072.stdout:Generating a dashboard self-signed certificate... 2024-04-26T23:35:00.648 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:00 smithi072 ceph-mon[28184]: [26/Apr/2024:23:34:58] ENGINE Serving on https://172.21.15.72:7150 2024-04-26T23:35:00.648 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:00 smithi072 ceph-mon[28184]: [26/Apr/2024:23:34:58] ENGINE Serving on http://172.21.15.72:8765 2024-04-26T23:35:00.648 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:00 smithi072 ceph-mon[28184]: [26/Apr/2024:23:34:58] ENGINE Bus STARTED 2024-04-26T23:35:00.648 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:00 smithi072 ceph-mon[28184]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-26T23:35:00.649 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:00 smithi072 ceph-mon[28184]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-26T23:35:01.542 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:01 smithi072 ceph-mon[28184]: mgrmap e12: x(active, since 2s) 2024-04-26T23:35:02.419 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-26T23:35:02.419 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.333+0000 7fdebb200700 1 Processor -- start 2024-04-26T23:35:02.419 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.333+0000 7fdebb200700 1 -- start start 2024-04-26T23:35:02.419 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.334+0000 7fdebb200700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdeb4108df0 0x7fdeb4109210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:35:02.419 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.334+0000 7fdebb200700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fdeb41097e0 con 0x7fdeb4108df0 2024-04-26T23:35:02.419 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.334+0000 7fdeb3e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdeb4108df0 0x7fdeb4109210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:35:02.420 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.334+0000 7fdeb3e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdeb4108df0 0x7fdeb4109210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34868/0 (socket says 172.21.15.72:34868) 2024-04-26T23:35:02.420 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.334+0000 7fdeb3e00700 1 -- 172.21.15.72:0/3630352774 learned_addr learned my addr 172.21.15.72:0/3630352774 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:35:02.420 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.335+0000 7fdeb3e00700 1 -- 172.21.15.72:0/3630352774 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdeb4109f50 con 0x7fdeb4108df0 2024-04-26T23:35:02.420 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.335+0000 7fdeb3e00700 1 --2- 172.21.15.72:0/3630352774 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdeb4108df0 0x7fdeb4109210 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fdea40098a0 tx=0x7fdea4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f920328b7d2fd080 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:35:02.420 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.336+0000 7fdeb2a00700 1 -- 172.21.15.72:0/3630352774 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdea4004160 con 0x7fdeb4108df0 2024-04-26T23:35:02.420 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.336+0000 7fdeb2a00700 1 -- 172.21.15.72:0/3630352774 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdea4004300 con 0x7fdeb4108df0 2024-04-26T23:35:02.420 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.336+0000 7fdeb2a00700 1 -- 172.21.15.72:0/3630352774 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdea4011a20 con 0x7fdeb4108df0 2024-04-26T23:35:02.420 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.337+0000 7fdebb200700 1 -- 172.21.15.72:0/3630352774 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdeb4108df0 msgr2=0x7fdeb4109210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:35:02.420 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.337+0000 7fdebb200700 1 --2- 172.21.15.72:0/3630352774 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdeb4108df0 0x7fdeb4109210 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fdea40098a0 tx=0x7fdea4009bb0 comp rx=0 tx=0).stop 2024-04-26T23:35:02.420 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.337+0000 7fdebb200700 1 -- 172.21.15.72:0/3630352774 shutdown_connections 2024-04-26T23:35:02.420 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.337+0000 7fdebb200700 1 --2- 172.21.15.72:0/3630352774 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdeb4108df0 0x7fdeb4109210 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:35:02.420 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.337+0000 7fdebb200700 1 -- 172.21.15.72:0/3630352774 >> 172.21.15.72:0/3630352774 conn(0x7fdeb4104450 msgr2=0x7fdeb4106870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:35:02.420 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.337+0000 7fdebb200700 1 -- 172.21.15.72:0/3630352774 shutdown_connections 2024-04-26T23:35:02.421 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.337+0000 7fdebb200700 1 -- 172.21.15.72:0/3630352774 wait complete. 2024-04-26T23:35:02.421 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.339+0000 7fdebb200700 1 Processor -- start 2024-04-26T23:35:02.421 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.339+0000 7fdebb200700 1 -- start start 2024-04-26T23:35:02.421 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.340+0000 7fdebb200700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdeb4108df0 0x7fdeb419cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:35:02.421 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.340+0000 7fdebb200700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fdeb410d300 con 0x7fdeb4108df0 2024-04-26T23:35:02.421 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.340+0000 7fdeb3e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdeb4108df0 0x7fdeb419cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:35:02.421 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.340+0000 7fdeb3e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdeb4108df0 0x7fdeb419cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34882/0 (socket says 172.21.15.72:34882) 2024-04-26T23:35:02.421 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.340+0000 7fdeb3e00700 1 -- 172.21.15.72:0/3129917395 learned_addr learned my addr 172.21.15.72:0/3129917395 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:35:02.421 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.341+0000 7fdeb3e00700 1 -- 172.21.15.72:0/3129917395 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdeb419d120 con 0x7fdeb4108df0 2024-04-26T23:35:02.421 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.341+0000 7fdeb3e00700 1 --2- 172.21.15.72:0/3129917395 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdeb4108df0 0x7fdeb419cbe0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdea4009520 tx=0x7fdea4011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:35:02.421 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.341+0000 7fdeb0c00700 1 -- 172.21.15.72:0/3129917395 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdea40039e0 con 0x7fdeb4108df0 2024-04-26T23:35:02.422 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.341+0000 7fdeb0c00700 1 -- 172.21.15.72:0/3129917395 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdea401b8f0 con 0x7fdeb4108df0 2024-04-26T23:35:02.422 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.341+0000 7fdeb0c00700 1 -- 172.21.15.72:0/3129917395 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdea4019770 con 0x7fdeb4108df0 2024-04-26T23:35:02.422 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.341+0000 7fdebb200700 1 -- 172.21.15.72:0/3129917395 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdeb419d3b0 con 0x7fdeb4108df0 2024-04-26T23:35:02.422 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.342+0000 7fdebb200700 1 -- 172.21.15.72:0/3129917395 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdeb41a00a0 con 0x7fdeb4108df0 2024-04-26T23:35:02.422 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.343+0000 7fdebb200700 1 -- 172.21.15.72:0/3129917395 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdeb4064a80 con 0x7fdeb4108df0 2024-04-26T23:35:02.422 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.343+0000 7fdeb0c00700 1 -- 172.21.15.72:0/3129917395 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7fdea401b440 con 0x7fdeb4108df0 2024-04-26T23:35:02.422 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.344+0000 7fdeb0c00700 1 --2- 172.21.15.72:0/3129917395 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fde98039650 0x7fde9803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:35:02.422 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.344+0000 7fdeb0c00700 1 -- 172.21.15.72:0/3129917395 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fdea404dff0 con 0x7fdeb4108df0 2024-04-26T23:35:02.422 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.344+0000 7fdeb3400700 1 --2- 172.21.15.72:0/3129917395 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fde98039650 0x7fde9803bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:35:02.422 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.345+0000 7fdeb3400700 1 --2- 172.21.15.72:0/3129917395 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fde98039650 0x7fde9803bb10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdea0006fd0 tx=0x7fdea0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:35:02.422 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.350+0000 7fdeb0c00700 1 -- 172.21.15.72:0/3129917395 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fdea401f030 con 0x7fdeb4108df0 2024-04-26T23:35:02.423 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.587+0000 7fdebb200700 1 -- 172.21.15.72:0/3129917395 --> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7fdeb41055c0 con 0x7fde98039650 2024-04-26T23:35:02.423 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.707+0000 7fdeb0c00700 1 -- 172.21.15.72:0/3129917395 <== mgr.14152 v2:172.21.15.72:6800/4281177709 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7fdeb41055c0 con 0x7fde98039650 2024-04-26T23:35:02.423 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.709+0000 7fdebb200700 1 -- 172.21.15.72:0/3129917395 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fde98039650 msgr2=0x7fde9803bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:35:02.423 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.709+0000 7fdebb200700 1 --2- 172.21.15.72:0/3129917395 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fde98039650 0x7fde9803bb10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdea0006fd0 tx=0x7fdea0006e40 comp rx=0 tx=0).stop 2024-04-26T23:35:02.423 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.709+0000 7fdebb200700 1 -- 172.21.15.72:0/3129917395 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdeb4108df0 msgr2=0x7fdeb419cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:35:02.423 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.709+0000 7fdebb200700 1 --2- 172.21.15.72:0/3129917395 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdeb4108df0 0x7fdeb419cbe0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdea4009520 tx=0x7fdea4011aa0 comp rx=0 tx=0).stop 2024-04-26T23:35:02.423 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.710+0000 7fdebb200700 1 -- 172.21.15.72:0/3129917395 shutdown_connections 2024-04-26T23:35:02.423 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.710+0000 7fdebb200700 1 --2- 172.21.15.72:0/3129917395 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fde98039650 0x7fde9803bb10 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:35:02.423 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.710+0000 7fdebb200700 1 --2- 172.21.15.72:0/3129917395 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdeb4108df0 0x7fdeb419cbe0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:35:02.423 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.710+0000 7fdebb200700 1 -- 172.21.15.72:0/3129917395 >> 172.21.15.72:0/3129917395 conn(0x7fdeb4104450 msgr2=0x7fdeb4104eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:35:02.423 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.710+0000 7fdebb200700 1 -- 172.21.15.72:0/3129917395 shutdown_connections 2024-04-26T23:35:02.423 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:01.710+0000 7fdebb200700 1 -- 172.21.15.72:0/3129917395 wait complete. 2024-04-26T23:35:02.424 INFO:teuthology.orchestra.run.smithi072.stdout:Creating initial admin user... 2024-04-26T23:35:02.703 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:02 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:02.761 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:02 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:02.761 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:02 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:03.524 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:03 smithi072 ceph-mon[28184]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:35:03.524 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:03 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:04.186 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$0.6r1WKi54Vt/n7y.d2nPe0Jnk8dJckW/TDajOJ/TL.oZ3OtGfLbK", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1714174503, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-26T23:35:04.186 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.001+0000 7f44c5800700 1 Processor -- start 2024-04-26T23:35:04.186 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.002+0000 7f44c5800700 1 -- start start 2024-04-26T23:35:04.186 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.002+0000 7f44c5800700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44c0109800 0x7f44c010bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:35:04.187 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.002+0000 7f44c5800700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f44c007cea0 con 0x7f44c0109800 2024-04-26T23:35:04.187 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.002+0000 7f44bea00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44c0109800 0x7f44c010bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:35:04.187 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.002+0000 7f44bea00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44c0109800 0x7f44c010bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34890/0 (socket says 172.21.15.72:34890) 2024-04-26T23:35:04.187 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.002+0000 7f44bea00700 1 -- 172.21.15.72:0/3375950581 learned_addr learned my addr 172.21.15.72:0/3375950581 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:35:04.187 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.003+0000 7f44bea00700 1 -- 172.21.15.72:0/3375950581 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44c010c130 con 0x7f44c0109800 2024-04-26T23:35:04.187 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.003+0000 7f44bea00700 1 --2- 172.21.15.72:0/3375950581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44c0109800 0x7f44c010bbf0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f44b00098a0 tx=0x7f44b0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cfb6e43ca1c741a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:35:04.187 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.003+0000 7f44bd600700 1 -- 172.21.15.72:0/3375950581 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f44b0004160 con 0x7f44c0109800 2024-04-26T23:35:04.187 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.004+0000 7f44bd600700 1 -- 172.21.15.72:0/3375950581 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44b0004300 con 0x7f44c0109800 2024-04-26T23:35:04.187 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.004+0000 7f44bd600700 1 -- 172.21.15.72:0/3375950581 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f44b0011a20 con 0x7f44c0109800 2024-04-26T23:35:04.187 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.004+0000 7f44c5800700 1 -- 172.21.15.72:0/3375950581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44c0109800 msgr2=0x7f44c010bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:35:04.187 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.004+0000 7f44c5800700 1 --2- 172.21.15.72:0/3375950581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44c0109800 0x7f44c010bbf0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f44b00098a0 tx=0x7f44b0009bb0 comp rx=0 tx=0).stop 2024-04-26T23:35:04.187 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.005+0000 7f44c5800700 1 -- 172.21.15.72:0/3375950581 shutdown_connections 2024-04-26T23:35:04.187 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.005+0000 7f44c5800700 1 --2- 172.21.15.72:0/3375950581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44c0109800 0x7f44c010bbf0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:35:04.188 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.005+0000 7f44c5800700 1 -- 172.21.15.72:0/3375950581 >> 172.21.15.72:0/3375950581 conn(0x7f44c01031f0 msgr2=0x7f44c0105670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:35:04.188 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.005+0000 7f44c5800700 1 -- 172.21.15.72:0/3375950581 shutdown_connections 2024-04-26T23:35:04.188 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.005+0000 7f44c5800700 1 -- 172.21.15.72:0/3375950581 wait complete. 2024-04-26T23:35:04.188 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.006+0000 7f44c5800700 1 Processor -- start 2024-04-26T23:35:04.188 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.006+0000 7f44c5800700 1 -- start start 2024-04-26T23:35:04.188 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.006+0000 7f44c5800700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44c0109800 0x7f44c019cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:35:04.188 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.006+0000 7f44c5800700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f44c010d140 con 0x7f44c0109800 2024-04-26T23:35:04.188 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.007+0000 7f44bea00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44c0109800 0x7f44c019cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:35:04.188 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.007+0000 7f44bea00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44c0109800 0x7f44c019cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34902/0 (socket says 172.21.15.72:34902) 2024-04-26T23:35:04.188 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.007+0000 7f44bea00700 1 -- 172.21.15.72:0/2966382444 learned_addr learned my addr 172.21.15.72:0/2966382444 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:35:04.188 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.007+0000 7f44bea00700 1 -- 172.21.15.72:0/2966382444 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44c019d1e0 con 0x7f44c0109800 2024-04-26T23:35:04.188 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.007+0000 7f44bea00700 1 --2- 172.21.15.72:0/2966382444 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44c0109800 0x7f44c019cca0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f44b00116e0 tx=0x7f44b0011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:35:04.188 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.008+0000 7f44b7400700 1 -- 172.21.15.72:0/2966382444 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f44b0003b20 con 0x7f44c0109800 2024-04-26T23:35:04.189 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.008+0000 7f44b7400700 1 -- 172.21.15.72:0/2966382444 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44b001b8f0 con 0x7f44c0109800 2024-04-26T23:35:04.189 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.008+0000 7f44c5800700 1 -- 172.21.15.72:0/2966382444 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44c019d470 con 0x7f44c0109800 2024-04-26T23:35:04.189 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.008+0000 7f44b7400700 1 -- 172.21.15.72:0/2966382444 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f44b00238c0 con 0x7f44c0109800 2024-04-26T23:35:04.189 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.008+0000 7f44c5800700 1 -- 172.21.15.72:0/2966382444 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44c01a0160 con 0x7f44c0109800 2024-04-26T23:35:04.189 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.009+0000 7f44c5800700 1 -- 172.21.15.72:0/2966382444 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44c0064a80 con 0x7f44c0109800 2024-04-26T23:35:04.189 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.010+0000 7f44b7400700 1 -- 172.21.15.72:0/2966382444 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f44b001baa0 con 0x7f44c0109800 2024-04-26T23:35:04.189 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.011+0000 7f44b7400700 1 --2- 172.21.15.72:0/2966382444 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f44a4039610 0x7f44a403bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:35:04.189 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.011+0000 7f44b7400700 1 -- 172.21.15.72:0/2966382444 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f44b004e670 con 0x7f44c0109800 2024-04-26T23:35:04.189 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.011+0000 7f44be000700 1 --2- 172.21.15.72:0/2966382444 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f44a4039610 0x7f44a403bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:35:04.189 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.012+0000 7f44be000700 1 --2- 172.21.15.72:0/2966382444 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f44a4039610 0x7f44a403bad0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f44ac006fd0 tx=0x7f44ac006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:35:04.189 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.016+0000 7f44b7400700 1 -- 172.21.15.72:0/2966382444 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f44b001f030 con 0x7f44c0109800 2024-04-26T23:35:04.189 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.263+0000 7f44c5800700 1 -- 172.21.15.72:0/2966382444 --> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f44c01a03f0 con 0x7f44a4039610 2024-04-26T23:35:04.189 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.522+0000 7f44b7400700 1 -- 172.21.15.72:0/2966382444 <== mgr.14152 v2:172.21.15.72:6800/4281177709 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f44c01a03f0 con 0x7f44a4039610 2024-04-26T23:35:04.190 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.524+0000 7f44c5800700 1 -- 172.21.15.72:0/2966382444 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f44a4039610 msgr2=0x7f44a403bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:35:04.190 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.524+0000 7f44c5800700 1 --2- 172.21.15.72:0/2966382444 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f44a4039610 0x7f44a403bad0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f44ac006fd0 tx=0x7f44ac006e40 comp rx=0 tx=0).stop 2024-04-26T23:35:04.190 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.524+0000 7f44c5800700 1 -- 172.21.15.72:0/2966382444 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44c0109800 msgr2=0x7f44c019cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:35:04.190 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.524+0000 7f44c5800700 1 --2- 172.21.15.72:0/2966382444 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44c0109800 0x7f44c019cca0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f44b00116e0 tx=0x7f44b0011aa0 comp rx=0 tx=0).stop 2024-04-26T23:35:04.190 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.524+0000 7f44c5800700 1 -- 172.21.15.72:0/2966382444 shutdown_connections 2024-04-26T23:35:04.190 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.524+0000 7f44c5800700 1 --2- 172.21.15.72:0/2966382444 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f44a4039610 0x7f44a403bad0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:35:04.190 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.524+0000 7f44c5800700 1 --2- 172.21.15.72:0/2966382444 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44c0109800 0x7f44c019cca0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:35:04.191 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.524+0000 7f44c5800700 1 -- 172.21.15.72:0/2966382444 >> 172.21.15.72:0/2966382444 conn(0x7f44c01031f0 msgr2=0x7f44c0103cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:35:04.191 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.525+0000 7f44c5800700 1 -- 172.21.15.72:0/2966382444 shutdown_connections 2024-04-26T23:35:04.191 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:03.525+0000 7f44c5800700 1 -- 172.21.15.72:0/2966382444 wait complete. 2024-04-26T23:35:04.191 INFO:teuthology.orchestra.run.smithi072.stdout:Fetching dashboard port number... 2024-04-26T23:35:04.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:04 smithi072 ceph-mon[28184]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:35:04.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:04 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:05.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:05 smithi072 ceph-mon[28184]: mgrmap e13: x(active, since 6s) 2024-04-26T23:35:05.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:05 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/247148032' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-26T23:35:06.238 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 8443 2024-04-26T23:35:06.238 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:04.829+0000 7f6b7b000700 1 Processor -- start 2024-04-26T23:35:06.238 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:04.829+0000 7f6b7b000700 1 -- start start 2024-04-26T23:35:06.238 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:04.829+0000 7f6b7b000700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b74108d50 0x7f6b74109170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:35:06.239 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:04.829+0000 7f6b7b000700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f6b74109740 con 0x7f6b74108d50 2024-04-26T23:35:06.239 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:04.830+0000 7f6b73e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b74108d50 0x7f6b74109170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:35:06.239 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:04.830+0000 7f6b73e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b74108d50 0x7f6b74109170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34910/0 (socket says 172.21.15.72:34910) 2024-04-26T23:35:06.239 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:04.830+0000 7f6b73e00700 1 -- 172.21.15.72:0/2753667771 learned_addr learned my addr 172.21.15.72:0/2753667771 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:35:06.239 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:04.830+0000 7f6b73e00700 1 -- 172.21.15.72:0/2753667771 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b74109f50 con 0x7f6b74108d50 2024-04-26T23:35:06.239 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:04.831+0000 7f6b73e00700 1 --2- 172.21.15.72:0/2753667771 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b74108d50 0x7f6b74109170 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6b640098a0 tx=0x7f6b64009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ff16629830fcb08 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:35:06.239 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:04.831+0000 7f6b72a00700 1 -- 172.21.15.72:0/2753667771 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6b64004160 con 0x7f6b74108d50 2024-04-26T23:35:06.239 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:04.831+0000 7f6b72a00700 1 -- 172.21.15.72:0/2753667771 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b64004300 con 0x7f6b74108d50 2024-04-26T23:35:06.239 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:04.831+0000 7f6b72a00700 1 -- 172.21.15.72:0/2753667771 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6b64011a20 con 0x7f6b74108d50 2024-04-26T23:35:06.239 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:04.832+0000 7f6b7b000700 1 -- 172.21.15.72:0/2753667771 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b74108d50 msgr2=0x7f6b74109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:35:06.239 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:04.832+0000 7f6b7b000700 1 --2- 172.21.15.72:0/2753667771 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b74108d50 0x7f6b74109170 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6b640098a0 tx=0x7f6b64009bb0 comp rx=0 tx=0).stop 2024-04-26T23:35:06.239 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:04.832+0000 7f6b7b000700 1 -- 172.21.15.72:0/2753667771 shutdown_connections 2024-04-26T23:35:06.239 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:04.832+0000 7f6b7b000700 1 --2- 172.21.15.72:0/2753667771 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b74108d50 0x7f6b74109170 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:35:06.239 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:04.832+0000 7f6b7b000700 1 -- 172.21.15.72:0/2753667771 >> 172.21.15.72:0/2753667771 conn(0x7f6b741042d0 msgr2=0x7f6b74106730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:35:06.239 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:04.832+0000 7f6b7b000700 1 -- 172.21.15.72:0/2753667771 shutdown_connections 2024-04-26T23:35:06.240 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:04.832+0000 7f6b7b000700 1 -- 172.21.15.72:0/2753667771 wait complete. 2024-04-26T23:35:06.240 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:04.833+0000 7f6b7b000700 1 Processor -- start 2024-04-26T23:35:06.240 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:04.833+0000 7f6b7b000700 1 -- start start 2024-04-26T23:35:06.240 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:04.833+0000 7f6b7b000700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b74108d50 0x7f6b7419cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:35:06.240 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:04.833+0000 7f6b7b000700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f6b7410d300 con 0x7f6b74108d50 2024-04-26T23:35:06.240 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:04.834+0000 7f6b73e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b74108d50 0x7f6b7419cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:35:06.240 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:04.834+0000 7f6b73e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b74108d50 0x7f6b7419cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34924/0 (socket says 172.21.15.72:34924) 2024-04-26T23:35:06.240 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:04.834+0000 7f6b73e00700 1 -- 172.21.15.72:0/247148032 learned_addr learned my addr 172.21.15.72:0/247148032 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:35:06.240 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:04.834+0000 7f6b73e00700 1 -- 172.21.15.72:0/247148032 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b7419d1c0 con 0x7f6b74108d50 2024-04-26T23:35:06.240 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:04.834+0000 7f6b73e00700 1 --2- 172.21.15.72:0/247148032 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b74108d50 0x7f6b7419cc80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6b64000c00 tx=0x7f6b64011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:35:06.240 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:04.835+0000 7f6b70c00700 1 -- 172.21.15.72:0/247148032 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6b64003aa0 con 0x7f6b74108d50 2024-04-26T23:35:06.240 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:04.835+0000 7f6b70c00700 1 -- 172.21.15.72:0/247148032 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b6401b8f0 con 0x7f6b74108d50 2024-04-26T23:35:06.240 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:04.835+0000 7f6b70c00700 1 -- 172.21.15.72:0/247148032 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6b64023890 con 0x7f6b74108d50 2024-04-26T23:35:06.240 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:04.835+0000 7f6b7b000700 1 -- 172.21.15.72:0/247148032 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b7419d450 con 0x7f6b74108d50 2024-04-26T23:35:06.240 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:04.835+0000 7f6b7b000700 1 -- 172.21.15.72:0/247148032 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b741a0140 con 0x7f6b74108d50 2024-04-26T23:35:06.241 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:04.836+0000 7f6b70c00700 1 -- 172.21.15.72:0/247148032 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f6b6401b440 con 0x7f6b74108d50 2024-04-26T23:35:06.241 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:04.836+0000 7f6b7b000700 1 -- 172.21.15.72:0/247148032 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b74064a80 con 0x7f6b74108d50 2024-04-26T23:35:06.241 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:04.837+0000 7f6b70c00700 1 --2- 172.21.15.72:0/247148032 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6b58039700 0x7f6b5803bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:35:06.241 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:04.837+0000 7f6b70c00700 1 -- 172.21.15.72:0/247148032 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f6b6404e100 con 0x7f6b74108d50 2024-04-26T23:35:06.241 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:04.837+0000 7f6b73400700 1 --2- 172.21.15.72:0/247148032 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6b58039700 0x7f6b5803bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:35:06.241 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:04.837+0000 7f6b73400700 1 --2- 172.21.15.72:0/247148032 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6b58039700 0x7f6b5803bbc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6b60006fd0 tx=0x7f6b60006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:35:06.241 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:04.842+0000 7f6b70c00700 1 -- 172.21.15.72:0/247148032 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6b6401f030 con 0x7f6b74108d50 2024-04-26T23:35:06.241 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:05.069+0000 7f6b7b000700 1 -- 172.21.15.72:0/247148032 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f6b741a0730 con 0x7f6b74108d50 2024-04-26T23:35:06.241 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:05.069+0000 7f6b70c00700 1 -- 172.21.15.72:0/247148032 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f6b640189e0 con 0x7f6b74108d50 2024-04-26T23:35:06.241 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:05.072+0000 7f6b7b000700 1 -- 172.21.15.72:0/247148032 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6b58039700 msgr2=0x7f6b5803bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:35:06.241 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:05.072+0000 7f6b7b000700 1 --2- 172.21.15.72:0/247148032 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6b58039700 0x7f6b5803bbc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6b60006fd0 tx=0x7f6b60006e40 comp rx=0 tx=0).stop 2024-04-26T23:35:06.241 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:05.072+0000 7f6b7b000700 1 -- 172.21.15.72:0/247148032 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b74108d50 msgr2=0x7f6b7419cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:35:06.241 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:05.072+0000 7f6b7b000700 1 --2- 172.21.15.72:0/247148032 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b74108d50 0x7f6b7419cc80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6b64000c00 tx=0x7f6b64011aa0 comp rx=0 tx=0).stop 2024-04-26T23:35:06.242 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:05.072+0000 7f6b7b000700 1 -- 172.21.15.72:0/247148032 shutdown_connections 2024-04-26T23:35:06.242 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:05.072+0000 7f6b7b000700 1 --2- 172.21.15.72:0/247148032 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6b58039700 0x7f6b5803bbc0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:35:06.242 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:05.072+0000 7f6b7b000700 1 --2- 172.21.15.72:0/247148032 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b74108d50 0x7f6b7419cc80 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:35:06.242 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:05.072+0000 7f6b7b000700 1 -- 172.21.15.72:0/247148032 >> 172.21.15.72:0/247148032 conn(0x7f6b741042d0 msgr2=0x7f6b74104db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:35:06.242 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:05.073+0000 7f6b7b000700 1 -- 172.21.15.72:0/247148032 shutdown_connections 2024-04-26T23:35:06.242 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:05.073+0000 7f6b7b000700 1 -- 172.21.15.72:0/247148032 wait complete. 2024-04-26T23:35:06.263 INFO:teuthology.orchestra.run.smithi072.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T23:35:06.263 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stdout disabled 2024-04-26T23:35:06.287 INFO:teuthology.orchestra.run.smithi072.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T23:35:06.287 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stdout inactive 2024-04-26T23:35:06.288 INFO:teuthology.orchestra.run.smithi072.stdout:firewalld.service is not enabled 2024-04-26T23:35:06.288 INFO:teuthology.orchestra.run.smithi072.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-26T23:35:06.295 INFO:teuthology.orchestra.run.smithi072.stdout:Ceph Dashboard is now available at: 2024-04-26T23:35:06.295 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:35:06.295 INFO:teuthology.orchestra.run.smithi072.stdout: URL: https://smithi072.front.sepia.ceph.com:8443/ 2024-04-26T23:35:06.295 INFO:teuthology.orchestra.run.smithi072.stdout: User: admin 2024-04-26T23:35:06.296 INFO:teuthology.orchestra.run.smithi072.stdout: Password: cftp3azlpf 2024-04-26T23:35:06.296 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:35:06.296 INFO:teuthology.orchestra.run.smithi072.stdout:Saving cluster configuration to /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/config directory 2024-04-26T23:35:06.299 INFO:teuthology.orchestra.run.smithi072.stdout:Enabling autotune for osd_memory_target 2024-04-26T23:35:08.321 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.408+0000 7f809f600700 1 Processor -- start 2024-04-26T23:35:08.321 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.408+0000 7f809f600700 1 -- start start 2024-04-26T23:35:08.321 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.408+0000 7f809f600700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8098082dd0 0x7f80980831f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:35:08.321 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.408+0000 7f809f600700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f80980837c0 con 0x7f8098082dd0 2024-04-26T23:35:08.321 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.408+0000 7f809cc00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8098082dd0 0x7f80980831f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:35:08.321 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.408+0000 7f809cc00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8098082dd0 0x7f80980831f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36006/0 (socket says 172.21.15.72:36006) 2024-04-26T23:35:08.322 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.408+0000 7f809cc00700 1 -- 172.21.15.72:0/2812585598 learned_addr learned my addr 172.21.15.72:0/2812585598 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:35:08.322 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.409+0000 7f809cc00700 1 -- 172.21.15.72:0/2812585598 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8098110550 con 0x7f8098082dd0 2024-04-26T23:35:08.322 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.409+0000 7f809cc00700 1 --2- 172.21.15.72:0/2812585598 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8098082dd0 0x7f80980831f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f80880098a0 tx=0x7f8088009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee4203f0a71e023 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:35:08.322 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.409+0000 7f8097e00700 1 -- 172.21.15.72:0/2812585598 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8088004160 con 0x7f8098082dd0 2024-04-26T23:35:08.322 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.410+0000 7f8097e00700 1 -- 172.21.15.72:0/2812585598 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8088004300 con 0x7f8098082dd0 2024-04-26T23:35:08.322 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.410+0000 7f8097e00700 1 -- 172.21.15.72:0/2812585598 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8088011a20 con 0x7f8098082dd0 2024-04-26T23:35:08.322 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.410+0000 7f809f600700 1 -- 172.21.15.72:0/2812585598 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8098082dd0 msgr2=0x7f80980831f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:35:08.322 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.410+0000 7f809f600700 1 --2- 172.21.15.72:0/2812585598 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8098082dd0 0x7f80980831f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f80880098a0 tx=0x7f8088009bb0 comp rx=0 tx=0).stop 2024-04-26T23:35:08.326 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.410+0000 7f809f600700 1 -- 172.21.15.72:0/2812585598 shutdown_connections 2024-04-26T23:35:08.326 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.410+0000 7f809f600700 1 --2- 172.21.15.72:0/2812585598 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8098082dd0 0x7f80980831f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:35:08.326 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.410+0000 7f809f600700 1 -- 172.21.15.72:0/2812585598 >> 172.21.15.72:0/2812585598 conn(0x7f809807e390 msgr2=0x7f80980807b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:35:08.326 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.410+0000 7f809f600700 1 -- 172.21.15.72:0/2812585598 shutdown_connections 2024-04-26T23:35:08.326 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.410+0000 7f809f600700 1 -- 172.21.15.72:0/2812585598 wait complete. 2024-04-26T23:35:08.326 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.411+0000 7f809f600700 1 Processor -- start 2024-04-26T23:35:08.326 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.412+0000 7f809f600700 1 -- start start 2024-04-26T23:35:08.326 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.412+0000 7f809f600700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8098082dd0 0x7f80981a32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:35:08.327 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.412+0000 7f809f600700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f8098113900 con 0x7f8098082dd0 2024-04-26T23:35:08.327 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.412+0000 7f809cc00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8098082dd0 0x7f80981a32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:35:08.327 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.412+0000 7f809cc00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8098082dd0 0x7f80981a32a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36016/0 (socket says 172.21.15.72:36016) 2024-04-26T23:35:08.327 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.412+0000 7f809cc00700 1 -- 172.21.15.72:0/1382593724 learned_addr learned my addr 172.21.15.72:0/1382593724 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:35:08.327 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.413+0000 7f809cc00700 1 -- 172.21.15.72:0/1382593724 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80981a37e0 con 0x7f8098082dd0 2024-04-26T23:35:08.327 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.413+0000 7f809cc00700 1 --2- 172.21.15.72:0/1382593724 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8098082dd0 0x7f80981a32a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f80880119f0 tx=0x7f8088011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:35:08.327 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.413+0000 7f8096000700 1 -- 172.21.15.72:0/1382593724 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f80880039e0 con 0x7f8098082dd0 2024-04-26T23:35:08.327 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.413+0000 7f8096000700 1 -- 172.21.15.72:0/1382593724 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f808801b8f0 con 0x7f8098082dd0 2024-04-26T23:35:08.327 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.413+0000 7f809f600700 1 -- 172.21.15.72:0/1382593724 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80981a3a70 con 0x7f8098082dd0 2024-04-26T23:35:08.327 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.413+0000 7f8096000700 1 -- 172.21.15.72:0/1382593724 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f80880237a0 con 0x7f8098082dd0 2024-04-26T23:35:08.327 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.413+0000 7f809f600700 1 -- 172.21.15.72:0/1382593724 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80981a6760 con 0x7f8098082dd0 2024-04-26T23:35:08.327 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.414+0000 7f8096000700 1 -- 172.21.15.72:0/1382593724 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f8088023a50 con 0x7f8098082dd0 2024-04-26T23:35:08.327 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.414+0000 7f809f600700 1 -- 172.21.15.72:0/1382593724 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8098064a80 con 0x7f8098082dd0 2024-04-26T23:35:08.328 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.415+0000 7f8096000700 1 --2- 172.21.15.72:0/1382593724 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f807c0396b0 0x7f807c03bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:35:08.328 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.415+0000 7f8096000700 1 -- 172.21.15.72:0/1382593724 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f808804e0d0 con 0x7f8098082dd0 2024-04-26T23:35:08.328 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.415+0000 7f808fe00700 1 --2- 172.21.15.72:0/1382593724 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f807c0396b0 0x7f807c03bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:35:08.328 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.416+0000 7f808fe00700 1 --2- 172.21.15.72:0/1382593724 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f807c0396b0 0x7f807c03bb70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8080006fd0 tx=0x7f8080006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:35:08.328 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.420+0000 7f8096000700 1 -- 172.21.15.72:0/1382593724 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f808801f030 con 0x7f8098082dd0 2024-04-26T23:35:08.328 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.645+0000 7f809f600700 1 -- 172.21.15.72:0/1382593724 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f80981a6d80 con 0x7f8098082dd0 2024-04-26T23:35:08.328 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.646+0000 7f8096000700 1 -- 172.21.15.72:0/1382593724 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f8088018c10 con 0x7f8098082dd0 2024-04-26T23:35:08.328 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.649+0000 7f809f600700 1 -- 172.21.15.72:0/1382593724 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f807c0396b0 msgr2=0x7f807c03bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:35:08.328 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.649+0000 7f809f600700 1 --2- 172.21.15.72:0/1382593724 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f807c0396b0 0x7f807c03bb70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8080006fd0 tx=0x7f8080006e40 comp rx=0 tx=0).stop 2024-04-26T23:35:08.328 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.649+0000 7f809f600700 1 -- 172.21.15.72:0/1382593724 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8098082dd0 msgr2=0x7f80981a32a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:35:08.328 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.649+0000 7f809f600700 1 --2- 172.21.15.72:0/1382593724 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8098082dd0 0x7f80981a32a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f80880119f0 tx=0x7f8088011ad0 comp rx=0 tx=0).stop 2024-04-26T23:35:08.328 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.649+0000 7f809f600700 1 -- 172.21.15.72:0/1382593724 shutdown_connections 2024-04-26T23:35:08.328 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.649+0000 7f809f600700 1 --2- 172.21.15.72:0/1382593724 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f807c0396b0 0x7f807c03bb70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:35:08.328 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.649+0000 7f809f600700 1 --2- 172.21.15.72:0/1382593724 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8098082dd0 0x7f80981a32a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:35:08.329 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.649+0000 7f809f600700 1 -- 172.21.15.72:0/1382593724 >> 172.21.15.72:0/1382593724 conn(0x7f809807e390 msgr2=0x7f809807eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:35:08.329 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.649+0000 7f809f600700 1 -- 172.21.15.72:0/1382593724 shutdown_connections 2024-04-26T23:35:08.329 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:07.649+0000 7f809f600700 1 -- 172.21.15.72:0/1382593724 wait complete. 2024-04-26T23:35:10.002 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:08.975+0000 7fdc9f400700 1 Processor -- start 2024-04-26T23:35:10.002 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:08.975+0000 7fdc9f400700 1 -- start start 2024-04-26T23:35:10.003 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:08.976+0000 7fdc9f400700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc98108d70 0x7fdc98109190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:35:10.003 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:08.976+0000 7fdc9f400700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fdc98109760 con 0x7fdc98108d70 2024-04-26T23:35:10.003 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:08.976+0000 7fdc9ca00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc98108d70 0x7fdc98109190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:35:10.003 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:08.976+0000 7fdc9ca00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc98108d70 0x7fdc98109190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36024/0 (socket says 172.21.15.72:36024) 2024-04-26T23:35:10.003 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:08.976+0000 7fdc9ca00700 1 -- 172.21.15.72:0/3800979717 learned_addr learned my addr 172.21.15.72:0/3800979717 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:35:10.003 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:08.977+0000 7fdc9ca00700 1 -- 172.21.15.72:0/3800979717 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc98109f70 con 0x7fdc98108d70 2024-04-26T23:35:10.004 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:08.977+0000 7fdc9ca00700 1 --2- 172.21.15.72:0/3800979717 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc98108d70 0x7fdc98109190 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c0098a0 tx=0x7fdc8c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c81f7431f1a1771 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:35:10.004 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:08.977+0000 7fdc97400700 1 -- 172.21.15.72:0/3800979717 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdc8c004160 con 0x7fdc98108d70 2024-04-26T23:35:10.004 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:08.977+0000 7fdc97400700 1 -- 172.21.15.72:0/3800979717 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc8c004300 con 0x7fdc98108d70 2024-04-26T23:35:10.004 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:08.977+0000 7fdc97400700 1 -- 172.21.15.72:0/3800979717 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdc8c011a20 con 0x7fdc98108d70 2024-04-26T23:35:10.004 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:08.978+0000 7fdc9f400700 1 -- 172.21.15.72:0/3800979717 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc98108d70 msgr2=0x7fdc98109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:35:10.004 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:08.978+0000 7fdc9f400700 1 --2- 172.21.15.72:0/3800979717 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc98108d70 0x7fdc98109190 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c0098a0 tx=0x7fdc8c009bb0 comp rx=0 tx=0).stop 2024-04-26T23:35:10.005 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:08.978+0000 7fdc9f400700 1 -- 172.21.15.72:0/3800979717 shutdown_connections 2024-04-26T23:35:10.005 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:08.978+0000 7fdc9f400700 1 --2- 172.21.15.72:0/3800979717 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc98108d70 0x7fdc98109190 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:35:10.005 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:08.978+0000 7fdc9f400700 1 -- 172.21.15.72:0/3800979717 >> 172.21.15.72:0/3800979717 conn(0x7fdc981042d0 msgr2=0x7fdc98106750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:35:10.005 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:08.978+0000 7fdc9f400700 1 -- 172.21.15.72:0/3800979717 shutdown_connections 2024-04-26T23:35:10.005 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:08.978+0000 7fdc9f400700 1 -- 172.21.15.72:0/3800979717 wait complete. 2024-04-26T23:35:10.005 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:08.979+0000 7fdc9f400700 1 Processor -- start 2024-04-26T23:35:10.006 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:08.980+0000 7fdc9f400700 1 -- start start 2024-04-26T23:35:10.006 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:08.980+0000 7fdc9f400700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc98108d70 0x7fdc9819eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:35:10.006 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:08.980+0000 7fdc9f400700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fdc9810d320 con 0x7fdc98108d70 2024-04-26T23:35:10.006 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:08.980+0000 7fdc9ca00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc98108d70 0x7fdc9819eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:35:10.006 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:08.980+0000 7fdc9ca00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc98108d70 0x7fdc9819eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36026/0 (socket says 172.21.15.72:36026) 2024-04-26T23:35:10.006 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:08.980+0000 7fdc9ca00700 1 -- 172.21.15.72:0/2548932711 learned_addr learned my addr 172.21.15.72:0/2548932711 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:35:10.007 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:08.980+0000 7fdc9ca00700 1 -- 172.21.15.72:0/2548932711 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc9819f400 con 0x7fdc98108d70 2024-04-26T23:35:10.007 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:08.981+0000 7fdc9ca00700 1 --2- 172.21.15.72:0/2548932711 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc98108d70 0x7fdc9819eec0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c000c00 tx=0x7fdc8c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:35:10.007 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:08.981+0000 7fdc95600700 1 -- 172.21.15.72:0/2548932711 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdc8c003aa0 con 0x7fdc98108d70 2024-04-26T23:35:10.007 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:08.981+0000 7fdc95600700 1 -- 172.21.15.72:0/2548932711 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc8c01b8f0 con 0x7fdc98108d70 2024-04-26T23:35:10.007 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:08.981+0000 7fdc95600700 1 -- 172.21.15.72:0/2548932711 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdc8c023890 con 0x7fdc98108d70 2024-04-26T23:35:10.007 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:08.981+0000 7fdc9f400700 1 -- 172.21.15.72:0/2548932711 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc9819f690 con 0x7fdc98108d70 2024-04-26T23:35:10.008 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:08.981+0000 7fdc9f400700 1 -- 172.21.15.72:0/2548932711 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc981a2380 con 0x7fdc98108d70 2024-04-26T23:35:10.008 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:08.982+0000 7fdc95600700 1 -- 172.21.15.72:0/2548932711 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fdc8c01b440 con 0x7fdc98108d70 2024-04-26T23:35:10.008 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:08.982+0000 7fdc9f400700 1 -- 172.21.15.72:0/2548932711 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc98064a80 con 0x7fdc98108d70 2024-04-26T23:35:10.008 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:08.983+0000 7fdc95600700 1 --2- 172.21.15.72:0/2548932711 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fdc80039700 0x7fdc8003bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:35:10.008 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:08.983+0000 7fdc95600700 1 -- 172.21.15.72:0/2548932711 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fdc8c04e020 con 0x7fdc98108d70 2024-04-26T23:35:10.008 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:08.983+0000 7fdc97e00700 1 --2- 172.21.15.72:0/2548932711 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fdc80039700 0x7fdc8003bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:35:10.009 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:08.984+0000 7fdc97e00700 1 --2- 172.21.15.72:0/2548932711 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fdc80039700 0x7fdc8003bbc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fdc88006fd0 tx=0x7fdc88006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:35:10.009 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:08.989+0000 7fdc95600700 1 -- 172.21.15.72:0/2548932711 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fdc8c01f030 con 0x7fdc98108d70 2024-04-26T23:35:10.009 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:09.328+0000 7fdc9f400700 1 -- 172.21.15.72:0/2548932711 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7fdc981a2970 con 0x7fdc98108d70 2024-04-26T23:35:10.009 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:09.331+0000 7fdc95600700 1 -- 172.21.15.72:0/2548932711 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) ==== 153+0+0 (secure 0 0 0) 0x7fdc8c0185b0 con 0x7fdc98108d70 2024-04-26T23:35:10.009 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:09.334+0000 7fdc9f400700 1 -- 172.21.15.72:0/2548932711 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fdc80039700 msgr2=0x7fdc8003bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:35:10.009 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:09.334+0000 7fdc9f400700 1 --2- 172.21.15.72:0/2548932711 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fdc80039700 0x7fdc8003bbc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fdc88006fd0 tx=0x7fdc88006e40 comp rx=0 tx=0).stop 2024-04-26T23:35:10.010 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:09.335+0000 7fdc9f400700 1 -- 172.21.15.72:0/2548932711 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc98108d70 msgr2=0x7fdc9819eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:35:10.010 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:09.335+0000 7fdc9f400700 1 --2- 172.21.15.72:0/2548932711 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc98108d70 0x7fdc9819eec0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c000c00 tx=0x7fdc8c011aa0 comp rx=0 tx=0).stop 2024-04-26T23:35:10.010 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:09.335+0000 7fdc9f400700 1 -- 172.21.15.72:0/2548932711 shutdown_connections 2024-04-26T23:35:10.010 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:09.335+0000 7fdc9f400700 1 --2- 172.21.15.72:0/2548932711 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fdc80039700 0x7fdc8003bbc0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:35:10.010 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:09.335+0000 7fdc9f400700 1 --2- 172.21.15.72:0/2548932711 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc98108d70 0x7fdc9819eec0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:35:10.011 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:09.335+0000 7fdc9f400700 1 -- 172.21.15.72:0/2548932711 >> 172.21.15.72:0/2548932711 conn(0x7fdc981042d0 msgr2=0x7fdc98104d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:35:10.011 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:09.335+0000 7fdc9f400700 1 -- 172.21.15.72:0/2548932711 shutdown_connections 2024-04-26T23:35:10.011 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-04-26T23:35:09.335+0000 7fdc9f400700 1 -- 172.21.15.72:0/2548932711 wait complete. 2024-04-26T23:35:10.011 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-26T23:35:10.011 INFO:teuthology.orchestra.run.smithi072.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-26T23:35:10.011 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:35:10.012 INFO:teuthology.orchestra.run.smithi072.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-26T23:35:10.012 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:35:10.012 INFO:teuthology.orchestra.run.smithi072.stdout:Or, if you are only running a single cluster on this host: 2024-04-26T23:35:10.012 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:35:10.012 INFO:teuthology.orchestra.run.smithi072.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-26T23:35:10.013 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:35:10.013 INFO:teuthology.orchestra.run.smithi072.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-26T23:35:10.013 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:35:10.013 INFO:teuthology.orchestra.run.smithi072.stdout: ceph telemetry on 2024-04-26T23:35:10.013 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:35:10.013 INFO:teuthology.orchestra.run.smithi072.stdout:For more information see: 2024-04-26T23:35:10.014 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:35:10.014 INFO:teuthology.orchestra.run.smithi072.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-26T23:35:10.014 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:35:10.014 INFO:teuthology.orchestra.run.smithi072.stdout:Bootstrap complete. 2024-04-26T23:35:10.086 INFO:tasks.cephadm:Fetching config... 2024-04-26T23:35:10.086 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-26T23:35:10.086 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-26T23:35:10.133 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-26T23:35:10.133 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-26T23:35:10.133 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-26T23:35:10.216 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-26T23:35:10.216 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-26T23:35:10.216 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-26T23:35:10.333 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-26T23:35:10.333 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-26T23:35:10.334 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-26T23:35:10.428 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-26T23:35:10.428 DEBUG:teuthology.orchestra.run.smithi072:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQ8x+3x4cLVQ0JokUHrmSSlrt+iv2TrdIBotlq7h6PSXdQuM04AokPyvhQ0c+lcsrmeRA+epFiFOrD6Ms5mS/jJugqZFnFkakiHBdWXVO+zD/Zls8m/VPZF4MU9tFU4SJRG6KvxaMeO9/qncAUSiB2IIbOfM4TknJ0A73UU7AohshgENnurjWTif4PX2OVMqa1j0jkSsg1iupCZ7QKg/+6soF+uAHhFq9QqeM2G/k8Q4nzWmG0Woonkn561tB7rsAiJ4MyTvBUeKSdPtO+E+yFRFwQgPyZXjsAcDMLB7/vQA9JEmFCuyjl0OLefxmy+1KyTDSPvDOxNdFrTEHtz1/K9i661U/tujnDENADNZma+UCQGiB9OE9arzmlZWjb3EZkXTJn247mo2fKWPbiQcld+fUmvgi4PaYmIbHWwePlVOhFMeFlh9lN+4WIto15WVYstQqAsJ3ZB4s0pU2M61k873GMX5OO5sHEzUh+nWxZoO5Zflp/NBPYhIVa7ma+onM= ceph-3c8453ae-0425-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-26T23:35:10.536 INFO:teuthology.orchestra.run.smithi072.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQ8x+3x4cLVQ0JokUHrmSSlrt+iv2TrdIBotlq7h6PSXdQuM04AokPyvhQ0c+lcsrmeRA+epFiFOrD6Ms5mS/jJugqZFnFkakiHBdWXVO+zD/Zls8m/VPZF4MU9tFU4SJRG6KvxaMeO9/qncAUSiB2IIbOfM4TknJ0A73UU7AohshgENnurjWTif4PX2OVMqa1j0jkSsg1iupCZ7QKg/+6soF+uAHhFq9QqeM2G/k8Q4nzWmG0Woonkn561tB7rsAiJ4MyTvBUeKSdPtO+E+yFRFwQgPyZXjsAcDMLB7/vQA9JEmFCuyjl0OLefxmy+1KyTDSPvDOxNdFrTEHtz1/K9i661U/tujnDENADNZma+UCQGiB9OE9arzmlZWjb3EZkXTJn247mo2fKWPbiQcld+fUmvgi4PaYmIbHWwePlVOhFMeFlh9lN+4WIto15WVYstQqAsJ3ZB4s0pU2M61k873GMX5OO5sHEzUh+nWxZoO5Zflp/NBPYhIVa7ma+onM= ceph-3c8453ae-0425-11ef-bc93-c7b262605968 2024-04-26T23:35:10.571 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:10 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/2548932711' entity='client.admin' 2024-04-26T23:35:10.588 DEBUG:teuthology.orchestra.run.smithi076:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQ8x+3x4cLVQ0JokUHrmSSlrt+iv2TrdIBotlq7h6PSXdQuM04AokPyvhQ0c+lcsrmeRA+epFiFOrD6Ms5mS/jJugqZFnFkakiHBdWXVO+zD/Zls8m/VPZF4MU9tFU4SJRG6KvxaMeO9/qncAUSiB2IIbOfM4TknJ0A73UU7AohshgENnurjWTif4PX2OVMqa1j0jkSsg1iupCZ7QKg/+6soF+uAHhFq9QqeM2G/k8Q4nzWmG0Woonkn561tB7rsAiJ4MyTvBUeKSdPtO+E+yFRFwQgPyZXjsAcDMLB7/vQA9JEmFCuyjl0OLefxmy+1KyTDSPvDOxNdFrTEHtz1/K9i661U/tujnDENADNZma+UCQGiB9OE9arzmlZWjb3EZkXTJn247mo2fKWPbiQcld+fUmvgi4PaYmIbHWwePlVOhFMeFlh9lN+4WIto15WVYstQqAsJ3ZB4s0pU2M61k873GMX5OO5sHEzUh+nWxZoO5Zflp/NBPYhIVa7ma+onM= ceph-3c8453ae-0425-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-26T23:35:10.627 INFO:teuthology.orchestra.run.smithi076.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQ8x+3x4cLVQ0JokUHrmSSlrt+iv2TrdIBotlq7h6PSXdQuM04AokPyvhQ0c+lcsrmeRA+epFiFOrD6Ms5mS/jJugqZFnFkakiHBdWXVO+zD/Zls8m/VPZF4MU9tFU4SJRG6KvxaMeO9/qncAUSiB2IIbOfM4TknJ0A73UU7AohshgENnurjWTif4PX2OVMqa1j0jkSsg1iupCZ7QKg/+6soF+uAHhFq9QqeM2G/k8Q4nzWmG0Woonkn561tB7rsAiJ4MyTvBUeKSdPtO+E+yFRFwQgPyZXjsAcDMLB7/vQA9JEmFCuyjl0OLefxmy+1KyTDSPvDOxNdFrTEHtz1/K9i661U/tujnDENADNZma+UCQGiB9OE9arzmlZWjb3EZkXTJn247mo2fKWPbiQcld+fUmvgi4PaYmIbHWwePlVOhFMeFlh9lN+4WIto15WVYstQqAsJ3ZB4s0pU2M61k873GMX5OO5sHEzUh+nWxZoO5Zflp/NBPYhIVa7ma+onM= ceph-3c8453ae-0425-11ef-bc93-c7b262605968 2024-04-26T23:35:10.640 DEBUG:teuthology.orchestra.run.smithi136:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQ8x+3x4cLVQ0JokUHrmSSlrt+iv2TrdIBotlq7h6PSXdQuM04AokPyvhQ0c+lcsrmeRA+epFiFOrD6Ms5mS/jJugqZFnFkakiHBdWXVO+zD/Zls8m/VPZF4MU9tFU4SJRG6KvxaMeO9/qncAUSiB2IIbOfM4TknJ0A73UU7AohshgENnurjWTif4PX2OVMqa1j0jkSsg1iupCZ7QKg/+6soF+uAHhFq9QqeM2G/k8Q4nzWmG0Woonkn561tB7rsAiJ4MyTvBUeKSdPtO+E+yFRFwQgPyZXjsAcDMLB7/vQA9JEmFCuyjl0OLefxmy+1KyTDSPvDOxNdFrTEHtz1/K9i661U/tujnDENADNZma+UCQGiB9OE9arzmlZWjb3EZkXTJn247mo2fKWPbiQcld+fUmvgi4PaYmIbHWwePlVOhFMeFlh9lN+4WIto15WVYstQqAsJ3ZB4s0pU2M61k873GMX5OO5sHEzUh+nWxZoO5Zflp/NBPYhIVa7ma+onM= ceph-3c8453ae-0425-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-26T23:35:10.678 INFO:teuthology.orchestra.run.smithi136.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQ8x+3x4cLVQ0JokUHrmSSlrt+iv2TrdIBotlq7h6PSXdQuM04AokPyvhQ0c+lcsrmeRA+epFiFOrD6Ms5mS/jJugqZFnFkakiHBdWXVO+zD/Zls8m/VPZF4MU9tFU4SJRG6KvxaMeO9/qncAUSiB2IIbOfM4TknJ0A73UU7AohshgENnurjWTif4PX2OVMqa1j0jkSsg1iupCZ7QKg/+6soF+uAHhFq9QqeM2G/k8Q4nzWmG0Woonkn561tB7rsAiJ4MyTvBUeKSdPtO+E+yFRFwQgPyZXjsAcDMLB7/vQA9JEmFCuyjl0OLefxmy+1KyTDSPvDOxNdFrTEHtz1/K9i661U/tujnDENADNZma+UCQGiB9OE9arzmlZWjb3EZkXTJn247mo2fKWPbiQcld+fUmvgi4PaYmIbHWwePlVOhFMeFlh9lN+4WIto15WVYstQqAsJ3ZB4s0pU2M61k873GMX5OO5sHEzUh+nWxZoO5Zflp/NBPYhIVa7ma+onM= ceph-3c8453ae-0425-11ef-bc93-c7b262605968 2024-04-26T23:35:10.689 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-26T23:35:11.097 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:35:13.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:13.584+0000 7f79c7000700 1 -- 172.21.15.72:0/79284457 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79c01044e0 msgr2=0x7f79c0104900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:35:13.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:13.584+0000 7f79c7000700 1 --2- 172.21.15.72:0/79284457 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79c01044e0 0x7f79c0104900 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f79b0009910 tx=0x7f79b0009c20 comp rx=0 tx=0).stop 2024-04-26T23:35:13.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:13.584+0000 7f79c7000700 1 -- 172.21.15.72:0/79284457 shutdown_connections 2024-04-26T23:35:13.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:13.584+0000 7f79c7000700 1 --2- 172.21.15.72:0/79284457 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79c01044e0 0x7f79c0104900 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:35:13.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:13.584+0000 7f79c7000700 1 -- 172.21.15.72:0/79284457 >> 172.21.15.72:0/79284457 conn(0x7f79c00fe470 msgr2=0x7f79c0100890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:35:13.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:13.584+0000 7f79c7000700 1 -- 172.21.15.72:0/79284457 shutdown_connections 2024-04-26T23:35:13.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:13.584+0000 7f79c7000700 1 -- 172.21.15.72:0/79284457 wait complete. 2024-04-26T23:35:13.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:13.585+0000 7f79c7000700 1 Processor -- start 2024-04-26T23:35:13.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:13.585+0000 7f79c7000700 1 -- start start 2024-04-26T23:35:13.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:13.586+0000 7f79c7000700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79c01044e0 0x7f79c00770e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:35:13.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:13.586+0000 7f79c7000700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f79c0108e50 con 0x7f79c01044e0 2024-04-26T23:35:13.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:13.586+0000 7f79bfe00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79c01044e0 0x7f79c00770e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:35:13.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:13.586+0000 7f79bfe00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79c01044e0 0x7f79c00770e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36042/0 (socket says 172.21.15.72:36042) 2024-04-26T23:35:13.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:13.586+0000 7f79bfe00700 1 -- 172.21.15.72:0/1929549815 learned_addr learned my addr 172.21.15.72:0/1929549815 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:35:13.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:13.586+0000 7f79bfe00700 1 -- 172.21.15.72:0/1929549815 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79c0077620 con 0x7f79c01044e0 2024-04-26T23:35:13.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:13.587+0000 7f79bfe00700 1 --2- 172.21.15.72:0/1929549815 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79c01044e0 0x7f79c00770e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f79b0005950 tx=0x7f79b0004fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:35:13.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:13.587+0000 7f79bcc00700 1 -- 172.21.15.72:0/1929549815 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f79b0026070 con 0x7f79c01044e0 2024-04-26T23:35:13.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:13.587+0000 7f79bcc00700 1 -- 172.21.15.72:0/1929549815 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79b000bc00 con 0x7f79c01044e0 2024-04-26T23:35:13.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:13.587+0000 7f79bcc00700 1 -- 172.21.15.72:0/1929549815 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f79b0003a10 con 0x7f79c01044e0 2024-04-26T23:35:13.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:13.587+0000 7f79c7000700 1 -- 172.21.15.72:0/1929549815 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79c0075810 con 0x7f79c01044e0 2024-04-26T23:35:13.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:13.587+0000 7f79c7000700 1 -- 172.21.15.72:0/1929549815 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79c0075d50 con 0x7f79c01044e0 2024-04-26T23:35:13.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:13.589+0000 7f79bcc00700 1 -- 172.21.15.72:0/1929549815 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f79b0029070 con 0x7f79c01044e0 2024-04-26T23:35:13.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:13.589+0000 7f79c7000700 1 -- 172.21.15.72:0/1929549815 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79c01095d0 con 0x7f79c01044e0 2024-04-26T23:35:13.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:13.589+0000 7f79bcc00700 1 --2- 172.21.15.72:0/1929549815 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f79a40396b0 0x7f79a403bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:35:13.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:13.589+0000 7f79bcc00700 1 -- 172.21.15.72:0/1929549815 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f79b001e070 con 0x7f79c01044e0 2024-04-26T23:35:13.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:13.590+0000 7f79bf400700 1 --2- 172.21.15.72:0/1929549815 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f79a40396b0 0x7f79a403bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:35:13.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:13.590+0000 7f79bf400700 1 --2- 172.21.15.72:0/1929549815 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f79a40396b0 0x7f79a403bb70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f79ac006fd0 tx=0x7f79ac006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:35:13.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:13.595+0000 7f79bcc00700 1 -- 172.21.15.72:0/1929549815 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f79b0013070 con 0x7f79c01044e0 2024-04-26T23:35:13.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:13.820+0000 7f79c7000700 1 -- 172.21.15.72:0/1929549815 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f79c01917c0 con 0x7f79c01044e0 2024-04-26T23:35:13.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:13.826+0000 7f79bcc00700 1 -- 172.21.15.72:0/1929549815 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f79b00300c0 con 0x7f79c01044e0 2024-04-26T23:35:13.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:13.829+0000 7f79c7000700 1 -- 172.21.15.72:0/1929549815 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f79a40396b0 msgr2=0x7f79a403bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:35:13.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:13.829+0000 7f79c7000700 1 --2- 172.21.15.72:0/1929549815 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f79a40396b0 0x7f79a403bb70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f79ac006fd0 tx=0x7f79ac006e40 comp rx=0 tx=0).stop 2024-04-26T23:35:13.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:13.829+0000 7f79c7000700 1 -- 172.21.15.72:0/1929549815 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79c01044e0 msgr2=0x7f79c00770e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:35:13.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:13.829+0000 7f79c7000700 1 --2- 172.21.15.72:0/1929549815 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79c01044e0 0x7f79c00770e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f79b0005950 tx=0x7f79b0004fa0 comp rx=0 tx=0).stop 2024-04-26T23:35:13.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:13.830+0000 7f79c7000700 1 -- 172.21.15.72:0/1929549815 shutdown_connections 2024-04-26T23:35:13.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:13.830+0000 7f79c7000700 1 --2- 172.21.15.72:0/1929549815 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f79a40396b0 0x7f79a403bb70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:35:13.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:13.830+0000 7f79c7000700 1 --2- 172.21.15.72:0/1929549815 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79c01044e0 0x7f79c00770e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:35:13.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:13.830+0000 7f79c7000700 1 -- 172.21.15.72:0/1929549815 >> 172.21.15.72:0/1929549815 conn(0x7f79c00fe470 msgr2=0x7f79c00fef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:35:13.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:13.830+0000 7f79c7000700 1 -- 172.21.15.72:0/1929549815 shutdown_connections 2024-04-26T23:35:13.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:13.830+0000 7f79c7000700 1 -- 172.21.15.72:0/1929549815 wait complete. 2024-04-26T23:35:14.517 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:14 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:14.517 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:14 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:14.517 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:14 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi072", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:35:14.517 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:14 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:14.517 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:14 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:35:14.517 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:14 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:14.517 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:14 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/1929549815' entity='client.admin' 2024-04-26T23:35:14.517 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:14 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:35:14.518 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:14 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:14.583 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-26T23:35:14.583 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-26T23:35:15.007 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:35:17.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:17.006+0000 7f1795e00700 1 -- 172.21.15.72:0/2998192108 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1790103770 msgr2=0x7f1790105b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:35:17.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:17.006+0000 7f1795e00700 1 --2- 172.21.15.72:0/2998192108 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1790103770 0x7f1790105b60 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1784009910 tx=0x7f1784009c20 comp rx=0 tx=0).stop 2024-04-26T23:35:17.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:17.007+0000 7f1795e00700 1 -- 172.21.15.72:0/2998192108 shutdown_connections 2024-04-26T23:35:17.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:17.007+0000 7f1795e00700 1 --2- 172.21.15.72:0/2998192108 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1790103770 0x7f1790105b60 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:35:17.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:17.007+0000 7f1795e00700 1 -- 172.21.15.72:0/2998192108 >> 172.21.15.72:0/2998192108 conn(0x7f17900fd180 msgr2=0x7f17900ff5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:35:17.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:17.007+0000 7f1795e00700 1 -- 172.21.15.72:0/2998192108 shutdown_connections 2024-04-26T23:35:17.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:17.007+0000 7f1795e00700 1 -- 172.21.15.72:0/2998192108 wait complete. 2024-04-26T23:35:17.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:17.008+0000 7f1795e00700 1 Processor -- start 2024-04-26T23:35:17.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:17.008+0000 7f1795e00700 1 -- start start 2024-04-26T23:35:17.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:17.008+0000 7f1795e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1790103770 0x7f179019a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:35:17.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:17.008+0000 7f1795e00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f1790108bc0 con 0x7f1790103770 2024-04-26T23:35:17.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:17.009+0000 7f1794a00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1790103770 0x7f179019a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:35:17.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:17.009+0000 7f1794a00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1790103770 0x7f179019a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59790/0 (socket says 172.21.15.72:59790) 2024-04-26T23:35:17.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:17.009+0000 7f1794a00700 1 -- 172.21.15.72:0/931720205 learned_addr learned my addr 172.21.15.72:0/931720205 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:35:17.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:17.009+0000 7f1794a00700 1 -- 172.21.15.72:0/931720205 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f179019a9d0 con 0x7f1790103770 2024-04-26T23:35:17.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:17.009+0000 7f1794a00700 1 --2- 172.21.15.72:0/931720205 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1790103770 0x7f179019a490 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1784005190 tx=0x7f1784004f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:35:17.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:17.010+0000 7f178d600700 1 -- 172.21.15.72:0/931720205 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1784027070 con 0x7f1790103770 2024-04-26T23:35:17.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:17.010+0000 7f178d600700 1 -- 172.21.15.72:0/931720205 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1784022470 con 0x7f1790103770 2024-04-26T23:35:17.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:17.010+0000 7f1795e00700 1 -- 172.21.15.72:0/931720205 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f179019acc0 con 0x7f1790103770 2024-04-26T23:35:17.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:17.010+0000 7f178d600700 1 -- 172.21.15.72:0/931720205 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f178400f5f0 con 0x7f1790103770 2024-04-26T23:35:17.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:17.010+0000 7f1795e00700 1 -- 172.21.15.72:0/931720205 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f179019d950 con 0x7f1790103770 2024-04-26T23:35:17.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:17.011+0000 7f178d600700 1 -- 172.21.15.72:0/931720205 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f178402a070 con 0x7f1790103770 2024-04-26T23:35:17.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:17.011+0000 7f1795e00700 1 -- 172.21.15.72:0/931720205 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1790109340 con 0x7f1790103770 2024-04-26T23:35:17.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:17.012+0000 7f178d600700 1 --2- 172.21.15.72:0/931720205 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f17780396b0 0x7f177803bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:35:17.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:17.012+0000 7f178d600700 1 -- 172.21.15.72:0/931720205 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f178401e030 con 0x7f1790103770 2024-04-26T23:35:17.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:17.012+0000 7f178fe00700 1 --2- 172.21.15.72:0/931720205 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f17780396b0 0x7f177803bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:35:17.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:17.013+0000 7f178fe00700 1 --2- 172.21.15.72:0/931720205 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f17780396b0 0x7f177803bb70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f177c006fd0 tx=0x7f177c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:35:17.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:17.017+0000 7f178d600700 1 -- 172.21.15.72:0/931720205 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1784013070 con 0x7f1790103770 2024-04-26T23:35:17.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:17.250+0000 7f1795e00700 1 -- 172.21.15.72:0/931720205 --> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f179002e680 con 0x7f17780396b0 2024-04-26T23:35:17.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:17.257+0000 7f178d600700 1 -- 172.21.15.72:0/931720205 <== mgr.14152 v2:172.21.15.72:6800/4281177709 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f179002e680 con 0x7f17780396b0 2024-04-26T23:35:17.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:17.260+0000 7f1795e00700 1 -- 172.21.15.72:0/931720205 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f17780396b0 msgr2=0x7f177803bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:35:17.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:17.260+0000 7f1795e00700 1 --2- 172.21.15.72:0/931720205 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f17780396b0 0x7f177803bb70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f177c006fd0 tx=0x7f177c006e40 comp rx=0 tx=0).stop 2024-04-26T23:35:17.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:17.260+0000 7f1795e00700 1 -- 172.21.15.72:0/931720205 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1790103770 msgr2=0x7f179019a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:35:17.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:17.260+0000 7f1795e00700 1 --2- 172.21.15.72:0/931720205 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1790103770 0x7f179019a490 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1784005190 tx=0x7f1784004f80 comp rx=0 tx=0).stop 2024-04-26T23:35:17.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:17.260+0000 7f1795e00700 1 -- 172.21.15.72:0/931720205 shutdown_connections 2024-04-26T23:35:17.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:17.260+0000 7f1795e00700 1 --2- 172.21.15.72:0/931720205 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f17780396b0 0x7f177803bb70 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:35:17.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:17.260+0000 7f1795e00700 1 --2- 172.21.15.72:0/931720205 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1790103770 0x7f179019a490 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:35:17.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:17.260+0000 7f1795e00700 1 -- 172.21.15.72:0/931720205 >> 172.21.15.72:0/931720205 conn(0x7f17900fd180 msgr2=0x7f17900ff5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:35:17.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:17.261+0000 7f1795e00700 1 -- 172.21.15.72:0/931720205 shutdown_connections 2024-04-26T23:35:17.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:17.261+0000 7f1795e00700 1 -- 172.21.15.72:0/931720205 wait complete. 2024-04-26T23:35:18.051 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi076 2024-04-26T23:35:18.051 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-04-26T23:35:18.051 DEBUG:teuthology.orchestra.run.smithi076:> dd of=/etc/ceph/ceph.conf 2024-04-26T23:35:18.069 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-04-26T23:35:18.070 DEBUG:teuthology.orchestra.run.smithi076:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T23:35:18.126 INFO:tasks.cephadm:Adding host smithi076 to orchestrator... 2024-04-26T23:35:18.126 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph orch host add smithi076 2024-04-26T23:35:18.522 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:35:18.542 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:18 smithi072 ceph-mon[28184]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:35:18.542 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:18 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:18.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:18 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:35:18.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:18 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:35:18.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:18 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:35:18.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:18 smithi072 ceph-mon[28184]: Updating smithi072:/etc/ceph/ceph.conf 2024-04-26T23:35:19.542 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:19 smithi072 ceph-mon[28184]: Updating smithi072:/var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T23:35:19.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:19 smithi072 ceph-mon[28184]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:35:20.391 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:20 smithi072 ceph-mon[28184]: Updating smithi072:/etc/ceph/ceph.client.admin.keyring 2024-04-26T23:35:20.391 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:20 smithi072 ceph-mon[28184]: Updating smithi072:/var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T23:35:20.391 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:20 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:20.391 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:20 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:20.391 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:20 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:20.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:20.553+0000 7f606d000700 1 -- 172.21.15.72:0/2010027296 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6068103090 msgr2=0x7f6068103470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:35:20.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:20.553+0000 7f606d000700 1 --2- 172.21.15.72:0/2010027296 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6068103090 0x7f6068103470 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6058009910 tx=0x7f6058009c20 comp rx=0 tx=0).stop 2024-04-26T23:35:20.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:20.553+0000 7f606d000700 1 -- 172.21.15.72:0/2010027296 shutdown_connections 2024-04-26T23:35:20.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:20.553+0000 7f606d000700 1 --2- 172.21.15.72:0/2010027296 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6068103090 0x7f6068103470 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:35:20.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:20.553+0000 7f606d000700 1 -- 172.21.15.72:0/2010027296 >> 172.21.15.72:0/2010027296 conn(0x7f60680fe960 msgr2=0x7f6068100d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:35:20.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:20.553+0000 7f606d000700 1 -- 172.21.15.72:0/2010027296 shutdown_connections 2024-04-26T23:35:20.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:20.554+0000 7f606d000700 1 -- 172.21.15.72:0/2010027296 wait complete. 2024-04-26T23:35:20.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:20.555+0000 7f606d000700 1 Processor -- start 2024-04-26T23:35:20.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:20.555+0000 7f606d000700 1 -- start start 2024-04-26T23:35:20.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:20.556+0000 7f606d000700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6068103090 0x7f606819e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:35:20.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:20.556+0000 7f606d000700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f606810d310 con 0x7f6068103090 2024-04-26T23:35:20.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:20.556+0000 7f6065e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6068103090 0x7f606819e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:35:20.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:20.556+0000 7f6065e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6068103090 0x7f606819e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59806/0 (socket says 172.21.15.72:59806) 2024-04-26T23:35:20.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:20.556+0000 7f6065e00700 1 -- 172.21.15.72:0/2894969999 learned_addr learned my addr 172.21.15.72:0/2894969999 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:35:20.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:20.557+0000 7f6065e00700 1 -- 172.21.15.72:0/2894969999 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f606819eb00 con 0x7f6068103090 2024-04-26T23:35:20.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:20.557+0000 7f6065e00700 1 --2- 172.21.15.72:0/2894969999 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6068103090 0x7f606819e5c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f60580056d0 tx=0x7f60580057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:35:20.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:20.557+0000 7f605ea00700 1 -- 172.21.15.72:0/2894969999 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6058026070 con 0x7f6068103090 2024-04-26T23:35:20.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:20.557+0000 7f605ea00700 1 -- 172.21.15.72:0/2894969999 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f605800bba0 con 0x7f6068103090 2024-04-26T23:35:20.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:20.557+0000 7f605ea00700 1 -- 172.21.15.72:0/2894969999 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6058003920 con 0x7f6068103090 2024-04-26T23:35:20.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:20.557+0000 7f606d000700 1 -- 172.21.15.72:0/2894969999 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f606819edf0 con 0x7f6068103090 2024-04-26T23:35:20.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:20.557+0000 7f606d000700 1 -- 172.21.15.72:0/2894969999 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6068198860 con 0x7f6068103090 2024-04-26T23:35:20.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:20.558+0000 7f605ea00700 1 -- 172.21.15.72:0/2894969999 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f6058029070 con 0x7f6068103090 2024-04-26T23:35:20.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:20.559+0000 7f605ea00700 1 --2- 172.21.15.72:0/2894969999 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f604c0396b0 0x7f604c03bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:35:20.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:20.559+0000 7f606d000700 1 -- 172.21.15.72:0/2894969999 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6068104570 con 0x7f6068103090 2024-04-26T23:35:20.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:20.559+0000 7f605ea00700 1 -- 172.21.15.72:0/2894969999 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f605801e070 con 0x7f6068103090 2024-04-26T23:35:20.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:20.559+0000 7f6065400700 1 --2- 172.21.15.72:0/2894969999 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f604c0396b0 0x7f604c03bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:35:20.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:20.560+0000 7f6065400700 1 --2- 172.21.15.72:0/2894969999 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f604c0396b0 0x7f604c03bb70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6054006fd0 tx=0x7f6054006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:35:20.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:20.565+0000 7f605ea00700 1 -- 172.21.15.72:0/2894969999 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6058030020 con 0x7f6068103090 2024-04-26T23:35:20.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:20.815+0000 7f606d000700 1 -- 172.21.15.72:0/2894969999 --> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi076", "target": ["mon-mgr", ""]}) -- 0x7f6068199bf0 con 0x7f604c0396b0 2024-04-26T23:35:21.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:21 smithi072 ceph-mon[28184]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:35:22.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:22 smithi072 ceph-mon[28184]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi076", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:35:23.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:23 smithi072 ceph-mon[28184]: Deploying cephadm binary to smithi076 2024-04-26T23:35:23.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:23 smithi072 ceph-mon[28184]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:35:25.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:25.413+0000 7f605ea00700 1 -- 172.21.15.72:0/2894969999 <== mgr.14152 v2:172.21.15.72:6800/4281177709 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f6068199bf0 con 0x7f604c0396b0 2024-04-26T23:35:25.418 INFO:teuthology.orchestra.run.smithi072.stdout:Added host 'smithi076' with addr '172.21.15.76' 2024-04-26T23:35:25.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:25.416+0000 7f606d000700 1 -- 172.21.15.72:0/2894969999 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f604c0396b0 msgr2=0x7f604c03bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:35:25.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:25.416+0000 7f606d000700 1 --2- 172.21.15.72:0/2894969999 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f604c0396b0 0x7f604c03bb70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6054006fd0 tx=0x7f6054006e40 comp rx=0 tx=0).stop 2024-04-26T23:35:25.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:25.416+0000 7f606d000700 1 -- 172.21.15.72:0/2894969999 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6068103090 msgr2=0x7f606819e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:35:25.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:25.416+0000 7f606d000700 1 --2- 172.21.15.72:0/2894969999 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6068103090 0x7f606819e5c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f60580056d0 tx=0x7f60580057b0 comp rx=0 tx=0).stop 2024-04-26T23:35:25.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:25.416+0000 7f606d000700 1 -- 172.21.15.72:0/2894969999 shutdown_connections 2024-04-26T23:35:25.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:25.416+0000 7f606d000700 1 --2- 172.21.15.72:0/2894969999 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f604c0396b0 0x7f604c03bb70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:35:25.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:25.416+0000 7f606d000700 1 --2- 172.21.15.72:0/2894969999 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6068103090 0x7f606819e5c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:35:25.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:25.416+0000 7f606d000700 1 -- 172.21.15.72:0/2894969999 >> 172.21.15.72:0/2894969999 conn(0x7f60680fe960 msgr2=0x7f6068104a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:35:25.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:25.417+0000 7f606d000700 1 -- 172.21.15.72:0/2894969999 shutdown_connections 2024-04-26T23:35:25.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:25.417+0000 7f606d000700 1 -- 172.21.15.72:0/2894969999 wait complete. 2024-04-26T23:35:25.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:25 smithi072 ceph-mon[28184]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:35:26.215 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-26T23:35:26.657 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:35:26.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:26 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:26.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:26 smithi072 ceph-mon[28184]: Added host smithi076 2024-04-26T23:35:26.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:26 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:35:26.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:26 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:27.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:27 smithi072 ceph-mon[28184]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:35:27.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:27 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:28.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:28.699+0000 7f14c3600700 1 -- 172.21.15.72:0/2434389877 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14bc103090 msgr2=0x7f14bc103470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:35:28.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:28.699+0000 7f14c3600700 1 --2- 172.21.15.72:0/2434389877 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14bc103090 0x7f14bc103470 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f14ac009910 tx=0x7f14ac009c20 comp rx=0 tx=0).stop 2024-04-26T23:35:28.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:28.699+0000 7f14c3600700 1 -- 172.21.15.72:0/2434389877 shutdown_connections 2024-04-26T23:35:28.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:28.699+0000 7f14c3600700 1 --2- 172.21.15.72:0/2434389877 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14bc103090 0x7f14bc103470 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:35:28.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:28.699+0000 7f14c3600700 1 -- 172.21.15.72:0/2434389877 >> 172.21.15.72:0/2434389877 conn(0x7f14bc0fe960 msgr2=0x7f14bc100d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:35:28.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:28.699+0000 7f14c3600700 1 -- 172.21.15.72:0/2434389877 shutdown_connections 2024-04-26T23:35:28.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:28.699+0000 7f14c3600700 1 -- 172.21.15.72:0/2434389877 wait complete. 2024-04-26T23:35:28.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:28.701+0000 7f14c3600700 1 Processor -- start 2024-04-26T23:35:28.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:28.701+0000 7f14c3600700 1 -- start start 2024-04-26T23:35:28.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:28.701+0000 7f14c3600700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14bc103090 0x7f14bc075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:35:28.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:28.701+0000 7f14c3600700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f14bc10d310 con 0x7f14bc103090 2024-04-26T23:35:28.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:28.701+0000 7f14c0c00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14bc103090 0x7f14bc075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:35:28.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:28.701+0000 7f14c0c00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14bc103090 0x7f14bc075720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56290/0 (socket says 172.21.15.72:56290) 2024-04-26T23:35:28.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:28.701+0000 7f14c0c00700 1 -- 172.21.15.72:0/101476690 learned_addr learned my addr 172.21.15.72:0/101476690 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:35:28.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:28.702+0000 7f14c0c00700 1 -- 172.21.15.72:0/101476690 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14bc077610 con 0x7f14bc103090 2024-04-26T23:35:28.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:28.702+0000 7f14c0c00700 1 --2- 172.21.15.72:0/101476690 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14bc103090 0x7f14bc075720 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f14ac006010 tx=0x7f14ac005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:35:28.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:28.702+0000 7f14ba000700 1 -- 172.21.15.72:0/101476690 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f14ac026070 con 0x7f14bc103090 2024-04-26T23:35:28.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:28.702+0000 7f14ba000700 1 -- 172.21.15.72:0/101476690 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14ac00bc00 con 0x7f14bc103090 2024-04-26T23:35:28.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:28.702+0000 7f14ba000700 1 -- 172.21.15.72:0/101476690 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f14ac003910 con 0x7f14bc103090 2024-04-26T23:35:28.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:28.702+0000 7f14c3600700 1 -- 172.21.15.72:0/101476690 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14bc075cc0 con 0x7f14bc103090 2024-04-26T23:35:28.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:28.703+0000 7f14c3600700 1 -- 172.21.15.72:0/101476690 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14bc076180 con 0x7f14bc103090 2024-04-26T23:35:28.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:28.704+0000 7f14c3600700 1 -- 172.21.15.72:0/101476690 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14bc104570 con 0x7f14bc103090 2024-04-26T23:35:28.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:28.704+0000 7f14ba000700 1 -- 172.21.15.72:0/101476690 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f14ac029070 con 0x7f14bc103090 2024-04-26T23:35:28.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:28.704+0000 7f14ba000700 1 --2- 172.21.15.72:0/101476690 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f14a00396b0 0x7f14a003bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:35:28.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:28.704+0000 7f14ba000700 1 -- 172.21.15.72:0/101476690 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f14ac01e070 con 0x7f14bc103090 2024-04-26T23:35:28.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:28.704+0000 7f14b3e00700 1 --2- 172.21.15.72:0/101476690 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f14a00396b0 0x7f14a003bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:35:28.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:28.705+0000 7f14b3e00700 1 --2- 172.21.15.72:0/101476690 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f14a00396b0 0x7f14a003bb70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f14a4006fd0 tx=0x7f14a4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:35:28.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:28.710+0000 7f14ba000700 1 -- 172.21.15.72:0/101476690 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f14ac013070 con 0x7f14bc103090 2024-04-26T23:35:28.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:28.942+0000 7f14c3600700 1 -- 172.21.15.72:0/101476690 --> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f14bc076c50 con 0x7f14a00396b0 2024-04-26T23:35:28.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:28.944+0000 7f14ba000700 1 -- 172.21.15.72:0/101476690 <== mgr.14152 v2:172.21.15.72:6800/4281177709 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+159 (secure 0 0 0) 0x7f14bc076c50 con 0x7f14a00396b0 2024-04-26T23:35:28.948 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:35:28.948 INFO:teuthology.orchestra.run.smithi072.stdout:[{"addr": "172.21.15.72", "hostname": "smithi072", "labels": [], "status": ""}, {"addr": "172.21.15.76", "hostname": "smithi076", "labels": [], "status": ""}] 2024-04-26T23:35:28.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:28.947+0000 7f14c3600700 1 -- 172.21.15.72:0/101476690 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f14a00396b0 msgr2=0x7f14a003bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:35:28.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:28.947+0000 7f14c3600700 1 --2- 172.21.15.72:0/101476690 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f14a00396b0 0x7f14a003bb70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f14a4006fd0 tx=0x7f14a4006e40 comp rx=0 tx=0).stop 2024-04-26T23:35:28.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:28.947+0000 7f14c3600700 1 -- 172.21.15.72:0/101476690 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14bc103090 msgr2=0x7f14bc075720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:35:28.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:28.947+0000 7f14c3600700 1 --2- 172.21.15.72:0/101476690 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14bc103090 0x7f14bc075720 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f14ac006010 tx=0x7f14ac005780 comp rx=0 tx=0).stop 2024-04-26T23:35:28.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:28.947+0000 7f14c3600700 1 -- 172.21.15.72:0/101476690 shutdown_connections 2024-04-26T23:35:28.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:28.947+0000 7f14c3600700 1 --2- 172.21.15.72:0/101476690 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f14a00396b0 0x7f14a003bb70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:35:28.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:28.947+0000 7f14c3600700 1 --2- 172.21.15.72:0/101476690 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14bc103090 0x7f14bc075720 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:35:28.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:28.947+0000 7f14c3600700 1 -- 172.21.15.72:0/101476690 >> 172.21.15.72:0/101476690 conn(0x7f14bc0fe960 msgr2=0x7f14bc104a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:35:28.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:28.947+0000 7f14c3600700 1 -- 172.21.15.72:0/101476690 shutdown_connections 2024-04-26T23:35:28.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:28.947+0000 7f14c3600700 1 -- 172.21.15.72:0/101476690 wait complete. 2024-04-26T23:35:29.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:29 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:29.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:29 smithi072 ceph-mon[28184]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:35:29.710 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi136 2024-04-26T23:35:29.711 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-26T23:35:29.711 DEBUG:teuthology.orchestra.run.smithi136:> dd of=/etc/ceph/ceph.conf 2024-04-26T23:35:29.730 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-26T23:35:29.730 DEBUG:teuthology.orchestra.run.smithi136:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T23:35:29.787 INFO:tasks.cephadm:Adding host smithi136 to orchestrator... 2024-04-26T23:35:29.787 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph orch host add smithi136 2024-04-26T23:35:30.212 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:35:30.505 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:30 smithi072 ceph-mon[28184]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T23:35:31.463 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:31 smithi072 ceph-mon[28184]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:35:32.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:32.315+0000 7f4a5da00700 1 -- 172.21.15.72:0/4050381016 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a5810e3d0 msgr2=0x7f4a5810e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:35:32.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:32.315+0000 7f4a5da00700 1 --2- 172.21.15.72:0/4050381016 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a5810e3d0 0x7f4a5810e7b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4a48009910 tx=0x7f4a48009c20 comp rx=0 tx=0).stop 2024-04-26T23:35:32.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:32.315+0000 7f4a5da00700 1 -- 172.21.15.72:0/4050381016 shutdown_connections 2024-04-26T23:35:32.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:32.316+0000 7f4a5da00700 1 --2- 172.21.15.72:0/4050381016 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a5810e3d0 0x7f4a5810e7b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:35:32.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:32.316+0000 7f4a5da00700 1 -- 172.21.15.72:0/4050381016 >> 172.21.15.72:0/4050381016 conn(0x7f4a58078bc0 msgr2=0x7f4a5807afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:35:32.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:32.316+0000 7f4a5da00700 1 -- 172.21.15.72:0/4050381016 shutdown_connections 2024-04-26T23:35:32.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:32.316+0000 7f4a5da00700 1 -- 172.21.15.72:0/4050381016 wait complete. 2024-04-26T23:35:32.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:32.317+0000 7f4a5da00700 1 Processor -- start 2024-04-26T23:35:32.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:32.317+0000 7f4a5da00700 1 -- start start 2024-04-26T23:35:32.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:32.317+0000 7f4a5da00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a5810e3d0 0x7f4a5819eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:35:32.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:32.317+0000 7f4a5da00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f4a58113940 con 0x7f4a5810e3d0 2024-04-26T23:35:32.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:32.318+0000 7f4a56a00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a5810e3d0 0x7f4a5819eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:35:32.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:32.318+0000 7f4a56a00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a5810e3d0 0x7f4a5819eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56304/0 (socket says 172.21.15.72:56304) 2024-04-26T23:35:32.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:32.318+0000 7f4a56a00700 1 -- 172.21.15.72:0/3355407676 learned_addr learned my addr 172.21.15.72:0/3355407676 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:35:32.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:32.318+0000 7f4a56a00700 1 -- 172.21.15.72:0/3355407676 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a5819f420 con 0x7f4a5810e3d0 2024-04-26T23:35:32.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:32.318+0000 7f4a56a00700 1 --2- 172.21.15.72:0/3355407676 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a5810e3d0 0x7f4a5819eee0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4a480056d0 tx=0x7f4a480057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:35:32.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:32.319+0000 7f4a4f400700 1 -- 172.21.15.72:0/3355407676 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a48026070 con 0x7f4a5810e3d0 2024-04-26T23:35:32.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:32.319+0000 7f4a4f400700 1 -- 172.21.15.72:0/3355407676 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a48004370 con 0x7f4a5810e3d0 2024-04-26T23:35:32.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:32.319+0000 7f4a5da00700 1 -- 172.21.15.72:0/3355407676 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a5819f710 con 0x7f4a5810e3d0 2024-04-26T23:35:32.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:32.319+0000 7f4a4f400700 1 -- 172.21.15.72:0/3355407676 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a480039b0 con 0x7f4a5810e3d0 2024-04-26T23:35:32.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:32.319+0000 7f4a5da00700 1 -- 172.21.15.72:0/3355407676 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a581a37b0 con 0x7f4a5810e3d0 2024-04-26T23:35:32.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:32.320+0000 7f4a4f400700 1 -- 172.21.15.72:0/3355407676 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f4a48029070 con 0x7f4a5810e3d0 2024-04-26T23:35:32.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:32.320+0000 7f4a5da00700 1 -- 172.21.15.72:0/3355407676 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a5810ab60 con 0x7f4a5810e3d0 2024-04-26T23:35:32.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:32.320+0000 7f4a4f400700 1 --2- 172.21.15.72:0/3355407676 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f4a3c0396b0 0x7f4a3c03bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:35:32.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:32.320+0000 7f4a4f400700 1 -- 172.21.15.72:0/3355407676 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f4a4801e070 con 0x7f4a5810e3d0 2024-04-26T23:35:32.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:32.321+0000 7f4a56000700 1 --2- 172.21.15.72:0/3355407676 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f4a3c0396b0 0x7f4a3c03bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:35:32.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:32.322+0000 7f4a56000700 1 --2- 172.21.15.72:0/3355407676 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f4a3c0396b0 0x7f4a3c03bb70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4a44006fd0 tx=0x7f4a44006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:35:32.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:32.326+0000 7f4a4f400700 1 -- 172.21.15.72:0/3355407676 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4a48013070 con 0x7f4a5810e3d0 2024-04-26T23:35:32.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:32.560+0000 7f4a5da00700 1 -- 172.21.15.72:0/3355407676 --> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi136", "target": ["mon-mgr", ""]}) -- 0x7f4a581a0240 con 0x7f4a3c0396b0 2024-04-26T23:35:32.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:32 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:32.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:32 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:32.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:32 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:32.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:32 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:32.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:32 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi076", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:35:32.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:32 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:35:32.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:32 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:35:32.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:32 smithi072 ceph-mon[28184]: Updating smithi076:/etc/ceph/ceph.conf 2024-04-26T23:35:33.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:33 smithi072 ceph-mon[28184]: Updating smithi076:/var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T23:35:33.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:33 smithi072 ceph-mon[28184]: Updating smithi076:/etc/ceph/ceph.client.admin.keyring 2024-04-26T23:35:33.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:33 smithi072 ceph-mon[28184]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:35:33.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:33 smithi072 ceph-mon[28184]: Updating smithi076:/var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T23:35:33.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:33 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:33.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:33 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:33.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:33 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:34.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:34 smithi072 ceph-mon[28184]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi136", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:35:34.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:34 smithi072 ceph-mon[28184]: Deploying cephadm binary to smithi136 2024-04-26T23:35:35.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:35 smithi072 ceph-mon[28184]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:35:37.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:37.261+0000 7f4a4f400700 1 -- 172.21.15.72:0/3355407676 <== mgr.14152 v2:172.21.15.72:6800/4281177709 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f4a581a0240 con 0x7f4a3c0396b0 2024-04-26T23:35:37.265 INFO:teuthology.orchestra.run.smithi072.stdout:Added host 'smithi136' with addr '172.21.15.136' 2024-04-26T23:35:37.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:37.265+0000 7f4a5da00700 1 -- 172.21.15.72:0/3355407676 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f4a3c0396b0 msgr2=0x7f4a3c03bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:35:37.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:37.265+0000 7f4a5da00700 1 --2- 172.21.15.72:0/3355407676 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f4a3c0396b0 0x7f4a3c03bb70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4a44006fd0 tx=0x7f4a44006e40 comp rx=0 tx=0).stop 2024-04-26T23:35:37.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:37.265+0000 7f4a5da00700 1 -- 172.21.15.72:0/3355407676 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a5810e3d0 msgr2=0x7f4a5819eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:35:37.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:37.265+0000 7f4a5da00700 1 --2- 172.21.15.72:0/3355407676 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a5810e3d0 0x7f4a5819eee0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4a480056d0 tx=0x7f4a480057b0 comp rx=0 tx=0).stop 2024-04-26T23:35:37.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:37.265+0000 7f4a5da00700 1 -- 172.21.15.72:0/3355407676 shutdown_connections 2024-04-26T23:35:37.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:37.265+0000 7f4a5da00700 1 --2- 172.21.15.72:0/3355407676 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f4a3c0396b0 0x7f4a3c03bb70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:35:37.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:37.265+0000 7f4a5da00700 1 --2- 172.21.15.72:0/3355407676 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a5810e3d0 0x7f4a5819eee0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:35:37.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:37.265+0000 7f4a5da00700 1 -- 172.21.15.72:0/3355407676 >> 172.21.15.72:0/3355407676 conn(0x7f4a58078bc0 msgr2=0x7f4a58110f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:35:37.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:37.265+0000 7f4a5da00700 1 -- 172.21.15.72:0/3355407676 shutdown_connections 2024-04-26T23:35:37.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:37.265+0000 7f4a5da00700 1 -- 172.21.15.72:0/3355407676 wait complete. 2024-04-26T23:35:37.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:37 smithi072 ceph-mon[28184]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:35:37.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:37 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:37.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:37 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:35:38.028 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-26T23:35:38.456 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:35:38.744 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:38 smithi072 ceph-mon[28184]: Added host smithi136 2024-04-26T23:35:38.744 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:38 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:38.744 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:38 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:39.618 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:39 smithi072 ceph-mon[28184]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:35:40.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:40.432+0000 7f957c800700 1 -- 172.21.15.72:0/2459391409 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f957410e3d0 msgr2=0x7f957410e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:35:40.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:40.432+0000 7f957c800700 1 --2- 172.21.15.72:0/2459391409 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f957410e3d0 0x7f957410e7b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9568009910 tx=0x7f9568009c20 comp rx=0 tx=0).stop 2024-04-26T23:35:40.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:40.432+0000 7f957c800700 1 -- 172.21.15.72:0/2459391409 shutdown_connections 2024-04-26T23:35:40.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:40.432+0000 7f957c800700 1 --2- 172.21.15.72:0/2459391409 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f957410e3d0 0x7f957410e7b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:35:40.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:40.432+0000 7f957c800700 1 -- 172.21.15.72:0/2459391409 >> 172.21.15.72:0/2459391409 conn(0x7f9574078bc0 msgr2=0x7f957407afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:35:40.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:40.433+0000 7f957c800700 1 -- 172.21.15.72:0/2459391409 shutdown_connections 2024-04-26T23:35:40.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:40.433+0000 7f957c800700 1 -- 172.21.15.72:0/2459391409 wait complete. 2024-04-26T23:35:40.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:40.434+0000 7f957c800700 1 Processor -- start 2024-04-26T23:35:40.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:40.434+0000 7f957c800700 1 -- start start 2024-04-26T23:35:40.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:40.434+0000 7f957c800700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f957410e3d0 0x7f957419ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:35:40.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:40.434+0000 7f957c800700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f9574113940 con 0x7f957410e3d0 2024-04-26T23:35:40.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:40.434+0000 7f9579e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f957410e3d0 0x7f957419ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:35:40.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:40.434+0000 7f9579e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f957410e3d0 0x7f957419ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33924/0 (socket says 172.21.15.72:33924) 2024-04-26T23:35:40.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:40.434+0000 7f9579e00700 1 -- 172.21.15.72:0/3199602680 learned_addr learned my addr 172.21.15.72:0/3199602680 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:35:40.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:40.435+0000 7f9579e00700 1 -- 172.21.15.72:0/3199602680 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f957419f3b0 con 0x7f957410e3d0 2024-04-26T23:35:40.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:40.435+0000 7f9579e00700 1 --2- 172.21.15.72:0/3199602680 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f957410e3d0 0x7f957419ee70 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9568006010 tx=0x7f9568005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:35:40.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:40.435+0000 7f956ea00700 1 -- 172.21.15.72:0/3199602680 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9568026070 con 0x7f957410e3d0 2024-04-26T23:35:40.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:40.435+0000 7f956ea00700 1 -- 172.21.15.72:0/3199602680 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f956800bba0 con 0x7f957410e3d0 2024-04-26T23:35:40.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:40.435+0000 7f957c800700 1 -- 172.21.15.72:0/3199602680 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f957419f6a0 con 0x7f957410e3d0 2024-04-26T23:35:40.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:40.435+0000 7f956ea00700 1 -- 172.21.15.72:0/3199602680 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9568003930 con 0x7f957410e3d0 2024-04-26T23:35:40.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:40.436+0000 7f957c800700 1 -- 172.21.15.72:0/3199602680 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95741a3740 con 0x7f957410e3d0 2024-04-26T23:35:40.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:40.437+0000 7f956ea00700 1 -- 172.21.15.72:0/3199602680 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f9568029070 con 0x7f957410e3d0 2024-04-26T23:35:40.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:40.437+0000 7f957c800700 1 -- 172.21.15.72:0/3199602680 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f957410ab60 con 0x7f957410e3d0 2024-04-26T23:35:40.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:40.437+0000 7f956ea00700 1 --2- 172.21.15.72:0/3199602680 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f955c0396b0 0x7f955c03bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:35:40.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:40.437+0000 7f956ea00700 1 -- 172.21.15.72:0/3199602680 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f956801e070 con 0x7f957410e3d0 2024-04-26T23:35:40.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:40.438+0000 7f9579400700 1 --2- 172.21.15.72:0/3199602680 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f955c0396b0 0x7f955c03bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:35:40.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:40.438+0000 7f9579400700 1 --2- 172.21.15.72:0/3199602680 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f955c0396b0 0x7f955c03bb70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9564006fd0 tx=0x7f9564006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:35:40.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:40.443+0000 7f956ea00700 1 -- 172.21.15.72:0/3199602680 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9568013070 con 0x7f957410e3d0 2024-04-26T23:35:40.682 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:40.678+0000 7f957c800700 1 -- 172.21.15.72:0/3199602680 --> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f95741a01d0 con 0x7f955c0396b0 2024-04-26T23:35:40.683 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:40.679+0000 7f956ea00700 1 -- 172.21.15.72:0/3199602680 <== mgr.14152 v2:172.21.15.72:6800/4281177709 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+239 (secure 0 0 0) 0x7f95741a01d0 con 0x7f955c0396b0 2024-04-26T23:35:40.683 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:35:40.683 INFO:teuthology.orchestra.run.smithi072.stdout:[{"addr": "172.21.15.72", "hostname": "smithi072", "labels": [], "status": ""}, {"addr": "172.21.15.76", "hostname": "smithi076", "labels": [], "status": ""}, {"addr": "172.21.15.136", "hostname": "smithi136", "labels": [], "status": ""}] 2024-04-26T23:35:40.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:40.683+0000 7f957c800700 1 -- 172.21.15.72:0/3199602680 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f955c0396b0 msgr2=0x7f955c03bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:35:40.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:40.683+0000 7f957c800700 1 --2- 172.21.15.72:0/3199602680 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f955c0396b0 0x7f955c03bb70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9564006fd0 tx=0x7f9564006e40 comp rx=0 tx=0).stop 2024-04-26T23:35:40.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:40.683+0000 7f957c800700 1 -- 172.21.15.72:0/3199602680 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f957410e3d0 msgr2=0x7f957419ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:35:40.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:40.683+0000 7f957c800700 1 --2- 172.21.15.72:0/3199602680 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f957410e3d0 0x7f957419ee70 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9568006010 tx=0x7f9568005780 comp rx=0 tx=0).stop 2024-04-26T23:35:40.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:40.683+0000 7f957c800700 1 -- 172.21.15.72:0/3199602680 shutdown_connections 2024-04-26T23:35:40.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:40.683+0000 7f957c800700 1 --2- 172.21.15.72:0/3199602680 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f955c0396b0 0x7f955c03bb70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:35:40.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:40.683+0000 7f957c800700 1 --2- 172.21.15.72:0/3199602680 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f957410e3d0 0x7f957419ee70 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:35:40.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:40.683+0000 7f957c800700 1 -- 172.21.15.72:0/3199602680 >> 172.21.15.72:0/3199602680 conn(0x7f9574078bc0 msgr2=0x7f9574110f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:35:40.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:40.683+0000 7f957c800700 1 -- 172.21.15.72:0/3199602680 shutdown_connections 2024-04-26T23:35:40.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:40.683+0000 7f957c800700 1 -- 172.21.15.72:0/3199602680 wait complete. 2024-04-26T23:35:41.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:41 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:41.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:41 smithi072 ceph-mon[28184]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:35:41.376 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-26T23:35:41.376 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-26T23:35:41.805 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:35:42.232 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:42 smithi072 ceph-mon[28184]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T23:35:43.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:43 smithi072 ceph-mon[28184]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:35:43.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:43.833+0000 7fb967e00700 1 -- 172.21.15.72:0/3519998599 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9680757f0 msgr2=0x7fb968075bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:35:43.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:43.833+0000 7fb967e00700 1 --2- 172.21.15.72:0/3519998599 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9680757f0 0x7fb968075bd0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb958009910 tx=0x7fb958009c20 comp rx=0 tx=0).stop 2024-04-26T23:35:43.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:43.833+0000 7fb967e00700 1 -- 172.21.15.72:0/3519998599 shutdown_connections 2024-04-26T23:35:43.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:43.833+0000 7fb967e00700 1 --2- 172.21.15.72:0/3519998599 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9680757f0 0x7fb968075bd0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:35:43.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:43.833+0000 7fb967e00700 1 -- 172.21.15.72:0/3519998599 >> 172.21.15.72:0/3519998599 conn(0x7fb9680fe7c0 msgr2=0x7fb968100be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:35:43.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:43.833+0000 7fb967e00700 1 -- 172.21.15.72:0/3519998599 shutdown_connections 2024-04-26T23:35:43.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:43.833+0000 7fb967e00700 1 -- 172.21.15.72:0/3519998599 wait complete. 2024-04-26T23:35:43.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:43.834+0000 7fb967e00700 1 Processor -- start 2024-04-26T23:35:43.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:43.834+0000 7fb967e00700 1 -- start start 2024-04-26T23:35:43.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:43.835+0000 7fb967e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9680757f0 0x7fb9681a0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:35:43.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:43.835+0000 7fb967e00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fb96810f3e0 con 0x7fb9680757f0 2024-04-26T23:35:43.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:43.835+0000 7fb966a00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9680757f0 0x7fb9681a0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:35:43.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:43.835+0000 7fb966a00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9680757f0 0x7fb9681a0600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33938/0 (socket says 172.21.15.72:33938) 2024-04-26T23:35:43.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:43.835+0000 7fb966a00700 1 -- 172.21.15.72:0/931525612 learned_addr learned my addr 172.21.15.72:0/931525612 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:35:43.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:43.835+0000 7fb966a00700 1 -- 172.21.15.72:0/931525612 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9681a0b40 con 0x7fb9680757f0 2024-04-26T23:35:43.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:43.836+0000 7fb966a00700 1 --2- 172.21.15.72:0/931525612 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9680757f0 0x7fb9681a0600 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb9580056d0 tx=0x7fb9580057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:35:43.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:43.836+0000 7fb96ca00700 1 -- 172.21.15.72:0/931525612 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb958026070 con 0x7fb9680757f0 2024-04-26T23:35:43.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:43.836+0000 7fb96ca00700 1 -- 172.21.15.72:0/931525612 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb958004370 con 0x7fb9680757f0 2024-04-26T23:35:43.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:43.836+0000 7fb96ca00700 1 -- 172.21.15.72:0/931525612 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb9580039b0 con 0x7fb9680757f0 2024-04-26T23:35:43.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:43.836+0000 7fb967e00700 1 -- 172.21.15.72:0/931525612 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9681a0dd0 con 0x7fb9680757f0 2024-04-26T23:35:43.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:43.836+0000 7fb967e00700 1 -- 172.21.15.72:0/931525612 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb96819a780 con 0x7fb9680757f0 2024-04-26T23:35:43.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:43.837+0000 7fb96ca00700 1 -- 172.21.15.72:0/931525612 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fb958029070 con 0x7fb9680757f0 2024-04-26T23:35:43.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:43.838+0000 7fb967e00700 1 -- 172.21.15.72:0/931525612 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb968076ea0 con 0x7fb9680757f0 2024-04-26T23:35:43.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:43.838+0000 7fb96ca00700 1 --2- 172.21.15.72:0/931525612 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fb94c039650 0x7fb94c03bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:35:43.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:43.838+0000 7fb96ca00700 1 -- 172.21.15.72:0/931525612 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb95801e070 con 0x7fb9680757f0 2024-04-26T23:35:43.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:43.838+0000 7fb966000700 1 --2- 172.21.15.72:0/931525612 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fb94c039650 0x7fb94c03bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:35:43.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:43.839+0000 7fb966000700 1 --2- 172.21.15.72:0/931525612 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fb94c039650 0x7fb94c03bb10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb954006fd0 tx=0x7fb954006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:35:43.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:43.844+0000 7fb96ca00700 1 -- 172.21.15.72:0/931525612 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb958013070 con 0x7fb9680757f0 2024-04-26T23:35:44.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:44.117+0000 7fb967e00700 1 -- 172.21.15.72:0/931525612 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fb968064a80 con 0x7fb9680757f0 2024-04-26T23:35:44.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:44.397+0000 7fb96ca00700 1 -- 172.21.15.72:0/931525612 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7fb9580282c0 con 0x7fb9680757f0 2024-04-26T23:35:44.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:44.400+0000 7fb967e00700 1 -- 172.21.15.72:0/931525612 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fb94c039650 msgr2=0x7fb94c03bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:35:44.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:44.400+0000 7fb967e00700 1 --2- 172.21.15.72:0/931525612 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fb94c039650 0x7fb94c03bb10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb954006fd0 tx=0x7fb954006e40 comp rx=0 tx=0).stop 2024-04-26T23:35:44.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:44.400+0000 7fb967e00700 1 -- 172.21.15.72:0/931525612 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9680757f0 msgr2=0x7fb9681a0600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:35:44.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:44.400+0000 7fb967e00700 1 --2- 172.21.15.72:0/931525612 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9680757f0 0x7fb9681a0600 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb9580056d0 tx=0x7fb9580057b0 comp rx=0 tx=0).stop 2024-04-26T23:35:44.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:44.401+0000 7fb967e00700 1 -- 172.21.15.72:0/931525612 shutdown_connections 2024-04-26T23:35:44.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:44.401+0000 7fb967e00700 1 --2- 172.21.15.72:0/931525612 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fb94c039650 0x7fb94c03bb10 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:35:44.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:44.401+0000 7fb967e00700 1 --2- 172.21.15.72:0/931525612 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9680757f0 0x7fb9681a0600 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:35:44.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:44.401+0000 7fb967e00700 1 -- 172.21.15.72:0/931525612 >> 172.21.15.72:0/931525612 conn(0x7fb9680fe7c0 msgr2=0x7fb968106f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:35:44.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:44.401+0000 7fb967e00700 1 -- 172.21.15.72:0/931525612 shutdown_connections 2024-04-26T23:35:44.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:35:44.402+0000 7fb967e00700 1 -- 172.21.15.72:0/931525612 wait complete. 2024-04-26T23:35:44.406 INFO:teuthology.orchestra.run.smithi072.stderr:adjusted tunables profile to default 2024-04-26T23:35:44.764 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:44 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:44.764 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:44 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:44.764 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:44 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:44.764 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:44 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:44.764 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:44 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi136", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:35:44.764 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:44 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:35:44.764 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:44 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:35:44.764 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:44 smithi072 ceph-mon[28184]: Updating smithi136:/etc/ceph/ceph.conf 2024-04-26T23:35:44.764 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:44 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/931525612' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-26T23:35:45.147 INFO:tasks.cephadm:Adding mon.a on smithi072 2024-04-26T23:35:45.147 INFO:tasks.cephadm:Adding mon.b on smithi076 2024-04-26T23:35:45.147 INFO:tasks.cephadm:Adding mon.c on smithi136 2024-04-26T23:35:45.147 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi072:172.21.15.72=a;smithi076:172.21.15.76=b;smithi136:172.21.15.136=c' 2024-04-26T23:35:45.372 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T23:35:45.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:45 smithi072 ceph-mon[28184]: Updating smithi136:/var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T23:35:45.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:45 smithi072 ceph-mon[28184]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2024-04-26T23:35:45.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:45 smithi072 ceph-mon[28184]: Updating smithi136:/var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T23:35:45.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:45 smithi072 ceph-mon[28184]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:35:45.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:45 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/931525612' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-26T23:35:45.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:45 smithi072 ceph-mon[28184]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T23:35:45.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:45 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:45.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:45 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:45.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:45 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:46.706 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:46.702+0000 7efed7e88700 1 -- 172.21.15.136:0/726068655 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efed0103090 msgr2=0x7efed0103470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:35:46.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:46.702+0000 7efed7e88700 1 --2- 172.21.15.136:0/726068655 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efed0103090 0x7efed0103470 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7efecc009910 tx=0x7efecc009c20 comp rx=0 tx=0).stop 2024-04-26T23:35:46.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:46.702+0000 7efed7e88700 1 -- 172.21.15.136:0/726068655 shutdown_connections 2024-04-26T23:35:46.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:46.702+0000 7efed7e88700 1 --2- 172.21.15.136:0/726068655 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efed0103090 0x7efed0103470 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:35:46.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:46.702+0000 7efed7e88700 1 -- 172.21.15.136:0/726068655 >> 172.21.15.136:0/726068655 conn(0x7efed00fe960 msgr2=0x7efed0100d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:35:46.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:46.702+0000 7efed7e88700 1 -- 172.21.15.136:0/726068655 shutdown_connections 2024-04-26T23:35:46.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:46.703+0000 7efed7e88700 1 -- 172.21.15.136:0/726068655 wait complete. 2024-04-26T23:35:46.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:46.703+0000 7efed7e88700 1 Processor -- start 2024-04-26T23:35:46.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:46.703+0000 7efed7e88700 1 -- start start 2024-04-26T23:35:46.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:46.703+0000 7efed7e88700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efed0103090 0x7efed019e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:35:46.707 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:46.703+0000 7efed7e88700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7efed010d310 con 0x7efed0103090 2024-04-26T23:35:46.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:46.704+0000 7efed5c24700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efed0103090 0x7efed019e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:35:46.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:46.704+0000 7efed5c24700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efed0103090 0x7efed019e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.136:33458/0 (socket says 172.21.15.136:33458) 2024-04-26T23:35:46.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:46.704+0000 7efed5c24700 1 -- 172.21.15.136:0/470219067 learned_addr learned my addr 172.21.15.136:0/470219067 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-26T23:35:46.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:46.705+0000 7efed5c24700 1 -- 172.21.15.136:0/470219067 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efed019eaf0 con 0x7efed0103090 2024-04-26T23:35:46.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:46.705+0000 7efed5c24700 1 --2- 172.21.15.136:0/470219067 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efed0103090 0x7efed019e5b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7efecc006010 tx=0x7efecc005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:35:46.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:46.705+0000 7efec6ffd700 1 -- 172.21.15.136:0/470219067 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efecc026070 con 0x7efed0103090 2024-04-26T23:35:46.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:46.705+0000 7efed7e88700 1 -- 172.21.15.136:0/470219067 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efed019ede0 con 0x7efed0103090 2024-04-26T23:35:46.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:46.705+0000 7efed7e88700 1 -- 172.21.15.136:0/470219067 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efed0198850 con 0x7efed0103090 2024-04-26T23:35:46.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:46.705+0000 7efec6ffd700 1 -- 172.21.15.136:0/470219067 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efecc00bba0 con 0x7efed0103090 2024-04-26T23:35:46.709 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:46.705+0000 7efec6ffd700 1 -- 172.21.15.136:0/470219067 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efecc030060 con 0x7efed0103090 2024-04-26T23:35:46.710 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:46.706+0000 7efed7e88700 1 -- 172.21.15.136:0/470219067 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efed0104570 con 0x7efed0103090 2024-04-26T23:35:46.710 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:46.707+0000 7efec6ffd700 1 -- 172.21.15.136:0/470219067 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7efecc029070 con 0x7efed0103090 2024-04-26T23:35:46.710 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:46.707+0000 7efec6ffd700 1 --2- 172.21.15.136:0/470219067 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7efebc039650 0x7efebc03bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:35:46.710 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:46.707+0000 7efec6ffd700 1 -- 172.21.15.136:0/470219067 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7efecc01e070 con 0x7efed0103090 2024-04-26T23:35:46.711 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:46.708+0000 7efed5423700 1 --2- 172.21.15.136:0/470219067 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7efebc039650 0x7efebc03bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:35:46.711 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:46.708+0000 7efed5423700 1 --2- 172.21.15.136:0/470219067 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7efebc039650 0x7efebc03bb10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7efec0006fd0 tx=0x7efec0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:35:46.724 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:46.721+0000 7efec6ffd700 1 -- 172.21.15.136:0/470219067 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7efecc013070 con 0x7efed0103090 2024-04-26T23:35:46.934 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:46.931+0000 7efed7e88700 1 -- 172.21.15.136:0/470219067 --> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi072:172.21.15.72=a;smithi076:172.21.15.76=b;smithi136:172.21.15.136=c", "target": ["mon-mgr", ""]}) -- 0x7efed00008d0 con 0x7efebc039650 2024-04-26T23:35:46.949 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:46.946+0000 7efec6ffd700 1 -- 172.21.15.136:0/470219067 <== mgr.14152 v2:172.21.15.72:6800/4281177709 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7efed00008d0 con 0x7efebc039650 2024-04-26T23:35:46.949 INFO:teuthology.orchestra.run.smithi136.stdout:Scheduled mon update... 2024-04-26T23:35:46.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:46.948+0000 7efed7e88700 1 -- 172.21.15.136:0/470219067 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7efebc039650 msgr2=0x7efebc03bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:35:46.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:46.948+0000 7efed7e88700 1 --2- 172.21.15.136:0/470219067 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7efebc039650 0x7efebc03bb10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7efec0006fd0 tx=0x7efec0006e40 comp rx=0 tx=0).stop 2024-04-26T23:35:46.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:46.948+0000 7efed7e88700 1 -- 172.21.15.136:0/470219067 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efed0103090 msgr2=0x7efed019e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:35:46.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:46.948+0000 7efed7e88700 1 --2- 172.21.15.136:0/470219067 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efed0103090 0x7efed019e5b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7efecc006010 tx=0x7efecc005780 comp rx=0 tx=0).stop 2024-04-26T23:35:46.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:46.948+0000 7efed7e88700 1 -- 172.21.15.136:0/470219067 shutdown_connections 2024-04-26T23:35:46.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:46.948+0000 7efed7e88700 1 --2- 172.21.15.136:0/470219067 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7efebc039650 0x7efebc03bb10 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:35:46.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:46.948+0000 7efed7e88700 1 --2- 172.21.15.136:0/470219067 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efed0103090 0x7efed019e5b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:35:46.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:46.948+0000 7efed7e88700 1 -- 172.21.15.136:0/470219067 >> 172.21.15.136:0/470219067 conn(0x7efed00fe960 msgr2=0x7efed0104a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:35:46.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:46.948+0000 7efed7e88700 1 -- 172.21.15.136:0/470219067 shutdown_connections 2024-04-26T23:35:46.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:46.948+0000 7efed7e88700 1 -- 172.21.15.136:0/470219067 wait complete. 2024-04-26T23:35:47.528 DEBUG:teuthology.orchestra.run.smithi076:mon.b> sudo journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mon.b.service 2024-04-26T23:35:47.531 DEBUG:teuthology.orchestra.run.smithi136:mon.c> sudo journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mon.c.service 2024-04-26T23:35:47.533 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-26T23:35:47.533 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-26T23:35:47.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:47 smithi072 ceph-mon[28184]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:35:47.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:47 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:47.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:47 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:35:47.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:47 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:35:47.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:47 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:35:47.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:47 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:47.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:47 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T23:35:47.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:47 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:35:48.205 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T23:35:48.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:48 smithi072 ceph-mon[28184]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi072:172.21.15.72=a;smithi076:172.21.15.76=b;smithi136:172.21.15.136=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:35:48.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:48 smithi072 ceph-mon[28184]: Saving service mon spec with placement smithi072:172.21.15.72=a;smithi076:172.21.15.76=b;smithi136:172.21.15.136=c;count:3 2024-04-26T23:35:48.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:48 smithi072 ceph-mon[28184]: Deploying daemon mon.c on smithi136 2024-04-26T23:35:49.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:49.533+0000 7f1678fca700 1 -- 172.21.15.136:0/541095130 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f16740757c0 msgr2=0x7f1674075ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:35:49.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:49.533+0000 7f1678fca700 1 --2- 172.21.15.136:0/541095130 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f16740757c0 0x7f1674075ba0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1664009910 tx=0x7f1664009c20 comp rx=0 tx=0).stop 2024-04-26T23:35:49.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:49.533+0000 7f1678fca700 1 -- 172.21.15.136:0/541095130 shutdown_connections 2024-04-26T23:35:49.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:49.533+0000 7f1678fca700 1 --2- 172.21.15.136:0/541095130 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f16740757c0 0x7f1674075ba0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:35:49.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:49.533+0000 7f1678fca700 1 -- 172.21.15.136:0/541095130 >> 172.21.15.136:0/541095130 conn(0x7f16740fe960 msgr2=0x7f1674100d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:35:49.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:49.533+0000 7f1678fca700 1 -- 172.21.15.136:0/541095130 shutdown_connections 2024-04-26T23:35:49.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:49.533+0000 7f1678fca700 1 -- 172.21.15.136:0/541095130 wait complete. 2024-04-26T23:35:49.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:49.534+0000 7f1678fca700 1 Processor -- start 2024-04-26T23:35:49.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:49.534+0000 7f1678fca700 1 -- start start 2024-04-26T23:35:49.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:49.534+0000 7f1678fca700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f16740757c0 0x7f167419eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:35:49.539 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:49.534+0000 7f1678fca700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f1674113980 con 0x7f16740757c0 2024-04-26T23:35:49.541 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:49.535+0000 7f167259c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f16740757c0 0x7f167419eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:35:49.542 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:49.535+0000 7f167259c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f16740757c0 0x7f167419eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.136:33482/0 (socket says 172.21.15.136:33482) 2024-04-26T23:35:49.542 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:49.535+0000 7f167259c700 1 -- 172.21.15.136:0/614627154 learned_addr learned my addr 172.21.15.136:0/614627154 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-26T23:35:49.542 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:49.536+0000 7f167259c700 1 -- 172.21.15.136:0/614627154 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f167419f400 con 0x7f16740757c0 2024-04-26T23:35:49.542 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:49.536+0000 7f167259c700 1 --2- 172.21.15.136:0/614627154 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f16740757c0 0x7f167419eec0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f16640056d0 tx=0x7f16640057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:35:49.542 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:49.536+0000 7f16637fe700 1 -- 172.21.15.136:0/614627154 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1664026070 con 0x7f16740757c0 2024-04-26T23:35:49.542 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:49.536+0000 7f16637fe700 1 -- 172.21.15.136:0/614627154 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1664004370 con 0x7f16740757c0 2024-04-26T23:35:49.542 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:49.536+0000 7f1678fca700 1 -- 172.21.15.136:0/614627154 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f167419f690 con 0x7f16740757c0 2024-04-26T23:35:49.542 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:49.536+0000 7f1678fca700 1 -- 172.21.15.136:0/614627154 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f167419faf0 con 0x7f16740757c0 2024-04-26T23:35:49.542 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:49.536+0000 7f16637fe700 1 -- 172.21.15.136:0/614627154 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f16640039a0 con 0x7f16740757c0 2024-04-26T23:35:49.542 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:49.537+0000 7f1678fca700 1 -- 172.21.15.136:0/614627154 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1674076f00 con 0x7f16740757c0 2024-04-26T23:35:49.542 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:49.538+0000 7f16637fe700 1 -- 172.21.15.136:0/614627154 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f1664029070 con 0x7f16740757c0 2024-04-26T23:35:49.542 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:49.538+0000 7f16637fe700 1 --2- 172.21.15.136:0/614627154 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f165c039650 0x7f165c03bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:35:49.543 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:49.538+0000 7f16637fe700 1 -- 172.21.15.136:0/614627154 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f166401e070 con 0x7f16740757c0 2024-04-26T23:35:49.543 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:49.538+0000 7f1671d9b700 1 --2- 172.21.15.136:0/614627154 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f165c039650 0x7f165c03bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:35:49.543 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:49.539+0000 7f1671d9b700 1 --2- 172.21.15.136:0/614627154 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f165c039650 0x7f165c03bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1668006fd0 tx=0x7f1668006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:35:49.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:49.554+0000 7f16637fe700 1 -- 172.21.15.136:0/614627154 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1664013070 con 0x7f16740757c0 2024-04-26T23:35:49.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:49 smithi072 ceph-mon[28184]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:35:49.846 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:49.843+0000 7f1678fca700 1 -- 172.21.15.136:0/614627154 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f16741a0220 con 0x7f16740757c0 2024-04-26T23:35:49.847 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:49.844+0000 7f16637fe700 1 -- 172.21.15.136:0/614627154 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f166404c380 con 0x7f16740757c0 2024-04-26T23:35:49.847 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-26T23:35:49.848 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":1,"fsid":"3c8453ae-0425-11ef-bc93-c7b262605968","modified":"2024-04-26T23:33:30.337416Z","created":"2024-04-26T23:33:30.337416Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:3300","nonce":0},{"type":"v1","addr":"172.21.15.72:6789","nonce":0}]},"addr":"172.21.15.72:6789/0","public_addr":"172.21.15.72:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-26T23:35:49.850 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:49.846+0000 7f1678fca700 1 -- 172.21.15.136:0/614627154 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f165c039650 msgr2=0x7f165c03bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:35:49.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:49.846+0000 7f1678fca700 1 --2- 172.21.15.136:0/614627154 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f165c039650 0x7f165c03bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1668006fd0 tx=0x7f1668006e40 comp rx=0 tx=0).stop 2024-04-26T23:35:49.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:49.846+0000 7f1678fca700 1 -- 172.21.15.136:0/614627154 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f16740757c0 msgr2=0x7f167419eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:35:49.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:49.846+0000 7f1678fca700 1 --2- 172.21.15.136:0/614627154 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f16740757c0 0x7f167419eec0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f16640056d0 tx=0x7f16640057b0 comp rx=0 tx=0).stop 2024-04-26T23:35:49.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:49.846+0000 7f1678fca700 1 -- 172.21.15.136:0/614627154 shutdown_connections 2024-04-26T23:35:49.851 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:49.846+0000 7f1678fca700 1 --2- 172.21.15.136:0/614627154 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f165c039650 0x7f165c03bb10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:35:49.852 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:49.846+0000 7f1678fca700 1 --2- 172.21.15.136:0/614627154 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f16740757c0 0x7f167419eec0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:35:49.852 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:49.846+0000 7f1678fca700 1 -- 172.21.15.136:0/614627154 >> 172.21.15.136:0/614627154 conn(0x7f16740fe960 msgr2=0x7f1674104ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:35:49.852 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:49.847+0000 7f1678fca700 1 -- 172.21.15.136:0/614627154 shutdown_connections 2024-04-26T23:35:49.852 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:49.847+0000 7f1678fca700 1 -- 172.21.15.136:0/614627154 wait complete. 2024-04-26T23:35:49.852 INFO:teuthology.orchestra.run.smithi136.stderr:dumped monmap epoch 1 2024-04-26T23:35:50.694 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:50 smithi072 ceph-mon[28184]: from='client.? 172.21.15.136:0/614627154' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T23:35:51.731 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:51 smithi072 ceph-mon[28184]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:35:52.106 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-26T23:35:52.106 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-26T23:35:52.352 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T23:35:52.743 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:52 smithi136 systemd[1]: Starting Ceph mon.c for 3c8453ae-0425-11ef-bc93-c7b262605968... 2024-04-26T23:35:53.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:53 smithi072 ceph-mon[28184]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:35:54.046 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:54.042+0000 7f43f3f46700 1 -- 172.21.15.136:0/1091700596 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43ec1031a0 msgr2=0x7f43ec103580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:35:54.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:54.042+0000 7f43f3f46700 1 --2- 172.21.15.136:0/1091700596 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43ec1031a0 0x7f43ec103580 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f43dc009910 tx=0x7f43dc009c20 comp rx=0 tx=0).stop 2024-04-26T23:35:54.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:54.042+0000 7f43f3f46700 1 -- 172.21.15.136:0/1091700596 shutdown_connections 2024-04-26T23:35:54.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:54.042+0000 7f43f3f46700 1 --2- 172.21.15.136:0/1091700596 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43ec1031a0 0x7f43ec103580 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:35:54.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:54.042+0000 7f43f3f46700 1 -- 172.21.15.136:0/1091700596 >> 172.21.15.136:0/1091700596 conn(0x7f43ec0fea30 msgr2=0x7f43ec100e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:35:54.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:54.042+0000 7f43f3f46700 1 -- 172.21.15.136:0/1091700596 shutdown_connections 2024-04-26T23:35:54.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:54.042+0000 7f43f3f46700 1 -- 172.21.15.136:0/1091700596 wait complete. 2024-04-26T23:35:54.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:54.043+0000 7f43f3f46700 1 Processor -- start 2024-04-26T23:35:54.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:54.043+0000 7f43f3f46700 1 -- start start 2024-04-26T23:35:54.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:54.043+0000 7f43f3f46700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43ec1031a0 0x7f43ec19e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:35:54.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:54.043+0000 7f43f3f46700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f43ec10d420 con 0x7f43ec1031a0 2024-04-26T23:35:54.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:54.043+0000 7f43f1ce2700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43ec1031a0 0x7f43ec19e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:35:54.047 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:54.044+0000 7f43f1ce2700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43ec1031a0 0x7f43ec19e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.136:33488/0 (socket says 172.21.15.136:33488) 2024-04-26T23:35:54.048 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:54.044+0000 7f43f1ce2700 1 -- 172.21.15.136:0/3321307160 learned_addr learned my addr 172.21.15.136:0/3321307160 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-26T23:35:54.049 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:54.044+0000 7f43f1ce2700 1 -- 172.21.15.136:0/3321307160 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43ec19eb80 con 0x7f43ec1031a0 2024-04-26T23:35:54.049 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:54.044+0000 7f43f1ce2700 1 --2- 172.21.15.136:0/3321307160 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43ec1031a0 0x7f43ec19e640 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f43dc006010 tx=0x7f43dc005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:35:54.049 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:54.045+0000 7f43e2ffd700 1 -- 172.21.15.136:0/3321307160 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f43dc026070 con 0x7f43ec1031a0 2024-04-26T23:35:54.049 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:54.045+0000 7f43f3f46700 1 -- 172.21.15.136:0/3321307160 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43ec19ee70 con 0x7f43ec1031a0 2024-04-26T23:35:54.049 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:54.045+0000 7f43f3f46700 1 -- 172.21.15.136:0/3321307160 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43ec1987c0 con 0x7f43ec1031a0 2024-04-26T23:35:54.049 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:54.045+0000 7f43e2ffd700 1 -- 172.21.15.136:0/3321307160 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43dc00bba0 con 0x7f43ec1031a0 2024-04-26T23:35:54.049 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:54.045+0000 7f43e2ffd700 1 -- 172.21.15.136:0/3321307160 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f43dc0038a0 con 0x7f43ec1031a0 2024-04-26T23:35:54.049 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:54.046+0000 7f43f3f46700 1 -- 172.21.15.136:0/3321307160 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43ec104680 con 0x7f43ec1031a0 2024-04-26T23:35:54.050 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:54.046+0000 7f43e2ffd700 1 -- 172.21.15.136:0/3321307160 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f43dc029070 con 0x7f43ec1031a0 2024-04-26T23:35:54.050 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:54.046+0000 7f43e2ffd700 1 --2- 172.21.15.136:0/3321307160 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f43d8039650 0x7f43d803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:35:54.050 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:54.046+0000 7f43e2ffd700 1 -- 172.21.15.136:0/3321307160 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f43dc01e070 con 0x7f43ec1031a0 2024-04-26T23:35:54.050 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:54.047+0000 7f43f14e1700 1 --2- 172.21.15.136:0/3321307160 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f43d8039650 0x7f43d803bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:35:54.051 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:54.048+0000 7f43f14e1700 1 --2- 172.21.15.136:0/3321307160 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f43d8039650 0x7f43d803bb10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f43e8006fd0 tx=0x7f43e8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:35:54.065 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:54.062+0000 7f43e2ffd700 1 -- 172.21.15.136:0/3321307160 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f43dc013070 con 0x7f43ec1031a0 2024-04-26T23:35:54.365 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:54.361+0000 7f43f3f46700 1 -- 172.21.15.136:0/3321307160 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f43ec199cf0 con 0x7f43ec1031a0 2024-04-26T23:35:54.365 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:54.362+0000 7f43e2ffd700 1 -- 172.21.15.136:0/3321307160 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f43dc034090 con 0x7f43ec1031a0 2024-04-26T23:35:54.365 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-26T23:35:54.366 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":1,"fsid":"3c8453ae-0425-11ef-bc93-c7b262605968","modified":"2024-04-26T23:33:30.337416Z","created":"2024-04-26T23:33:30.337416Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:3300","nonce":0},{"type":"v1","addr":"172.21.15.72:6789","nonce":0}]},"addr":"172.21.15.72:6789/0","public_addr":"172.21.15.72:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-26T23:35:54.368 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:54.364+0000 7f43f3f46700 1 -- 172.21.15.136:0/3321307160 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f43d8039650 msgr2=0x7f43d803bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:35:54.368 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:54.364+0000 7f43f3f46700 1 --2- 172.21.15.136:0/3321307160 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f43d8039650 0x7f43d803bb10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f43e8006fd0 tx=0x7f43e8006e40 comp rx=0 tx=0).stop 2024-04-26T23:35:54.368 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:54.364+0000 7f43f3f46700 1 -- 172.21.15.136:0/3321307160 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43ec1031a0 msgr2=0x7f43ec19e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:35:54.368 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:54.364+0000 7f43f3f46700 1 --2- 172.21.15.136:0/3321307160 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43ec1031a0 0x7f43ec19e640 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f43dc006010 tx=0x7f43dc005780 comp rx=0 tx=0).stop 2024-04-26T23:35:54.368 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:54.364+0000 7f43f3f46700 1 -- 172.21.15.136:0/3321307160 shutdown_connections 2024-04-26T23:35:54.368 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:54.364+0000 7f43f3f46700 1 --2- 172.21.15.136:0/3321307160 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f43d8039650 0x7f43d803bb10 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:35:54.368 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:54.364+0000 7f43f3f46700 1 --2- 172.21.15.136:0/3321307160 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43ec1031a0 0x7f43ec19e640 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:35:54.368 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:54.364+0000 7f43f3f46700 1 -- 172.21.15.136:0/3321307160 >> 172.21.15.136:0/3321307160 conn(0x7f43ec0fea30 msgr2=0x7f43ec104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:35:54.368 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:54.364+0000 7f43f3f46700 1 -- 172.21.15.136:0/3321307160 shutdown_connections 2024-04-26T23:35:54.368 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:35:54.364+0000 7f43f3f46700 1 -- 172.21.15.136:0/3321307160 wait complete. 2024-04-26T23:35:54.369 INFO:teuthology.orchestra.run.smithi136.stderr:dumped monmap epoch 1 2024-04-26T23:35:54.471 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 podman[33018]: 2024-04-26 23:35:54.218418696 +0000 UTC m=+0.109702396 container create dcf9bcf7ae9cc96e6a1c35346fd3c5c540d8d994a5083f1523ae91ab0ae4429c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mon-c, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T23:35:54.471 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 podman[33018]: 2024-04-26 23:35:54.13333545 +0000 UTC m=+0.024619169 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T23:35:54.471 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 podman[33018]: 2024-04-26 23:35:54.466060724 +0000 UTC m=+0.357344424 container init dcf9bcf7ae9cc96e6a1c35346fd3c5c540d8d994a5083f1523ae91ab0ae4429c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mon-c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-26T23:35:54.728 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 podman[33018]: 2024-04-26 23:35:54.469968262 +0000 UTC m=+0.361251963 container start dcf9bcf7ae9cc96e6a1c35346fd3c5c540d8d994a5083f1523ae91ab0ae4429c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mon-c, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424) 2024-04-26T23:35:54.728 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: set uid:gid to 167:167 (ceph:ceph) 2024-04-26T23:35:54.728 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev), process ceph-mon, pid 2 2024-04-26T23:35:54.728 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: pidfile_write: ignore empty --pid-file 2024-04-26T23:35:54.728 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: load: jerasure load: lrc 2024-04-26T23:35:54.728 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: RocksDB version: 7.9.2 2024-04-26T23:35:54.728 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Git sha 0 2024-04-26T23:35:54.728 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Compile date 2024-04-26 11:39:23 2024-04-26T23:35:54.728 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: DB SUMMARY 2024-04-26T23:35:54.728 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: DB Session ID: CNY991MREZ5G3J3OP59S 2024-04-26T23:35:54.728 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: CURRENT file: CURRENT 2024-04-26T23:35:54.729 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: IDENTITY file: IDENTITY 2024-04-26T23:35:54.729 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-26T23:35:54.729 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-04-26T23:35:54.729 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-04-26T23:35:54.729 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.error_if_exists: 0 2024-04-26T23:35:54.729 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.create_if_missing: 0 2024-04-26T23:35:54.729 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.paranoid_checks: 1 2024-04-26T23:35:54.729 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-26T23:35:54.729 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-26T23:35:54.729 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-26T23:35:54.729 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.env: 0x5587968fbcc0 2024-04-26T23:35:54.729 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.fs: PosixFileSystem 2024-04-26T23:35:54.729 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.info_log: 0x558797c49d80 2024-04-26T23:35:54.729 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.max_file_opening_threads: 16 2024-04-26T23:35:54.730 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.statistics: (nil) 2024-04-26T23:35:54.730 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.use_fsync: 0 2024-04-26T23:35:54.730 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.max_log_file_size: 0 2024-04-26T23:35:54.730 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-26T23:35:54.730 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-26T23:35:54.730 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.keep_log_file_num: 1000 2024-04-26T23:35:54.730 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.recycle_log_file_num: 0 2024-04-26T23:35:54.730 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.allow_fallocate: 1 2024-04-26T23:35:54.732 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.allow_mmap_reads: 0 2024-04-26T23:35:54.732 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.allow_mmap_writes: 0 2024-04-26T23:35:54.732 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.use_direct_reads: 0 2024-04-26T23:35:54.732 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-26T23:35:54.732 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.create_missing_column_families: 0 2024-04-26T23:35:54.732 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.db_log_dir: 2024-04-26T23:35:54.732 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.wal_dir: 2024-04-26T23:35:54.732 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-26T23:35:54.732 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-26T23:35:54.732 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-26T23:35:54.732 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-26T23:35:54.732 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-26T23:35:54.733 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-26T23:35:54.733 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.advise_random_on_open: 1 2024-04-26T23:35:54.733 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.db_write_buffer_size: 0 2024-04-26T23:35:54.733 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.write_buffer_manager: 0x558797c1b0e0 2024-04-26T23:35:54.733 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-26T23:35:54.733 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-26T23:35:54.733 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-26T23:35:54.733 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.rate_limiter: (nil) 2024-04-26T23:35:54.733 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-26T23:35:54.733 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.wal_recovery_mode: 2 2024-04-26T23:35:54.733 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.enable_thread_tracking: 0 2024-04-26T23:35:54.733 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.enable_pipelined_write: 0 2024-04-26T23:35:54.733 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.unordered_write: 0 2024-04-26T23:35:54.734 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-26T23:35:54.734 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-26T23:35:54.734 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-26T23:35:54.734 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-26T23:35:54.734 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.row_cache: None 2024-04-26T23:35:54.734 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.wal_filter: None 2024-04-26T23:35:54.734 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-26T23:35:54.734 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.allow_ingest_behind: 0 2024-04-26T23:35:54.734 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.two_write_queues: 0 2024-04-26T23:35:54.734 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.manual_wal_flush: 0 2024-04-26T23:35:54.734 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.wal_compression: 0 2024-04-26T23:35:54.734 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.atomic_flush: 0 2024-04-26T23:35:54.734 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-26T23:35:54.734 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-26T23:35:54.735 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-26T23:35:54.735 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.log_readahead_size: 0 2024-04-26T23:35:54.735 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-26T23:35:54.735 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.best_efforts_recovery: 0 2024-04-26T23:35:54.735 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-26T23:35:54.735 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-26T23:35:54.735 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.allow_data_in_errors: 0 2024-04-26T23:35:54.735 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.db_host_id: __hostname__ 2024-04-26T23:35:54.735 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-26T23:35:54.735 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.max_background_jobs: 2 2024-04-26T23:35:54.735 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.max_background_compactions: -1 2024-04-26T23:35:54.735 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.max_subcompactions: 1 2024-04-26T23:35:54.735 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-26T23:35:54.735 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-26T23:35:54.736 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-26T23:35:54.736 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.max_total_wal_size: 0 2024-04-26T23:35:54.736 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-26T23:35:54.736 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-26T23:35:54.736 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-26T23:35:54.736 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-26T23:35:54.736 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.max_open_files: -1 2024-04-26T23:35:54.736 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.bytes_per_sync: 0 2024-04-26T23:35:54.736 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-26T23:35:54.736 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-26T23:35:54.736 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.compaction_readahead_size: 0 2024-04-26T23:35:54.736 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.max_background_flushes: -1 2024-04-26T23:35:54.736 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Compression algorithms supported: 2024-04-26T23:35:54.736 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-26T23:35:54.736 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: kZSTD supported: 0 2024-04-26T23:35:54.737 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: kXpressCompression supported: 0 2024-04-26T23:35:54.737 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: kLZ4HCCompression supported: 1 2024-04-26T23:35:54.737 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: kZlibCompression supported: 1 2024-04-26T23:35:54.737 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: kSnappyCompression supported: 1 2024-04-26T23:35:54.737 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: kLZ4Compression supported: 1 2024-04-26T23:35:54.737 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: kBZip2Compression supported: 0 2024-04-26T23:35:54.737 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-26T23:35:54.737 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-26T23:35:54.737 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-04-26T23:35:54.737 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-26T23:35:54.737 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-26T23:35:54.737 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.merge_operator: 2024-04-26T23:35:54.737 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.compaction_filter: None 2024-04-26T23:35:54.737 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.compaction_filter_factory: None 2024-04-26T23:35:54.737 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.sst_partitioner_factory: None 2024-04-26T23:35:54.738 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-26T23:35:54.738 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-26T23:35:54.738 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x558797c49f40) 2024-04-26T23:35:54.738 INFO:journalctl@ceph.mon.c.smithi136.stdout: cache_index_and_filter_blocks: 1 2024-04-26T23:35:54.738 INFO:journalctl@ceph.mon.c.smithi136.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-26T23:35:54.738 INFO:journalctl@ceph.mon.c.smithi136.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-26T23:35:54.738 INFO:journalctl@ceph.mon.c.smithi136.stdout: pin_top_level_index_and_filter: 1 2024-04-26T23:35:54.738 INFO:journalctl@ceph.mon.c.smithi136.stdout: index_type: 0 2024-04-26T23:35:54.738 INFO:journalctl@ceph.mon.c.smithi136.stdout: data_block_index_type: 0 2024-04-26T23:35:54.738 INFO:journalctl@ceph.mon.c.smithi136.stdout: index_shortening: 1 2024-04-26T23:35:54.738 INFO:journalctl@ceph.mon.c.smithi136.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-26T23:35:54.738 INFO:journalctl@ceph.mon.c.smithi136.stdout: checksum: 4 2024-04-26T23:35:54.738 INFO:journalctl@ceph.mon.c.smithi136.stdout: no_block_cache: 0 2024-04-26T23:35:54.738 INFO:journalctl@ceph.mon.c.smithi136.stdout: block_cache: 0x558797c4d1f0 2024-04-26T23:35:54.738 INFO:journalctl@ceph.mon.c.smithi136.stdout: block_cache_name: BinnedLRUCache 2024-04-26T23:35:54.739 INFO:journalctl@ceph.mon.c.smithi136.stdout: block_cache_options: 2024-04-26T23:35:54.739 INFO:journalctl@ceph.mon.c.smithi136.stdout: capacity : 536870912 2024-04-26T23:35:54.739 INFO:journalctl@ceph.mon.c.smithi136.stdout: num_shard_bits : 4 2024-04-26T23:35:54.739 INFO:journalctl@ceph.mon.c.smithi136.stdout: strict_capacity_limit : 0 2024-04-26T23:35:54.739 INFO:journalctl@ceph.mon.c.smithi136.stdout: high_pri_pool_ratio: 0.000 2024-04-26T23:35:54.739 INFO:journalctl@ceph.mon.c.smithi136.stdout: block_cache_compressed: (nil) 2024-04-26T23:35:54.739 INFO:journalctl@ceph.mon.c.smithi136.stdout: persistent_cache: (nil) 2024-04-26T23:35:54.739 INFO:journalctl@ceph.mon.c.smithi136.stdout: block_size: 4096 2024-04-26T23:35:54.739 INFO:journalctl@ceph.mon.c.smithi136.stdout: block_size_deviation: 10 2024-04-26T23:35:54.739 INFO:journalctl@ceph.mon.c.smithi136.stdout: block_restart_interval: 16 2024-04-26T23:35:54.739 INFO:journalctl@ceph.mon.c.smithi136.stdout: index_block_restart_interval: 1 2024-04-26T23:35:54.739 INFO:journalctl@ceph.mon.c.smithi136.stdout: metadata_block_size: 4096 2024-04-26T23:35:54.739 INFO:journalctl@ceph.mon.c.smithi136.stdout: partition_filters: 0 2024-04-26T23:35:54.739 INFO:journalctl@ceph.mon.c.smithi136.stdout: use_delta_encoding: 1 2024-04-26T23:35:54.739 INFO:journalctl@ceph.mon.c.smithi136.stdout: filter_policy: bloomfilter 2024-04-26T23:35:54.740 INFO:journalctl@ceph.mon.c.smithi136.stdout: whole_key_filtering: 1 2024-04-26T23:35:54.740 INFO:journalctl@ceph.mon.c.smithi136.stdout: verify_compression: 0 2024-04-26T23:35:54.740 INFO:journalctl@ceph.mon.c.smithi136.stdout: read_amp_bytes_per_bit: 0 2024-04-26T23:35:54.740 INFO:journalctl@ceph.mon.c.smithi136.stdout: format_version: 5 2024-04-26T23:35:54.740 INFO:journalctl@ceph.mon.c.smithi136.stdout: enable_index_compression: 1 2024-04-26T23:35:54.740 INFO:journalctl@ceph.mon.c.smithi136.stdout: block_align: 0 2024-04-26T23:35:54.740 INFO:journalctl@ceph.mon.c.smithi136.stdout: max_auto_readahead_size: 262144 2024-04-26T23:35:54.740 INFO:journalctl@ceph.mon.c.smithi136.stdout: prepopulate_block_cache: 0 2024-04-26T23:35:54.740 INFO:journalctl@ceph.mon.c.smithi136.stdout: initial_auto_readahead_size: 8192 2024-04-26T23:35:54.740 INFO:journalctl@ceph.mon.c.smithi136.stdout: num_file_reads_for_auto_readahead: 2 2024-04-26T23:35:54.740 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.write_buffer_size: 33554432 2024-04-26T23:35:54.740 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.max_write_buffer_number: 2 2024-04-26T23:35:54.740 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.compression: NoCompression 2024-04-26T23:35:54.740 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.bottommost_compression: Disabled 2024-04-26T23:35:54.740 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.prefix_extractor: nullptr 2024-04-26T23:35:54.741 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-26T23:35:54.741 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.num_levels: 7 2024-04-26T23:35:54.741 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-26T23:35:54.741 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-26T23:35:54.741 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-26T23:35:54.741 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-26T23:35:54.741 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-26T23:35:54.741 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-26T23:35:54.741 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-26T23:35:54.741 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-26T23:35:54.741 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-26T23:35:54.741 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-26T23:35:54.741 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-26T23:35:54.741 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-26T23:35:54.741 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-26T23:35:54.742 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.compression_opts.level: 32767 2024-04-26T23:35:54.742 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.compression_opts.strategy: 0 2024-04-26T23:35:54.742 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-26T23:35:54.742 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-26T23:35:54.742 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-26T23:35:54.742 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-26T23:35:54.742 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.compression_opts.enabled: false 2024-04-26T23:35:54.742 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-26T23:35:54.742 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-26T23:35:54.742 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-26T23:35:54.742 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-26T23:35:54.742 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.target_file_size_base: 67108864 2024-04-26T23:35:54.742 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-26T23:35:54.742 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-26T23:35:54.742 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-26T23:35:54.743 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-26T23:35:54.743 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-26T23:35:54.743 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-26T23:35:54.743 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-26T23:35:54.743 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-26T23:35:54.743 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-26T23:35:54.743 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-26T23:35:54.743 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-26T23:35:54.743 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-26T23:35:54.743 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-26T23:35:54.743 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-26T23:35:54.743 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.arena_block_size: 1048576 2024-04-26T23:35:54.743 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-26T23:35:54.743 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-26T23:35:54.743 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.disable_auto_compactions: 0 2024-04-26T23:35:54.743 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-26T23:35:54.743 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-26T23:35:54.744 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-26T23:35:54.744 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-26T23:35:54.744 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-26T23:35:54.744 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-26T23:35:54.744 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-26T23:35:54.744 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-26T23:35:54.744 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-26T23:35:54.744 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-26T23:35:54.744 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-26T23:35:54.744 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.inplace_update_support: 0 2024-04-26T23:35:54.744 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-26T23:35:54.744 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-26T23:35:54.744 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-26T23:35:54.744 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-26T23:35:54.744 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.bloom_locality: 0 2024-04-26T23:35:54.745 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.max_successive_merges: 0 2024-04-26T23:35:54.745 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-26T23:35:54.745 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.paranoid_file_checks: 0 2024-04-26T23:35:54.745 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.force_consistency_checks: 1 2024-04-26T23:35:54.745 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.report_bg_io_stats: 0 2024-04-26T23:35:54.745 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.ttl: 2592000 2024-04-26T23:35:54.745 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-26T23:35:54.745 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-26T23:35:54.745 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-26T23:35:54.745 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.enable_blob_files: false 2024-04-26T23:35:54.745 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.min_blob_size: 0 2024-04-26T23:35:54.745 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.blob_file_size: 268435456 2024-04-26T23:35:54.745 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-26T23:35:54.745 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-26T23:35:54.746 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-26T23:35:54.746 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-26T23:35:54.746 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-26T23:35:54.746 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.blob_file_starting_level: 0 2024-04-26T23:35:54.746 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-26T23:35:54.746 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-26T23:35:54.746 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-26T23:35:54.746 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: d18c75d2-6fe4-4804-a7ca-64d3085b6cf5 2024-04-26T23:35:54.746 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714174554500323, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-26T23:35:54.746 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-26T23:35:54.746 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714174554500734, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714174554, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d18c75d2-6fe4-4804-a7ca-64d3085b6cf5", "db_session_id": "CNY991MREZ5G3J3OP59S", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-26T23:35:54.746 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714174554500813, "job": 1, "event": "recovery_finished"} 2024-04-26T23:35:54.746 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-26T23:35:54.747 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-26T23:35:54.747 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x558797d6a000 2024-04-26T23:35:54.747 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: DB pointer 0x558797d56000 2024-04-26T23:35:54.747 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-26T23:35:54.747 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-26T23:35:54.747 INFO:journalctl@ceph.mon.c.smithi136.stdout: ** DB Stats ** 2024-04-26T23:35:54.747 INFO:journalctl@ceph.mon.c.smithi136.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T23:35:54.747 INFO:journalctl@ceph.mon.c.smithi136.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-26T23:35:54.747 INFO:journalctl@ceph.mon.c.smithi136.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T23:35:54.747 INFO:journalctl@ceph.mon.c.smithi136.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T23:35:54.747 INFO:journalctl@ceph.mon.c.smithi136.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-26T23:35:54.747 INFO:journalctl@ceph.mon.c.smithi136.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T23:35:54.747 INFO:journalctl@ceph.mon.c.smithi136.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T23:35:54.747 INFO:journalctl@ceph.mon.c.smithi136.stdout: 2024-04-26T23:35:54.747 INFO:journalctl@ceph.mon.c.smithi136.stdout: ** Compaction Stats [default] ** 2024-04-26T23:35:54.747 INFO:journalctl@ceph.mon.c.smithi136.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T23:35:54.748 INFO:journalctl@ceph.mon.c.smithi136.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-26T23:35:54.748 INFO:journalctl@ceph.mon.c.smithi136.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-26T23:35:54.748 INFO:journalctl@ceph.mon.c.smithi136.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-26T23:35:54.748 INFO:journalctl@ceph.mon.c.smithi136.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-26T23:35:54.748 INFO:journalctl@ceph.mon.c.smithi136.stdout: 2024-04-26T23:35:54.748 INFO:journalctl@ceph.mon.c.smithi136.stdout: ** Compaction Stats [default] ** 2024-04-26T23:35:54.748 INFO:journalctl@ceph.mon.c.smithi136.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T23:35:54.748 INFO:journalctl@ceph.mon.c.smithi136.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-26T23:35:54.748 INFO:journalctl@ceph.mon.c.smithi136.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-26T23:35:54.748 INFO:journalctl@ceph.mon.c.smithi136.stdout: 2024-04-26T23:35:54.748 INFO:journalctl@ceph.mon.c.smithi136.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-26T23:35:54.748 INFO:journalctl@ceph.mon.c.smithi136.stdout: 2024-04-26T23:35:54.748 INFO:journalctl@ceph.mon.c.smithi136.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T23:35:54.748 INFO:journalctl@ceph.mon.c.smithi136.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-26T23:35:54.748 INFO:journalctl@ceph.mon.c.smithi136.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-26T23:35:54.749 INFO:journalctl@ceph.mon.c.smithi136.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-26T23:35:54.749 INFO:journalctl@ceph.mon.c.smithi136.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-26T23:35:54.749 INFO:journalctl@ceph.mon.c.smithi136.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-26T23:35:54.749 INFO:journalctl@ceph.mon.c.smithi136.stdout: Cumulative compaction: 0.00 GB write, 0.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T23:35:54.749 INFO:journalctl@ceph.mon.c.smithi136.stdout: Interval compaction: 0.00 GB write, 0.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T23:35:54.749 INFO:journalctl@ceph.mon.c.smithi136.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-26T23:35:54.749 INFO:journalctl@ceph.mon.c.smithi136.stdout: Block cache BinnedLRUCache@0x558797c4d1f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-26T23:35:54.749 INFO:journalctl@ceph.mon.c.smithi136.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-26T23:35:54.749 INFO:journalctl@ceph.mon.c.smithi136.stdout: 2024-04-26T23:35:54.749 INFO:journalctl@ceph.mon.c.smithi136.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-26T23:35:54.749 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-04-26T23:35:54.749 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: using public_addr v2:172.21.15.136:0/0 -> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] 2024-04-26T23:35:54.749 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: starting mon.c rank -1 at public addrs [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] at bind addrs [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 3c8453ae-0425-11ef-bc93-c7b262605968 2024-04-26T23:35:54.749 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: mon.c@-1(???) e0 preinit fsid 3c8453ae-0425-11ef-bc93-c7b262605968 2024-04-26T23:35:54.749 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: mon.c@-1(synchronizing).mds e1 new map 2024-04-26T23:35:54.750 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: mon.c@-1(synchronizing).mds e1 print_map 2024-04-26T23:35:54.750 INFO:journalctl@ceph.mon.c.smithi136.stdout: e1 2024-04-26T23:35:54.750 INFO:journalctl@ceph.mon.c.smithi136.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-26T23:35:54.750 INFO:journalctl@ceph.mon.c.smithi136.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-26T23:35:54.750 INFO:journalctl@ceph.mon.c.smithi136.stdout: legacy client fscid: -1 2024-04-26T23:35:54.750 INFO:journalctl@ceph.mon.c.smithi136.stdout: 2024-04-26T23:35:54.750 INFO:journalctl@ceph.mon.c.smithi136.stdout: No filesystems configured 2024-04-26T23:35:54.750 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-26T23:35:54.750 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-26T23:35:54.750 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-26T23:35:54.750 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-26T23:35:54.750 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-26T23:35:54.750 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-26T23:35:54.750 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-26T23:35:54.751 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T23:35:54.751 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T23:35:54.751 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T23:35:54.751 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:35:54.751 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi076", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:35:54.751 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: Deploying cephadm binary to smithi076 2024-04-26T23:35:54.751 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:35:54.751 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:35:54.751 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:54.751 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: Added host smithi076 2024-04-26T23:35:54.751 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:35:54.751 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:54.751 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:35:54.751 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:54.752 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:54.752 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:35:54.752 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T23:35:54.752 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:35:54.752 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:54.752 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:54.752 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:54.752 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:54.752 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi076", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:35:54.752 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:35:54.752 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:35:54.752 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: Updating smithi076:/etc/ceph/ceph.conf 2024-04-26T23:35:54.752 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: Updating smithi076:/var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T23:35:54.752 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: Updating smithi076:/etc/ceph/ceph.client.admin.keyring 2024-04-26T23:35:54.753 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:35:54.753 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: Updating smithi076:/var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T23:35:54.753 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:54.753 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:54.753 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:54.753 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi136", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:35:54.753 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: Deploying cephadm binary to smithi136 2024-04-26T23:35:54.753 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:35:54.753 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:35:54.753 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:54.753 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:35:54.753 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: Added host smithi136 2024-04-26T23:35:54.753 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:54.753 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:54.753 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:35:54.753 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:54.754 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:35:54.754 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T23:35:54.754 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:35:54.754 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:54.754 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:54.754 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:54.755 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:54.755 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi136", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:35:54.755 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:35:54.755 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:35:54.755 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: Updating smithi136:/etc/ceph/ceph.conf 2024-04-26T23:35:54.755 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/931525612' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-26T23:35:54.755 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: Updating smithi136:/var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T23:35:54.755 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2024-04-26T23:35:54.755 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: Updating smithi136:/var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T23:35:54.755 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:35:54.755 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/931525612' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-26T23:35:54.755 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T23:35:54.755 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:54.755 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:54.755 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:54.756 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:35:54.756 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:54.756 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:35:54.756 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:35:54.756 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:35:54.756 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:35:54.756 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T23:35:54.756 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:35:54.756 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi072:172.21.15.72=a;smithi076:172.21.15.76=b;smithi136:172.21.15.136=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:35:54.756 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: Saving service mon spec with placement smithi072:172.21.15.72=a;smithi076:172.21.15.76=b;smithi136:172.21.15.136=c;count:3 2024-04-26T23:35:54.756 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: Deploying daemon mon.c on smithi136 2024-04-26T23:35:54.756 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:35:54.756 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: from='client.? 172.21.15.136:0/614627154' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T23:35:54.756 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:35:54.756 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:35:54.757 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: from='client.? 172.21.15.136:0/3321307160' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T23:35:54.757 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 ceph-mon[33050]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-26T23:35:54.757 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 bash[33018]: dcf9bcf7ae9cc96e6a1c35346fd3c5c540d8d994a5083f1523ae91ab0ae4429c 2024-04-26T23:35:54.757 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:54 smithi136 systemd[1]: Started Ceph mon.c for 3c8453ae-0425-11ef-bc93-c7b262605968. 2024-04-26T23:35:54.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:54 smithi072 ceph-mon[28184]: from='client.? 172.21.15.136:0/3321307160' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T23:35:56.115 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-26T23:35:56.116 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-26T23:35:56.360 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T23:36:00.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:59 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T23:36:00.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:59 smithi072 ceph-mon[28184]: mon.a calling monitor election 2024-04-26T23:36:00.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:59 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:36:00.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:59 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:36:00.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:59 smithi072 ceph-mon[28184]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:00.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:59 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:36:00.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:59 smithi072 ceph-mon[28184]: mon.c calling monitor election 2024-04-26T23:36:00.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:59 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:36:00.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:59 smithi072 ceph-mon[28184]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:00.045 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:59 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:36:00.045 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:59 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:36:00.045 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:59 smithi072 ceph-mon[28184]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T23:36:00.045 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:59 smithi072 ceph-mon[28184]: monmap epoch 2 2024-04-26T23:36:00.045 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:59 smithi072 ceph-mon[28184]: fsid 3c8453ae-0425-11ef-bc93-c7b262605968 2024-04-26T23:36:00.045 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:59 smithi072 ceph-mon[28184]: last_changed 2024-04-26T23:35:54.532964+0000 2024-04-26T23:36:00.045 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:59 smithi072 ceph-mon[28184]: created 2024-04-26T23:33:30.337416+0000 2024-04-26T23:36:00.045 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:59 smithi072 ceph-mon[28184]: min_mon_release 19 (squid) 2024-04-26T23:36:00.045 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:59 smithi072 ceph-mon[28184]: election_strategy: 1 2024-04-26T23:36:00.045 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:59 smithi072 ceph-mon[28184]: 0: [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] mon.a 2024-04-26T23:36:00.045 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:59 smithi072 ceph-mon[28184]: 1: [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] mon.c 2024-04-26T23:36:00.045 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:59 smithi072 ceph-mon[28184]: fsmap 2024-04-26T23:36:00.045 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:59 smithi072 ceph-mon[28184]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T23:36:00.045 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:59 smithi072 ceph-mon[28184]: mgrmap e13: x(active, since 61s) 2024-04-26T23:36:00.045 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:59 smithi072 ceph-mon[28184]: overall HEALTH_OK 2024-04-26T23:36:00.046 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:59 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:00.046 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:59 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:00.046 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:59 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:00.046 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:59 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T23:36:00.046 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:35:59 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:36:00.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:59 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T23:36:00.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:59 smithi136 ceph-mon[33050]: mon.a calling monitor election 2024-04-26T23:36:00.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:59 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:36:00.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:59 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:36:00.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:59 smithi136 ceph-mon[33050]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:00.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:59 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:36:00.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:59 smithi136 ceph-mon[33050]: mon.c calling monitor election 2024-04-26T23:36:00.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:59 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:36:00.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:59 smithi136 ceph-mon[33050]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:00.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:59 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:36:00.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:59 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:36:00.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:59 smithi136 ceph-mon[33050]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T23:36:00.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:59 smithi136 ceph-mon[33050]: monmap epoch 2 2024-04-26T23:36:00.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:59 smithi136 ceph-mon[33050]: fsid 3c8453ae-0425-11ef-bc93-c7b262605968 2024-04-26T23:36:00.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:59 smithi136 ceph-mon[33050]: last_changed 2024-04-26T23:35:54.532964+0000 2024-04-26T23:36:00.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:59 smithi136 ceph-mon[33050]: created 2024-04-26T23:33:30.337416+0000 2024-04-26T23:36:00.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:59 smithi136 ceph-mon[33050]: min_mon_release 19 (squid) 2024-04-26T23:36:00.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:59 smithi136 ceph-mon[33050]: election_strategy: 1 2024-04-26T23:36:00.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:59 smithi136 ceph-mon[33050]: 0: [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] mon.a 2024-04-26T23:36:00.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:59 smithi136 ceph-mon[33050]: 1: [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] mon.c 2024-04-26T23:36:00.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:59 smithi136 ceph-mon[33050]: fsmap 2024-04-26T23:36:00.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:59 smithi136 ceph-mon[33050]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T23:36:00.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:59 smithi136 ceph-mon[33050]: mgrmap e13: x(active, since 61s) 2024-04-26T23:36:00.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:59 smithi136 ceph-mon[33050]: overall HEALTH_OK 2024-04-26T23:36:00.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:59 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:00.052 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:59 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:00.052 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:59 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:00.052 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:59 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T23:36:00.052 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:35:59 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:36:00.883 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:00.879+0000 7f60b84f1700 1 -- 172.21.15.136:0/2693539836 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f609c0035c0 msgr2=0x7f609c005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:36:00.883 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:00.879+0000 7f60b84f1700 1 --2- 172.21.15.136:0/2693539836 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f609c0035c0 0x7f609c005a50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f60b0101d60 tx=0x7f60a000feb0 comp rx=0 tx=0).stop 2024-04-26T23:36:00.883 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:00.880+0000 7f60b84f1700 1 -- 172.21.15.136:0/2693539836 shutdown_connections 2024-04-26T23:36:00.884 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:00.880+0000 7f60b84f1700 1 --2- 172.21.15.136:0/2693539836 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f609c0035c0 0x7f609c005a50 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:36:00.884 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:00.880+0000 7f60b84f1700 1 -- 172.21.15.136:0/2693539836 >> 172.21.15.136:0/2693539836 conn(0x7f60b00fd920 msgr2=0x7f60b00ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:36:00.884 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:00.880+0000 7f60b84f1700 1 -- 172.21.15.136:0/2693539836 shutdown_connections 2024-04-26T23:36:00.884 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:00.880+0000 7f60b84f1700 1 -- 172.21.15.136:0/2693539836 wait complete. 2024-04-26T23:36:00.884 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:00.881+0000 7f60b84f1700 1 Processor -- start 2024-04-26T23:36:00.884 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:00.881+0000 7f60b84f1700 1 -- start start 2024-04-26T23:36:00.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:00 smithi136 ceph-mon[33050]: Deploying daemon mon.b on smithi076 2024-04-26T23:36:00.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:00 smithi136 ceph-mon[33050]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:00.885 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:00 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:36:00.886 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:00.881+0000 7f60b84f1700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f609c0035c0 0x7f60b0103b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:36:00.886 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:00.881+0000 7f60b84f1700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f60b0104070 0x7f60b01044f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:36:00.886 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:00.881+0000 7f60b84f1700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f60b0109d80 con 0x7f609c0035c0 2024-04-26T23:36:00.886 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:00.881+0000 7f60b84f1700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f60b0109c00 con 0x7f60b0104070 2024-04-26T23:36:00.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:00.881+0000 7f60b5a8c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f60b0104070 0x7f60b01044f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:36:00.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:00.881+0000 7f60b5a8c700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f60b0104070 0x7f60b01044f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:39910/0 (socket says 172.21.15.136:39910) 2024-04-26T23:36:00.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:00.881+0000 7f60b5a8c700 1 -- 172.21.15.136:0/3622902924 learned_addr learned my addr 172.21.15.136:0/3622902924 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-26T23:36:00.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:00.881+0000 7f60b5a8c700 1 -- 172.21.15.136:0/3622902924 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f60b0104070 msgr2=0x7f60b01044f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-26T23:36:00.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:00.881+0000 7f60b5a8c700 1 -- 172.21.15.136:0/3622902924 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f60b0104070 msgr2=0x7f60b01044f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T23:36:00.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:00.881+0000 7f60b5a8c700 1 --2- 172.21.15.136:0/3622902924 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f60b0104070 0x7f60b01044f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T23:36:00.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:00.881+0000 7f60b5a8c700 1 --2- 172.21.15.136:0/3622902924 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f60b0104070 0x7f60b01044f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T23:36:00.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:00.882+0000 7f60b628d700 1 --2- 172.21.15.136:0/3622902924 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f609c0035c0 0x7f60b0103b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:36:00.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:00.882+0000 7f60b628d700 1 -- 172.21.15.136:0/3622902924 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f60b0104070 msgr2=0x7f60b01044f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T23:36:00.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:00.882+0000 7f60b628d700 1 --2- 172.21.15.136:0/3622902924 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f60b0104070 0x7f60b01044f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:36:00.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:00.882+0000 7f60b628d700 1 -- 172.21.15.136:0/3622902924 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60b0105a50 con 0x7f609c0035c0 2024-04-26T23:36:00.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:00.883+0000 7f60b628d700 1 --2- 172.21.15.136:0/3622902924 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f609c0035c0 0x7f60b0103b30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f60a0010010 tx=0x7f60a0005cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:36:00.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:00.883+0000 7f60a77fe700 1 -- 172.21.15.136:0/3622902924 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f60a0013960 con 0x7f609c0035c0 2024-04-26T23:36:00.887 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:00.883+0000 7f60b84f1700 1 -- 172.21.15.136:0/3622902924 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60b01a4c00 con 0x7f609c0035c0 2024-04-26T23:36:00.888 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:00.883+0000 7f60a77fe700 1 -- 172.21.15.136:0/3622902924 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60a00044e0 con 0x7f609c0035c0 2024-04-26T23:36:00.888 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:00.883+0000 7f60a77fe700 1 -- 172.21.15.136:0/3622902924 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f60a0003ce0 con 0x7f609c0035c0 2024-04-26T23:36:00.888 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:00.883+0000 7f60b84f1700 1 -- 172.21.15.136:0/3622902924 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60b01a5140 con 0x7f609c0035c0 2024-04-26T23:36:00.888 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:00.884+0000 7f60b84f1700 1 -- 172.21.15.136:0/3622902924 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60b0103570 con 0x7f609c0035c0 2024-04-26T23:36:00.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:00.884+0000 7f60a77fe700 1 -- 172.21.15.136:0/3622902924 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f60a0023070 con 0x7f609c0035c0 2024-04-26T23:36:00.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:00.885+0000 7f60a77fe700 1 --2- 172.21.15.136:0/3622902924 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6094039590 0x7f609403ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:36:00.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:00.885+0000 7f60a77fe700 1 -- 172.21.15.136:0/3622902924 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f60a0020050 con 0x7f609c0035c0 2024-04-26T23:36:00.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:00.885+0000 7f60b5a8c700 1 --2- 172.21.15.136:0/3622902924 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6094039590 0x7f609403ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:36:00.889 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:00.886+0000 7f60b5a8c700 1 --2- 172.21.15.136:0/3622902924 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6094039590 0x7f609403ba50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f60ac009180 tx=0x7f60ac008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:36:00.903 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:00.900+0000 7f60a77fe700 1 -- 172.21.15.136:0/3622902924 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f60a0028020 con 0x7f609c0035c0 2024-04-26T23:36:01.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:00 smithi072 ceph-mon[28184]: Deploying daemon mon.b on smithi076 2024-04-26T23:36:01.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:00 smithi072 ceph-mon[28184]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:01.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:00 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:36:01.181 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:01.177+0000 7f60b84f1700 1 -- 172.21.15.136:0/3622902924 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f60b002e490 con 0x7f609c0035c0 2024-04-26T23:36:01.181 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:01.178+0000 7f60a77fe700 1 -- 172.21.15.136:0/3622902924 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f60a001a390 con 0x7f609c0035c0 2024-04-26T23:36:01.182 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-26T23:36:01.182 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":2,"fsid":"3c8453ae-0425-11ef-bc93-c7b262605968","modified":"2024-04-26T23:35:54.532964Z","created":"2024-04-26T23:33:30.337416Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:3300","nonce":0},{"type":"v1","addr":"172.21.15.72:6789","nonce":0}]},"addr":"172.21.15.72:6789/0","public_addr":"172.21.15.72:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:3300","nonce":0},{"type":"v1","addr":"172.21.15.136:6789","nonce":0}]},"addr":"172.21.15.136:6789/0","public_addr":"172.21.15.136:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-26T23:36:01.184 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:01.180+0000 7f60b84f1700 1 -- 172.21.15.136:0/3622902924 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6094039590 msgr2=0x7f609403ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:36:01.184 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:01.180+0000 7f60b84f1700 1 --2- 172.21.15.136:0/3622902924 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6094039590 0x7f609403ba50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f60ac009180 tx=0x7f60ac008040 comp rx=0 tx=0).stop 2024-04-26T23:36:01.184 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:01.180+0000 7f60b84f1700 1 -- 172.21.15.136:0/3622902924 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f609c0035c0 msgr2=0x7f60b0103b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:36:01.184 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:01.180+0000 7f60b84f1700 1 --2- 172.21.15.136:0/3622902924 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f609c0035c0 0x7f60b0103b30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f60a0010010 tx=0x7f60a0005cc0 comp rx=0 tx=0).stop 2024-04-26T23:36:01.184 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:01.180+0000 7f60b84f1700 1 -- 172.21.15.136:0/3622902924 shutdown_connections 2024-04-26T23:36:01.184 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:01.180+0000 7f60b84f1700 1 --2- 172.21.15.136:0/3622902924 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6094039590 0x7f609403ba50 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:36:01.184 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:01.180+0000 7f60b84f1700 1 --2- 172.21.15.136:0/3622902924 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f609c0035c0 0x7f60b0103b30 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:36:01.184 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:01.181+0000 7f60b84f1700 1 --2- 172.21.15.136:0/3622902924 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f60b0104070 0x7f60b01044f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:36:01.184 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:01.181+0000 7f60b84f1700 1 -- 172.21.15.136:0/3622902924 >> 172.21.15.136:0/3622902924 conn(0x7f60b00fd920 msgr2=0x7f60b018f420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:36:01.184 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:01.181+0000 7f60b84f1700 1 -- 172.21.15.136:0/3622902924 shutdown_connections 2024-04-26T23:36:01.184 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:01.181+0000 7f60b84f1700 1 -- 172.21.15.136:0/3622902924 wait complete. 2024-04-26T23:36:01.185 INFO:teuthology.orchestra.run.smithi136.stderr:dumped monmap epoch 2 2024-04-26T23:36:01.759 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:01 smithi136 ceph-mon[33050]: from='client.? 172.21.15.136:0/3622902924' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T23:36:02.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:01 smithi072 ceph-mon[28184]: from='client.? 172.21.15.136:0/3622902924' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T23:36:02.788 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-26T23:36:02.788 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-26T23:36:03.032 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T23:36:03.045 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:02 smithi072 ceph-mon[28184]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:03.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:02 smithi136 ceph-mon[33050]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:03.375 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:03 smithi076 ceph-mon[32774]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-26T23:36:03.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:03 smithi076 bash[32755]: 470c048cdec7ab5d3ee3267ed1b3b765f829a6135ebe2e39b308013a6ff9f05c 2024-04-26T23:36:03.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:03 smithi076 systemd[1]: Started Ceph mon.b for 3c8453ae-0425-11ef-bc93-c7b262605968. 2024-04-26T23:36:08.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:08 smithi072 ceph-mon[28184]: mon.c calling monitor election 2024-04-26T23:36:08.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:08 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T23:36:08.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:08 smithi072 ceph-mon[28184]: mon.a calling monitor election 2024-04-26T23:36:08.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:08 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:36:08.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:08 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:36:08.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:08 smithi072 ceph-mon[28184]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:08.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:08 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:36:08.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:08 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:36:08.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:08 smithi072 ceph-mon[28184]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:08.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:08 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:36:08.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:08 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:36:08.796 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:08 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:36:08.796 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:08 smithi072 ceph-mon[28184]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T23:36:08.796 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:08 smithi072 ceph-mon[28184]: monmap epoch 3 2024-04-26T23:36:08.796 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:08 smithi072 ceph-mon[28184]: fsid 3c8453ae-0425-11ef-bc93-c7b262605968 2024-04-26T23:36:08.796 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:08 smithi072 ceph-mon[28184]: last_changed 2024-04-26T23:36:03.346572+0000 2024-04-26T23:36:08.796 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:08 smithi072 ceph-mon[28184]: created 2024-04-26T23:33:30.337416+0000 2024-04-26T23:36:08.796 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:08 smithi072 ceph-mon[28184]: min_mon_release 19 (squid) 2024-04-26T23:36:08.796 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:08 smithi072 ceph-mon[28184]: election_strategy: 1 2024-04-26T23:36:08.796 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:08 smithi072 ceph-mon[28184]: 0: [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] mon.a 2024-04-26T23:36:08.796 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:08 smithi072 ceph-mon[28184]: 1: [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] mon.c 2024-04-26T23:36:08.796 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:08 smithi072 ceph-mon[28184]: 2: [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] mon.b 2024-04-26T23:36:08.796 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:08 smithi072 ceph-mon[28184]: fsmap 2024-04-26T23:36:08.796 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:08 smithi072 ceph-mon[28184]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T23:36:08.796 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:08 smithi072 ceph-mon[28184]: mgrmap e13: x(active, since 70s) 2024-04-26T23:36:08.797 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:08 smithi072 ceph-mon[28184]: overall HEALTH_OK 2024-04-26T23:36:08.797 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:08 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:08.797 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:08 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:08.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:08 smithi136 ceph-mon[33050]: mon.c calling monitor election 2024-04-26T23:36:08.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:08 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T23:36:08.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:08 smithi136 ceph-mon[33050]: mon.a calling monitor election 2024-04-26T23:36:08.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:08 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:36:08.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:08 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:36:08.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:08 smithi136 ceph-mon[33050]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:08.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:08 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:36:08.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:08 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:36:08.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:08 smithi136 ceph-mon[33050]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:08.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:08 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:36:08.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:08 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:36:08.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:08 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:36:08.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:08 smithi136 ceph-mon[33050]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T23:36:08.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:08 smithi136 ceph-mon[33050]: monmap epoch 3 2024-04-26T23:36:08.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:08 smithi136 ceph-mon[33050]: fsid 3c8453ae-0425-11ef-bc93-c7b262605968 2024-04-26T23:36:08.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:08 smithi136 ceph-mon[33050]: last_changed 2024-04-26T23:36:03.346572+0000 2024-04-26T23:36:08.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:08 smithi136 ceph-mon[33050]: created 2024-04-26T23:33:30.337416+0000 2024-04-26T23:36:08.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:08 smithi136 ceph-mon[33050]: min_mon_release 19 (squid) 2024-04-26T23:36:08.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:08 smithi136 ceph-mon[33050]: election_strategy: 1 2024-04-26T23:36:08.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:08 smithi136 ceph-mon[33050]: 0: [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] mon.a 2024-04-26T23:36:08.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:08 smithi136 ceph-mon[33050]: 1: [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] mon.c 2024-04-26T23:36:08.802 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:08 smithi136 ceph-mon[33050]: 2: [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] mon.b 2024-04-26T23:36:08.802 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:08 smithi136 ceph-mon[33050]: fsmap 2024-04-26T23:36:08.802 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:08 smithi136 ceph-mon[33050]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T23:36:08.802 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:08 smithi136 ceph-mon[33050]: mgrmap e13: x(active, since 70s) 2024-04-26T23:36:08.802 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:08 smithi136 ceph-mon[33050]: overall HEALTH_OK 2024-04-26T23:36:08.802 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:08 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:08.802 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:08 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:08.875 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:08.868+0000 7f6dcb4ae700 1 -- 172.21.15.136:0/1272132097 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6db00035c0 msgr2=0x7f6db0005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:36:08.875 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:08.868+0000 7f6dcb4ae700 1 --2- 172.21.15.136:0/1272132097 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6db00035c0 0x7f6db0005a50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6dc406b710 tx=0x7f6dbc00fe60 comp rx=0 tx=0).stop 2024-04-26T23:36:08.875 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:08.868+0000 7f6dcb4ae700 1 -- 172.21.15.136:0/1272132097 shutdown_connections 2024-04-26T23:36:08.875 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:08.868+0000 7f6dcb4ae700 1 --2- 172.21.15.136:0/1272132097 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6db00035c0 0x7f6db0005a50 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:36:08.875 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:08.868+0000 7f6dcb4ae700 1 -- 172.21.15.136:0/1272132097 >> 172.21.15.136:0/1272132097 conn(0x7f6dc40fd910 msgr2=0x7f6dc40ffd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:36:08.875 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:08.868+0000 7f6dcb4ae700 1 -- 172.21.15.136:0/1272132097 shutdown_connections 2024-04-26T23:36:08.875 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:08.868+0000 7f6dcb4ae700 1 -- 172.21.15.136:0/1272132097 wait complete. 2024-04-26T23:36:08.875 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:08.869+0000 7f6dcb4ae700 1 Processor -- start 2024-04-26T23:36:08.875 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:08.869+0000 7f6dcb4ae700 1 -- start start 2024-04-26T23:36:08.875 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:08.869+0000 7f6dcb4ae700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6db00035c0 0x7f6dc419e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:36:08.876 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:08.869+0000 7f6dcb4ae700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6dc419ed10 0x7f6dc41989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:36:08.876 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:08.869+0000 7f6dcb4ae700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6dc4198f20 0x7f6dc41993d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:36:08.876 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:08.869+0000 7f6dcb4ae700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f6dc410ae20 con 0x7f6dc4198f20 2024-04-26T23:36:08.876 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:08.869+0000 7f6dcb4ae700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f6dc410aca0 con 0x7f6dc419ed10 2024-04-26T23:36:08.876 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:08.869+0000 7f6dcb4ae700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f6dc410afa0 con 0x7f6db00035c0 2024-04-26T23:36:08.876 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:08.869+0000 7f6dc924a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6db00035c0 0x7f6dc419e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:36:08.876 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:08.869+0000 7f6dc924a700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6db00035c0 0x7f6dc419e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:49384/0 (socket says 172.21.15.136:49384) 2024-04-26T23:36:08.876 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:08.869+0000 7f6dc924a700 1 -- 172.21.15.136:0/3161686585 learned_addr learned my addr 172.21.15.136:0/3161686585 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-26T23:36:08.876 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:08.870+0000 7f6dc924a700 1 -- 172.21.15.136:0/3161686585 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6db00035c0 msgr2=0x7f6dc419e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T23:36:08.876 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:08.870+0000 7f6dc924a700 1 -- 172.21.15.136:0/3161686585 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6db00035c0 msgr2=0x7f6dc419e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T23:36:08.876 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:08.870+0000 7f6dc9a4b700 1 --2- 172.21.15.136:0/3161686585 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6dc4198f20 0x7f6dc41993d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:36:08.876 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:08.870+0000 7f6dc924a700 1 --2- 172.21.15.136:0/3161686585 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6db00035c0 0x7f6dc419e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T23:36:08.876 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:08.870+0000 7f6dc8a49700 1 --2- 172.21.15.136:0/3161686585 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6dc419ed10 0x7f6dc41989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:36:08.876 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:08.870+0000 7f6dc924a700 1 --2- 172.21.15.136:0/3161686585 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6db00035c0 0x7f6dc419e7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T23:36:08.877 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:08.870+0000 7f6dc8a49700 1 -- 172.21.15.136:0/3161686585 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6dc419ed10 msgr2=0x7f6dc41989e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-26T23:36:08.877 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:08.870+0000 7f6dc8a49700 1 -- 172.21.15.136:0/3161686585 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6dc419ed10 msgr2=0x7f6dc41989e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T23:36:08.877 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:08.870+0000 7f6dc8a49700 1 --2- 172.21.15.136:0/3161686585 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6dc419ed10 0x7f6dc41989e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T23:36:08.877 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:08.870+0000 7f6dc8a49700 1 --2- 172.21.15.136:0/3161686585 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6dc419ed10 0x7f6dc41989e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T23:36:08.877 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:08.870+0000 7f6dc9a4b700 1 -- 172.21.15.136:0/3161686585 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6db00035c0 msgr2=0x7f6dc419e7d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T23:36:08.877 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:08.870+0000 7f6dc9a4b700 1 --2- 172.21.15.136:0/3161686585 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6db00035c0 0x7f6dc419e7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:36:08.877 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:08.870+0000 7f6dc9a4b700 1 -- 172.21.15.136:0/3161686585 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6dc419ed10 msgr2=0x7f6dc41989e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T23:36:08.877 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:08.870+0000 7f6dc9a4b700 1 --2- 172.21.15.136:0/3161686585 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6dc419ed10 0x7f6dc41989e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:36:08.877 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:08.870+0000 7f6dc9a4b700 1 -- 172.21.15.136:0/3161686585 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dc41a56a0 con 0x7f6dc4198f20 2024-04-26T23:36:08.877 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:08.870+0000 7f6dc9a4b700 1 --2- 172.21.15.136:0/3161686585 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6dc4198f20 0x7f6dc41993d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6dc000bd50 tx=0x7f6dc000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:36:08.877 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:08.871+0000 7f6dba7fc700 1 -- 172.21.15.136:0/3161686585 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dc0019070 con 0x7f6dc4198f20 2024-04-26T23:36:08.877 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:08.871+0000 7f6dba7fc700 1 -- 172.21.15.136:0/3161686585 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6dc000cd10 con 0x7f6dc4198f20 2024-04-26T23:36:08.877 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:08.871+0000 7f6dcb4ae700 1 -- 172.21.15.136:0/3161686585 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dc41a58d0 con 0x7f6dc4198f20 2024-04-26T23:36:08.878 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:08.871+0000 7f6dba7fc700 1 -- 172.21.15.136:0/3161686585 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dc0012700 con 0x7f6dc4198f20 2024-04-26T23:36:08.878 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:08.871+0000 7f6dcb4ae700 1 -- 172.21.15.136:0/3161686585 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dc41a5dc0 con 0x7f6dc4198f20 2024-04-26T23:36:08.879 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:08.872+0000 7f6dcb4ae700 1 -- 172.21.15.136:0/3161686585 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6dc4064a80 con 0x7f6dc4198f20 2024-04-26T23:36:08.879 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:08.872+0000 7f6dba7fc700 1 -- 172.21.15.136:0/3161686585 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f6dc001c070 con 0x7f6dc4198f20 2024-04-26T23:36:08.879 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:08.872+0000 7f6dba7fc700 1 --2- 172.21.15.136:0/3161686585 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6da8039ae0 0x7f6da803bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:36:08.879 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:08.872+0000 7f6dba7fc700 1 -- 172.21.15.136:0/3161686585 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f6dc004c130 con 0x7f6dc4198f20 2024-04-26T23:36:08.879 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:08.873+0000 7f6dc924a700 1 --2- 172.21.15.136:0/3161686585 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6da8039ae0 0x7f6da803bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:36:08.879 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:08.873+0000 7f6dc924a700 1 --2- 172.21.15.136:0/3161686585 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6da8039ae0 0x7f6da803bfa0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6dbc00fa10 tx=0x7f6dbc005ad0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:36:08.884 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:08.880+0000 7f6dba7fc700 1 -- 172.21.15.136:0/3161686585 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6dc0014a10 con 0x7f6dc4198f20 2024-04-26T23:36:09.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:09.170+0000 7f6dcb4ae700 1 -- 172.21.15.136:0/3161686585 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f6dc41a60e0 con 0x7f6dc4198f20 2024-04-26T23:36:09.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:09.171+0000 7f6dba7fc700 1 -- 172.21.15.136:0/3161686585 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1279 (secure 0 0 0) 0x7f6dc0014440 con 0x7f6dc4198f20 2024-04-26T23:36:09.176 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-04-26T23:36:09.176 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":3,"fsid":"3c8453ae-0425-11ef-bc93-c7b262605968","modified":"2024-04-26T23:36:03.346572Z","created":"2024-04-26T23:33:30.337416Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:3300","nonce":0},{"type":"v1","addr":"172.21.15.72:6789","nonce":0}]},"addr":"172.21.15.72:6789/0","public_addr":"172.21.15.72:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:3300","nonce":0},{"type":"v1","addr":"172.21.15.136:6789","nonce":0}]},"addr":"172.21.15.136:6789/0","public_addr":"172.21.15.136:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:3300","nonce":0},{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-26T23:36:09.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:09.175+0000 7f6dcb4ae700 1 -- 172.21.15.136:0/3161686585 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6da8039ae0 msgr2=0x7f6da803bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:36:09.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:09.175+0000 7f6dcb4ae700 1 --2- 172.21.15.136:0/3161686585 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6da8039ae0 0x7f6da803bfa0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6dbc00fa10 tx=0x7f6dbc005ad0 comp rx=0 tx=0).stop 2024-04-26T23:36:09.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:09.175+0000 7f6dcb4ae700 1 -- 172.21.15.136:0/3161686585 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6dc4198f20 msgr2=0x7f6dc41993d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:36:09.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:09.175+0000 7f6dcb4ae700 1 --2- 172.21.15.136:0/3161686585 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6dc4198f20 0x7f6dc41993d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6dc000bd50 tx=0x7f6dc000be30 comp rx=0 tx=0).stop 2024-04-26T23:36:09.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:09.175+0000 7f6dcb4ae700 1 -- 172.21.15.136:0/3161686585 shutdown_connections 2024-04-26T23:36:09.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:09.175+0000 7f6dcb4ae700 1 --2- 172.21.15.136:0/3161686585 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6da8039ae0 0x7f6da803bfa0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:36:09.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:09.175+0000 7f6dcb4ae700 1 --2- 172.21.15.136:0/3161686585 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6dc4198f20 0x7f6dc41993d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:36:09.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:09.175+0000 7f6dcb4ae700 1 --2- 172.21.15.136:0/3161686585 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6db00035c0 0x7f6dc419e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:36:09.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:09.175+0000 7f6dcb4ae700 1 --2- 172.21.15.136:0/3161686585 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6dc419ed10 0x7f6dc41989e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:36:09.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:09.175+0000 7f6dcb4ae700 1 -- 172.21.15.136:0/3161686585 >> 172.21.15.136:0/3161686585 conn(0x7f6dc40fd910 msgr2=0x7f6dc41079b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:36:09.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:09.175+0000 7f6dcb4ae700 1 -- 172.21.15.136:0/3161686585 shutdown_connections 2024-04-26T23:36:09.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:09.175+0000 7f6dcb4ae700 1 -- 172.21.15.136:0/3161686585 wait complete. 2024-04-26T23:36:09.180 INFO:teuthology.orchestra.run.smithi136.stderr:dumped monmap epoch 3 2024-04-26T23:36:09.754 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:09 smithi136 ceph-mon[33050]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:09.755 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:09 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:09.755 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:09 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:09.755 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:09 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:36:09.755 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:09 smithi136 ceph-mon[33050]: from='client.? 172.21.15.136:0/3161686585' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T23:36:09.755 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:09 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:36:09.786 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-26T23:36:09.786 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-26T23:36:09.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:09 smithi072 ceph-mon[28184]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:09.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:09 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:09.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:09 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:09.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:09 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:36:09.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:09 smithi072 ceph-mon[28184]: from='client.? 172.21.15.136:0/3161686585' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T23:36:09.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:09 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:36:10.218 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:36:10.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: mon.c calling monitor election 2024-04-26T23:36:10.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T23:36:10.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: mon.a calling monitor election 2024-04-26T23:36:10.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:36:10.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:36:10.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:10.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:36:10.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:36:10.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:10.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:36:10.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:36:10.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:36:10.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T23:36:10.680 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: monmap epoch 3 2024-04-26T23:36:10.680 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: fsid 3c8453ae-0425-11ef-bc93-c7b262605968 2024-04-26T23:36:10.680 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: last_changed 2024-04-26T23:36:03.346572+0000 2024-04-26T23:36:10.680 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: created 2024-04-26T23:33:30.337416+0000 2024-04-26T23:36:10.680 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: min_mon_release 19 (squid) 2024-04-26T23:36:10.680 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: election_strategy: 1 2024-04-26T23:36:10.680 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: 0: [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] mon.a 2024-04-26T23:36:10.680 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: 1: [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] mon.c 2024-04-26T23:36:10.680 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: 2: [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] mon.b 2024-04-26T23:36:10.680 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: fsmap 2024-04-26T23:36:10.680 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T23:36:10.680 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: mgrmap e13: x(active, since 70s) 2024-04-26T23:36:10.680 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: overall HEALTH_OK 2024-04-26T23:36:10.681 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:10.681 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:10.681 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:10.681 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:10.681 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:10.681 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:36:10.681 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: from='client.? 172.21.15.136:0/3161686585' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T23:36:10.681 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:36:11.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:10 smithi072 ceph-mon[28184]: mon.b calling monitor election 2024-04-26T23:36:11.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:10 smithi072 ceph-mon[28184]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:11.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:10 smithi072 ceph-mon[28184]: mon.b calling monitor election 2024-04-26T23:36:11.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:10 smithi072 ceph-mon[28184]: mon.c calling monitor election 2024-04-26T23:36:11.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:10 smithi072 ceph-mon[28184]: mon.a calling monitor election 2024-04-26T23:36:11.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:10 smithi072 ceph-mon[28184]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-26T23:36:11.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:10 smithi072 ceph-mon[28184]: monmap epoch 3 2024-04-26T23:36:11.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:10 smithi072 ceph-mon[28184]: fsid 3c8453ae-0425-11ef-bc93-c7b262605968 2024-04-26T23:36:11.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:10 smithi072 ceph-mon[28184]: last_changed 2024-04-26T23:36:03.346572+0000 2024-04-26T23:36:11.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:10 smithi072 ceph-mon[28184]: created 2024-04-26T23:33:30.337416+0000 2024-04-26T23:36:11.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:10 smithi072 ceph-mon[28184]: min_mon_release 19 (squid) 2024-04-26T23:36:11.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:10 smithi072 ceph-mon[28184]: election_strategy: 1 2024-04-26T23:36:11.295 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:10 smithi072 ceph-mon[28184]: 0: [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] mon.a 2024-04-26T23:36:11.295 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:10 smithi072 ceph-mon[28184]: 1: [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] mon.c 2024-04-26T23:36:11.295 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:10 smithi072 ceph-mon[28184]: 2: [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] mon.b 2024-04-26T23:36:11.295 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:10 smithi072 ceph-mon[28184]: fsmap 2024-04-26T23:36:11.295 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:10 smithi072 ceph-mon[28184]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T23:36:11.295 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:10 smithi072 ceph-mon[28184]: mgrmap e13: x(active, since 72s) 2024-04-26T23:36:11.295 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:10 smithi072 ceph-mon[28184]: overall HEALTH_OK 2024-04-26T23:36:11.295 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:10 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:11.295 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:10 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:11.295 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:10 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:36:11.295 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:10 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:36:11.295 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:10 smithi072 ceph-mon[28184]: Updating smithi072:/etc/ceph/ceph.conf 2024-04-26T23:36:11.295 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:10 smithi072 ceph-mon[28184]: Updating smithi076:/etc/ceph/ceph.conf 2024-04-26T23:36:11.295 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:10 smithi072 ceph-mon[28184]: Updating smithi136:/etc/ceph/ceph.conf 2024-04-26T23:36:11.299 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:10 smithi136 ceph-mon[33050]: mon.b calling monitor election 2024-04-26T23:36:11.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:10 smithi136 ceph-mon[33050]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:11.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:10 smithi136 ceph-mon[33050]: mon.b calling monitor election 2024-04-26T23:36:11.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:10 smithi136 ceph-mon[33050]: mon.c calling monitor election 2024-04-26T23:36:11.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:10 smithi136 ceph-mon[33050]: mon.a calling monitor election 2024-04-26T23:36:11.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:10 smithi136 ceph-mon[33050]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-26T23:36:11.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:10 smithi136 ceph-mon[33050]: monmap epoch 3 2024-04-26T23:36:11.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:10 smithi136 ceph-mon[33050]: fsid 3c8453ae-0425-11ef-bc93-c7b262605968 2024-04-26T23:36:11.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:10 smithi136 ceph-mon[33050]: last_changed 2024-04-26T23:36:03.346572+0000 2024-04-26T23:36:11.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:10 smithi136 ceph-mon[33050]: created 2024-04-26T23:33:30.337416+0000 2024-04-26T23:36:11.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:10 smithi136 ceph-mon[33050]: min_mon_release 19 (squid) 2024-04-26T23:36:11.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:10 smithi136 ceph-mon[33050]: election_strategy: 1 2024-04-26T23:36:11.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:10 smithi136 ceph-mon[33050]: 0: [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] mon.a 2024-04-26T23:36:11.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:10 smithi136 ceph-mon[33050]: 1: [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] mon.c 2024-04-26T23:36:11.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:10 smithi136 ceph-mon[33050]: 2: [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] mon.b 2024-04-26T23:36:11.301 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:10 smithi136 ceph-mon[33050]: fsmap 2024-04-26T23:36:11.301 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:10 smithi136 ceph-mon[33050]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T23:36:11.301 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:10 smithi136 ceph-mon[33050]: mgrmap e13: x(active, since 72s) 2024-04-26T23:36:11.301 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:10 smithi136 ceph-mon[33050]: overall HEALTH_OK 2024-04-26T23:36:11.301 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:10 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:11.301 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:10 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:11.301 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:10 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:36:11.301 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:10 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:36:11.301 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:10 smithi136 ceph-mon[33050]: Updating smithi072:/etc/ceph/ceph.conf 2024-04-26T23:36:11.301 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:10 smithi136 ceph-mon[33050]: Updating smithi076:/etc/ceph/ceph.conf 2024-04-26T23:36:11.301 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:10 smithi136 ceph-mon[33050]: Updating smithi136:/etc/ceph/ceph.conf 2024-04-26T23:36:11.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: mon.b calling monitor election 2024-04-26T23:36:11.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:11.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: mon.b calling monitor election 2024-04-26T23:36:11.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: mon.c calling monitor election 2024-04-26T23:36:11.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: mon.a calling monitor election 2024-04-26T23:36:11.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-26T23:36:11.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: monmap epoch 3 2024-04-26T23:36:11.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: fsid 3c8453ae-0425-11ef-bc93-c7b262605968 2024-04-26T23:36:11.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: last_changed 2024-04-26T23:36:03.346572+0000 2024-04-26T23:36:11.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: created 2024-04-26T23:33:30.337416+0000 2024-04-26T23:36:11.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: min_mon_release 19 (squid) 2024-04-26T23:36:11.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: election_strategy: 1 2024-04-26T23:36:11.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: 0: [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] mon.a 2024-04-26T23:36:11.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: 1: [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] mon.c 2024-04-26T23:36:11.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: 2: [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] mon.b 2024-04-26T23:36:11.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: fsmap 2024-04-26T23:36:11.430 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T23:36:11.430 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: mgrmap e13: x(active, since 72s) 2024-04-26T23:36:11.430 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: overall HEALTH_OK 2024-04-26T23:36:11.430 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:11.430 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:11.430 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:36:11.430 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:36:11.430 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: Updating smithi072:/etc/ceph/ceph.conf 2024-04-26T23:36:11.430 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: Updating smithi076:/etc/ceph/ceph.conf 2024-04-26T23:36:11.430 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:10 smithi076 ceph-mon[32774]: Updating smithi136:/etc/ceph/ceph.conf 2024-04-26T23:36:12.274 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:12 smithi136 ceph-mon[33050]: Updating smithi136:/var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T23:36:12.274 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:12 smithi136 ceph-mon[33050]: Updating smithi076:/var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T23:36:12.274 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:12 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:12.275 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:12 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:12.275 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:12 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:12.275 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:12 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:12.275 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:12 smithi136 ceph-mon[33050]: Updating smithi072:/var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T23:36:12.275 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:12 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:36:12.275 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:12 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:12.275 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:12 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:12.275 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:12 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:12.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.308+0000 7f9fef600700 1 -- 172.21.15.72:0/1542880324 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9fe8101b60 msgr2=0x7f9fe8101f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:36:12.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.308+0000 7f9fef600700 1 --2- 172.21.15.72:0/1542880324 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9fe8101b60 0x7f9fe8101f40 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc009910 tx=0x7f9fdc009c20 comp rx=0 tx=0).stop 2024-04-26T23:36:12.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.308+0000 7f9fef600700 1 -- 172.21.15.72:0/1542880324 shutdown_connections 2024-04-26T23:36:12.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.308+0000 7f9fef600700 1 --2- 172.21.15.72:0/1542880324 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9fe8101b60 0x7f9fe8101f40 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:36:12.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.308+0000 7f9fef600700 1 -- 172.21.15.72:0/1542880324 >> 172.21.15.72:0/1542880324 conn(0x7f9fe80fd820 msgr2=0x7f9fe80ffc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:36:12.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.308+0000 7f9fef600700 1 -- 172.21.15.72:0/1542880324 shutdown_connections 2024-04-26T23:36:12.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.308+0000 7f9fef600700 1 -- 172.21.15.72:0/1542880324 wait complete. 2024-04-26T23:36:12.313 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:12 smithi072 ceph-mon[28184]: Updating smithi136:/var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T23:36:12.313 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:12 smithi072 ceph-mon[28184]: Updating smithi076:/var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T23:36:12.313 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:12 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:12.313 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:12 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:12.313 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:12 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:12.313 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:12 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:12.313 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:12 smithi072 ceph-mon[28184]: Updating smithi072:/var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T23:36:12.313 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:12 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:36:12.313 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:12 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:12.313 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:12 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:12.314 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:12 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:12.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.310+0000 7f9fef600700 1 Processor -- start 2024-04-26T23:36:12.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.310+0000 7f9fef600700 1 -- start start 2024-04-26T23:36:12.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.310+0000 7f9fef600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9fe8101b60 0x7f9fe819ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:36:12.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.310+0000 7f9fef600700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9fe819b260 0x7f9fe819f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:36:12.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.310+0000 7f9fef600700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9fe819fb30 0x7f9fe819ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:36:12.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.310+0000 7f9fef600700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe810f550 con 0x7f9fe819b260 2024-04-26T23:36:12.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.310+0000 7f9fef600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe810f3d0 con 0x7f9fe8101b60 2024-04-26T23:36:12.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.310+0000 7f9fef600700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe810f6d0 con 0x7f9fe819fb30 2024-04-26T23:36:12.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.311+0000 7f9fe7e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9fe819b260 0x7f9fe819f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:36:12.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.311+0000 7f9fe7e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9fe819b260 0x7f9fe819f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54996/0 (socket says 172.21.15.72:54996) 2024-04-26T23:36:12.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.311+0000 7f9fe7e00700 1 -- 172.21.15.72:0/3797489899 learned_addr learned my addr 172.21.15.72:0/3797489899 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:36:12.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.311+0000 7f9fecc00700 1 --2- 172.21.15.72:0/3797489899 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9fe8101b60 0x7f9fe819ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:36:12.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.311+0000 7f9fed600700 1 --2- 172.21.15.72:0/3797489899 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9fe819fb30 0x7f9fe819ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:36:12.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.311+0000 7f9fe7e00700 1 -- 172.21.15.72:0/3797489899 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9fe819fb30 msgr2=0x7f9fe819ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:36:12.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.311+0000 7f9fe7e00700 1 --2- 172.21.15.72:0/3797489899 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9fe819fb30 0x7f9fe819ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:36:12.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.311+0000 7f9fe7e00700 1 -- 172.21.15.72:0/3797489899 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9fe8101b60 msgr2=0x7f9fe819ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:36:12.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.311+0000 7f9fe7e00700 1 --2- 172.21.15.72:0/3797489899 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9fe8101b60 0x7f9fe819ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:36:12.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.311+0000 7f9fe7e00700 1 -- 172.21.15.72:0/3797489899 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fe81a78a0 con 0x7f9fe819b260 2024-04-26T23:36:12.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.312+0000 7f9fe7e00700 1 --2- 172.21.15.72:0/3797489899 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9fe819b260 0x7f9fe819f5f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9fd400b780 tx=0x7f9fd400ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:36:12.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.312+0000 7f9fe5600700 1 -- 172.21.15.72:0/3797489899 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd4016070 con 0x7f9fe819b260 2024-04-26T23:36:12.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.312+0000 7f9fe5600700 1 -- 172.21.15.72:0/3797489899 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9fd40036a0 con 0x7f9fe819b260 2024-04-26T23:36:12.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.312+0000 7f9fe5600700 1 -- 172.21.15.72:0/3797489899 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd40119c0 con 0x7f9fe819b260 2024-04-26T23:36:12.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.312+0000 7f9fef600700 1 -- 172.21.15.72:0/3797489899 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fe81a7ad0 con 0x7f9fe819b260 2024-04-26T23:36:12.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.313+0000 7f9fef600700 1 -- 172.21.15.72:0/3797489899 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fe81a7fc0 con 0x7f9fe819b260 2024-04-26T23:36:12.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.315+0000 7f9fe5600700 1 -- 172.21.15.72:0/3797489899 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f9fd4019070 con 0x7f9fe819b260 2024-04-26T23:36:12.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.315+0000 7f9fef600700 1 -- 172.21.15.72:0/3797489899 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fe8064a80 con 0x7f9fe819b260 2024-04-26T23:36:12.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.315+0000 7f9fe5600700 1 --2- 172.21.15.72:0/3797489899 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f9fd0039ae0 0x7f9fd003bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:36:12.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.315+0000 7f9fe5600700 1 -- 172.21.15.72:0/3797489899 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f9fd404b450 con 0x7f9fe819b260 2024-04-26T23:36:12.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.315+0000 7f9fecc00700 1 --2- 172.21.15.72:0/3797489899 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f9fd0039ae0 0x7f9fd003bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:36:12.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.317+0000 7f9fecc00700 1 --2- 172.21.15.72:0/3797489899 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f9fd0039ae0 0x7f9fd003bfa0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc006010 tx=0x7f9fdc01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:36:12.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.322+0000 7f9fe5600700 1 -- 172.21.15.72:0/3797489899 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9fd4010380 con 0x7f9fe819b260 2024-04-26T23:36:12.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:12 smithi076 ceph-mon[32774]: Updating smithi136:/var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T23:36:12.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:12 smithi076 ceph-mon[32774]: Updating smithi076:/var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T23:36:12.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:12 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:12.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:12 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:12.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:12 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:12.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:12 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:12.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:12 smithi076 ceph-mon[32774]: Updating smithi072:/var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T23:36:12.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:12 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:36:12.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:12 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:12.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:12 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:12.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:12 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:12.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.549+0000 7f9fef600700 1 -- 172.21.15.72:0/3797489899 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f9fe81a8370 con 0x7f9fe819b260 2024-04-26T23:36:12.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.550+0000 7f9fe5600700 1 -- 172.21.15.72:0/3797489899 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+273 (secure 0 0 0) 0x7f9fd4014120 con 0x7f9fe819b260 2024-04-26T23:36:12.554 INFO:teuthology.orchestra.run.smithi072.stdout:# minimal ceph.conf for 3c8453ae-0425-11ef-bc93-c7b262605968 2024-04-26T23:36:12.554 INFO:teuthology.orchestra.run.smithi072.stdout:[global] 2024-04-26T23:36:12.554 INFO:teuthology.orchestra.run.smithi072.stdout: fsid = 3c8453ae-0425-11ef-bc93-c7b262605968 2024-04-26T23:36:12.554 INFO:teuthology.orchestra.run.smithi072.stdout: mon_host = [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] 2024-04-26T23:36:12.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.553+0000 7f9fef600700 1 -- 172.21.15.72:0/3797489899 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f9fd0039ae0 msgr2=0x7f9fd003bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:36:12.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.553+0000 7f9fef600700 1 --2- 172.21.15.72:0/3797489899 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f9fd0039ae0 0x7f9fd003bfa0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc006010 tx=0x7f9fdc01a040 comp rx=0 tx=0).stop 2024-04-26T23:36:12.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.553+0000 7f9fef600700 1 -- 172.21.15.72:0/3797489899 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9fe819b260 msgr2=0x7f9fe819f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:36:12.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.553+0000 7f9fef600700 1 --2- 172.21.15.72:0/3797489899 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9fe819b260 0x7f9fe819f5f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9fd400b780 tx=0x7f9fd400ba90 comp rx=0 tx=0).stop 2024-04-26T23:36:12.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.553+0000 7f9fef600700 1 -- 172.21.15.72:0/3797489899 shutdown_connections 2024-04-26T23:36:12.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.553+0000 7f9fef600700 1 --2- 172.21.15.72:0/3797489899 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f9fd0039ae0 0x7f9fd003bfa0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:36:12.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.553+0000 7f9fef600700 1 --2- 172.21.15.72:0/3797489899 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9fe819fb30 0x7f9fe819ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:36:12.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.553+0000 7f9fef600700 1 --2- 172.21.15.72:0/3797489899 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9fe8101b60 0x7f9fe819ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:36:12.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.553+0000 7f9fef600700 1 --2- 172.21.15.72:0/3797489899 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9fe819b260 0x7f9fe819f5f0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:36:12.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.553+0000 7f9fef600700 1 -- 172.21.15.72:0/3797489899 >> 172.21.15.72:0/3797489899 conn(0x7f9fe80fd820 msgr2=0x7f9fe80fe870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:36:12.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.553+0000 7f9fef600700 1 -- 172.21.15.72:0/3797489899 shutdown_connections 2024-04-26T23:36:12.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:12.553+0000 7f9fef600700 1 -- 172.21.15.72:0/3797489899 wait complete. 2024-04-26T23:36:13.243 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:13 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:13.243 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:13 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:13.243 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:13 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:13.243 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:13 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:13.243 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:13 smithi072 ceph-mon[28184]: Reconfiguring mon.a (unknown last config time)... 2024-04-26T23:36:13.243 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:13 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T23:36:13.243 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:13 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T23:36:13.244 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:13 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:36:13.244 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:13 smithi072 ceph-mon[28184]: Reconfiguring daemon mon.a on smithi072 2024-04-26T23:36:13.244 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:13 smithi072 ceph-mon[28184]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:13.244 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:13 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/3797489899' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:36:13.425 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-26T23:36:13.426 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-26T23:36:13.426 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/etc/ceph/ceph.conf 2024-04-26T23:36:13.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:13 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:13.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:13 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:13.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:13 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:13.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:13 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:13.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:13 smithi076 ceph-mon[32774]: Reconfiguring mon.a (unknown last config time)... 2024-04-26T23:36:13.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:13 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T23:36:13.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:13 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T23:36:13.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:13 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:36:13.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:13 smithi076 ceph-mon[32774]: Reconfiguring daemon mon.a on smithi072 2024-04-26T23:36:13.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:13 smithi076 ceph-mon[32774]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:13.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:13 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/3797489899' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:36:13.501 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-26T23:36:13.501 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T23:36:13.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:13 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:13.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:13 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:13.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:13 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:13.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:13 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:13.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:13 smithi136 ceph-mon[33050]: Reconfiguring mon.a (unknown last config time)... 2024-04-26T23:36:13.551 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:13 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T23:36:13.551 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:13 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T23:36:13.551 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:13 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:36:13.551 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:13 smithi136 ceph-mon[33050]: Reconfiguring daemon mon.a on smithi072 2024-04-26T23:36:13.551 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:13 smithi136 ceph-mon[33050]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:13.551 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:13 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/3797489899' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:36:13.611 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-04-26T23:36:13.612 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/ceph/ceph.conf 2024-04-26T23:36:13.642 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-04-26T23:36:13.642 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T23:36:13.709 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-26T23:36:13.709 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/ceph/ceph.conf 2024-04-26T23:36:13.737 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-26T23:36:13.737 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T23:36:13.802 INFO:tasks.cephadm:Adding mgr.x on smithi072 2024-04-26T23:36:13.803 INFO:tasks.cephadm:Adding mgr.y on smithi076 2024-04-26T23:36:13.803 INFO:tasks.cephadm:Adding mgr.z on smithi136 2024-04-26T23:36:13.803 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph orch apply mgr '3;smithi072=x;smithi076=y;smithi136=z' 2024-04-26T23:36:14.071 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T23:36:14.549 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:14 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:14.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:14 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:14.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:14 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:15.416 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.410+0000 7f6b4872d700 1 -- 172.21.15.136:0/963391845 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b40102fb0 msgr2=0x7f6b401033d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:36:15.416 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.410+0000 7f6b4872d700 1 --2- 172.21.15.136:0/963391845 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b40102fb0 0x7f6b401033d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c009910 tx=0x7f6b3c009c20 comp rx=0 tx=0).stop 2024-04-26T23:36:15.416 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.410+0000 7f6b4872d700 1 -- 172.21.15.136:0/963391845 shutdown_connections 2024-04-26T23:36:15.416 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.410+0000 7f6b4872d700 1 --2- 172.21.15.136:0/963391845 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b40102fb0 0x7f6b401033d0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:36:15.416 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.410+0000 7f6b4872d700 1 -- 172.21.15.136:0/963391845 >> 172.21.15.136:0/963391845 conn(0x7f6b400fe550 msgr2=0x7f6b40100990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:36:15.416 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.410+0000 7f6b4872d700 1 -- 172.21.15.136:0/963391845 shutdown_connections 2024-04-26T23:36:15.416 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.410+0000 7f6b4872d700 1 -- 172.21.15.136:0/963391845 wait complete. 2024-04-26T23:36:15.416 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.411+0000 7f6b4872d700 1 Processor -- start 2024-04-26T23:36:15.416 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.411+0000 7f6b4872d700 1 -- start start 2024-04-26T23:36:15.416 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.411+0000 7f6b4872d700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b40102fb0 0x7f6b4019a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:36:15.417 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.411+0000 7f6b4872d700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6b4019ae10 0x7f6b4019fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:36:15.417 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.411+0000 7f6b4872d700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b4019b290 0x7f6b401a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:36:15.417 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.411+0000 7f6b4872d700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f6b40108ef0 con 0x7f6b40102fb0 2024-04-26T23:36:15.417 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.411+0000 7f6b4872d700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f6b40108d70 con 0x7f6b4019b290 2024-04-26T23:36:15.417 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.411+0000 7f6b4872d700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f6b40109070 con 0x7f6b4019ae10 2024-04-26T23:36:15.417 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.411+0000 7f6b45cc8700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6b4019ae10 0x7f6b4019fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:36:15.417 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.411+0000 7f6b45cc8700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6b4019ae10 0x7f6b4019fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:49396/0 (socket says 172.21.15.136:49396) 2024-04-26T23:36:15.417 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.411+0000 7f6b45cc8700 1 -- 172.21.15.136:0/4096669905 learned_addr learned my addr 172.21.15.136:0/4096669905 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-26T23:36:15.417 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.411+0000 7f6b46cca700 1 --2- 172.21.15.136:0/4096669905 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b4019b290 0x7f6b401a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:36:15.417 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.411+0000 7f6b464c9700 1 --2- 172.21.15.136:0/4096669905 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b40102fb0 0x7f6b4019a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:36:15.417 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.411+0000 7f6b45cc8700 1 -- 172.21.15.136:0/4096669905 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b4019b290 msgr2=0x7f6b401a2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:36:15.417 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.411+0000 7f6b45cc8700 1 --2- 172.21.15.136:0/4096669905 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b4019b290 0x7f6b401a2410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:36:15.417 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.411+0000 7f6b45cc8700 1 -- 172.21.15.136:0/4096669905 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b40102fb0 msgr2=0x7f6b4019a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:36:15.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.411+0000 7f6b45cc8700 1 --2- 172.21.15.136:0/4096669905 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b40102fb0 0x7f6b4019a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:36:15.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.411+0000 7f6b45cc8700 1 -- 172.21.15.136:0/4096669905 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b401a2ce0 con 0x7f6b4019ae10 2024-04-26T23:36:15.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.412+0000 7f6b45cc8700 1 --2- 172.21.15.136:0/4096669905 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6b4019ae10 0x7f6b4019fe80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6b3000b780 tx=0x7f6b3000bb40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:36:15.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.412+0000 7f6b377fe700 1 -- 172.21.15.136:0/4096669905 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b30004230 con 0x7f6b4019ae10 2024-04-26T23:36:15.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.412+0000 7f6b377fe700 1 -- 172.21.15.136:0/4096669905 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b300043d0 con 0x7f6b4019ae10 2024-04-26T23:36:15.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.412+0000 7f6b4872d700 1 -- 172.21.15.136:0/4096669905 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b401a2fd0 con 0x7f6b4019ae10 2024-04-26T23:36:15.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.412+0000 7f6b377fe700 1 -- 172.21.15.136:0/4096669905 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b30011bd0 con 0x7f6b4019ae10 2024-04-26T23:36:15.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.412+0000 7f6b4872d700 1 -- 172.21.15.136:0/4096669905 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b401a3550 con 0x7f6b4019ae10 2024-04-26T23:36:15.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.413+0000 7f6b377fe700 1 -- 172.21.15.136:0/4096669905 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f6b3001a400 con 0x7f6b4019ae10 2024-04-26T23:36:15.418 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.413+0000 7f6b4872d700 1 -- 172.21.15.136:0/4096669905 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b40064a80 con 0x7f6b4019ae10 2024-04-26T23:36:15.419 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.413+0000 7f6b377fe700 1 --2- 172.21.15.136:0/4096669905 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6b2c039ae0 0x7f6b2c03bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:36:15.419 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.413+0000 7f6b377fe700 1 -- 172.21.15.136:0/4096669905 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f6b3004c4b0 con 0x7f6b4019ae10 2024-04-26T23:36:15.419 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.413+0000 7f6b464c9700 1 --2- 172.21.15.136:0/4096669905 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6b2c039ae0 0x7f6b2c03bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:36:15.419 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.414+0000 7f6b464c9700 1 --2- 172.21.15.136:0/4096669905 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6b2c039ae0 0x7f6b2c03bfa0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c005e00 tx=0x7f6b3c005260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:36:15.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.418+0000 7f6b377fe700 1 -- 172.21.15.136:0/4096669905 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6b30018350 con 0x7f6b4019ae10 2024-04-26T23:36:15.616 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:15 smithi076 ceph-mon[32774]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:15.616 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:15 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:15.616 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:15 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:15.617 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:15 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T23:36:15.617 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:15 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T23:36:15.617 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:15 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:36:15.632 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.629+0000 7f6b4872d700 1 -- 172.21.15.136:0/4096669905 --> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi072=x;smithi076=y;smithi136=z", "target": ["mon-mgr", ""]}) -- 0x7f6b40063860 con 0x7f6b2c039ae0 2024-04-26T23:36:15.633 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:15 smithi136 ceph-mon[33050]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:15.633 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:15 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:15.633 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:15 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:15.633 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:15 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T23:36:15.633 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:15 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T23:36:15.633 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:15 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:36:15.647 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.644+0000 7f6b377fe700 1 -- 172.21.15.136:0/4096669905 <== mgr.14152 v2:172.21.15.72:6800/4281177709 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f6b40063860 con 0x7f6b2c039ae0 2024-04-26T23:36:15.647 INFO:teuthology.orchestra.run.smithi136.stdout:Scheduled mgr update... 2024-04-26T23:36:15.649 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.646+0000 7f6b4872d700 1 -- 172.21.15.136:0/4096669905 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6b2c039ae0 msgr2=0x7f6b2c03bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:36:15.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.646+0000 7f6b4872d700 1 --2- 172.21.15.136:0/4096669905 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6b2c039ae0 0x7f6b2c03bfa0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c005e00 tx=0x7f6b3c005260 comp rx=0 tx=0).stop 2024-04-26T23:36:15.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.646+0000 7f6b4872d700 1 -- 172.21.15.136:0/4096669905 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6b4019ae10 msgr2=0x7f6b4019fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:36:15.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.646+0000 7f6b4872d700 1 --2- 172.21.15.136:0/4096669905 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6b4019ae10 0x7f6b4019fe80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6b3000b780 tx=0x7f6b3000bb40 comp rx=0 tx=0).stop 2024-04-26T23:36:15.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.646+0000 7f6b4872d700 1 -- 172.21.15.136:0/4096669905 shutdown_connections 2024-04-26T23:36:15.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.646+0000 7f6b4872d700 1 --2- 172.21.15.136:0/4096669905 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6b2c039ae0 0x7f6b2c03bfa0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:36:15.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.646+0000 7f6b4872d700 1 --2- 172.21.15.136:0/4096669905 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b4019b290 0x7f6b401a2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:36:15.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.646+0000 7f6b4872d700 1 --2- 172.21.15.136:0/4096669905 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b40102fb0 0x7f6b4019a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:36:15.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.646+0000 7f6b4872d700 1 --2- 172.21.15.136:0/4096669905 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6b4019ae10 0x7f6b4019fe80 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:36:15.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.646+0000 7f6b4872d700 1 -- 172.21.15.136:0/4096669905 >> 172.21.15.136:0/4096669905 conn(0x7f6b400fe550 msgr2=0x7f6b40100990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:36:15.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.646+0000 7f6b4872d700 1 -- 172.21.15.136:0/4096669905 shutdown_connections 2024-04-26T23:36:15.650 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:36:15.646+0000 7f6b4872d700 1 -- 172.21.15.136:0/4096669905 wait complete. 2024-04-26T23:36:15.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:15 smithi072 ceph-mon[28184]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:15.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:15 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:15.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:15 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:15.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:15 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T23:36:15.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:15 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T23:36:15.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:15 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:36:16.173 DEBUG:teuthology.orchestra.run.smithi076:mgr.y> sudo journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mgr.y.service 2024-04-26T23:36:16.176 DEBUG:teuthology.orchestra.run.smithi136:mgr.z> sudo journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mgr.z.service 2024-04-26T23:36:16.178 INFO:tasks.cephadm:Deploying OSDs... 2024-04-26T23:36:16.178 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-26T23:36:16.178 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T23:36:16.235 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T23:36:16.235 DEBUG:teuthology.orchestra.run.smithi072:> stat /dev/vg_nvme/lv_1 2024-04-26T23:36:16.284 INFO:teuthology.orchestra.run.smithi072.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T23:36:16.284 INFO:teuthology.orchestra.run.smithi072.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T23:36:16.285 INFO:teuthology.orchestra.run.smithi072.stdout:Device: 6h/6d Inode: 630 Links: 1 2024-04-26T23:36:16.285 INFO:teuthology.orchestra.run.smithi072.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T23:36:16.285 INFO:teuthology.orchestra.run.smithi072.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T23:36:16.285 INFO:teuthology.orchestra.run.smithi072.stdout:Access: 2024-04-26 23:35:11.608034248 +0000 2024-04-26T23:36:16.285 INFO:teuthology.orchestra.run.smithi072.stdout:Modify: 2024-04-26 23:28:34.715622168 +0000 2024-04-26T23:36:16.285 INFO:teuthology.orchestra.run.smithi072.stdout:Change: 2024-04-26 23:28:34.715622168 +0000 2024-04-26T23:36:16.285 INFO:teuthology.orchestra.run.smithi072.stdout: Birth: 2024-04-26 23:28:34.715622168 +0000 2024-04-26T23:36:16.285 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T23:36:16.356 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records in 2024-04-26T23:36:16.356 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records out 2024-04-26T23:36:16.356 INFO:teuthology.orchestra.run.smithi072.stderr:512 bytes copied, 0.0005281 s, 970 kB/s 2024-04-26T23:36:16.359 DEBUG:teuthology.orchestra.run.smithi072:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T23:36:16.414 DEBUG:teuthology.orchestra.run.smithi072:> stat /dev/vg_nvme/lv_2 2024-04-26T23:36:16.463 INFO:teuthology.orchestra.run.smithi072.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T23:36:16.463 INFO:teuthology.orchestra.run.smithi072.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T23:36:16.463 INFO:teuthology.orchestra.run.smithi072.stdout:Device: 6h/6d Inode: 648 Links: 1 2024-04-26T23:36:16.463 INFO:teuthology.orchestra.run.smithi072.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T23:36:16.463 INFO:teuthology.orchestra.run.smithi072.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T23:36:16.463 INFO:teuthology.orchestra.run.smithi072.stdout:Access: 2024-04-26 23:35:11.654033312 +0000 2024-04-26T23:36:16.463 INFO:teuthology.orchestra.run.smithi072.stdout:Modify: 2024-04-26 23:28:35.320609920 +0000 2024-04-26T23:36:16.463 INFO:teuthology.orchestra.run.smithi072.stdout:Change: 2024-04-26 23:28:35.320609920 +0000 2024-04-26T23:36:16.463 INFO:teuthology.orchestra.run.smithi072.stdout: Birth: 2024-04-26 23:28:35.320609920 +0000 2024-04-26T23:36:16.463 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T23:36:16.545 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records in 2024-04-26T23:36:16.545 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records out 2024-04-26T23:36:16.545 INFO:teuthology.orchestra.run.smithi072.stderr:512 bytes copied, 0.000377475 s, 1.4 MB/s 2024-04-26T23:36:16.549 DEBUG:teuthology.orchestra.run.smithi072:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T23:36:16.549 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:16 smithi136 ceph-mon[33050]: Reconfiguring mon.b (monmap changed)... 2024-04-26T23:36:16.549 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:16 smithi136 ceph-mon[33050]: Reconfiguring daemon mon.b on smithi076 2024-04-26T23:36:16.549 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:16 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:16.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:16 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:16.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:16 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:16.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:16 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T23:36:16.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:16 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T23:36:16.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:16 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:36:16.607 DEBUG:teuthology.orchestra.run.smithi072:> stat /dev/vg_nvme/lv_3 2024-04-26T23:36:16.657 INFO:teuthology.orchestra.run.smithi072.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T23:36:16.658 INFO:teuthology.orchestra.run.smithi072.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T23:36:16.658 INFO:teuthology.orchestra.run.smithi072.stdout:Device: 6h/6d Inode: 669 Links: 1 2024-04-26T23:36:16.658 INFO:teuthology.orchestra.run.smithi072.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T23:36:16.658 INFO:teuthology.orchestra.run.smithi072.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T23:36:16.658 INFO:teuthology.orchestra.run.smithi072.stdout:Access: 2024-04-26 23:35:11.700032375 +0000 2024-04-26T23:36:16.658 INFO:teuthology.orchestra.run.smithi072.stdout:Modify: 2024-04-26 23:28:35.913597916 +0000 2024-04-26T23:36:16.658 INFO:teuthology.orchestra.run.smithi072.stdout:Change: 2024-04-26 23:28:35.913597916 +0000 2024-04-26T23:36:16.658 INFO:teuthology.orchestra.run.smithi072.stdout: Birth: 2024-04-26 23:28:35.913597916 +0000 2024-04-26T23:36:16.658 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T23:36:16.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:16 smithi076 ceph-mon[32774]: Reconfiguring mon.b (monmap changed)... 2024-04-26T23:36:16.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:16 smithi076 ceph-mon[32774]: Reconfiguring daemon mon.b on smithi076 2024-04-26T23:36:16.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:16 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:16.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:16 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:16.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:16 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:16.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:16 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T23:36:16.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:16 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T23:36:16.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:16 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:36:16.724 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:16 smithi072 ceph-mon[28184]: Reconfiguring mon.b (monmap changed)... 2024-04-26T23:36:16.724 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:16 smithi072 ceph-mon[28184]: Reconfiguring daemon mon.b on smithi076 2024-04-26T23:36:16.724 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:16 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:16.724 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:16 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:16.724 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:16 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:16.724 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:16 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T23:36:16.725 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:16 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T23:36:16.725 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:16 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:36:16.734 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records in 2024-04-26T23:36:16.734 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records out 2024-04-26T23:36:16.734 INFO:teuthology.orchestra.run.smithi072.stderr:512 bytes copied, 0.000529923 s, 966 kB/s 2024-04-26T23:36:16.738 DEBUG:teuthology.orchestra.run.smithi072:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T23:36:16.793 DEBUG:teuthology.orchestra.run.smithi072:> stat /dev/vg_nvme/lv_4 2024-04-26T23:36:16.843 INFO:teuthology.orchestra.run.smithi072.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T23:36:16.843 INFO:teuthology.orchestra.run.smithi072.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T23:36:16.843 INFO:teuthology.orchestra.run.smithi072.stdout:Device: 6h/6d Inode: 689 Links: 1 2024-04-26T23:36:16.843 INFO:teuthology.orchestra.run.smithi072.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T23:36:16.843 INFO:teuthology.orchestra.run.smithi072.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T23:36:16.843 INFO:teuthology.orchestra.run.smithi072.stdout:Access: 2024-04-26 23:35:11.747031418 +0000 2024-04-26T23:36:16.843 INFO:teuthology.orchestra.run.smithi072.stdout:Modify: 2024-04-26 23:28:36.512585790 +0000 2024-04-26T23:36:16.843 INFO:teuthology.orchestra.run.smithi072.stdout:Change: 2024-04-26 23:28:36.512585790 +0000 2024-04-26T23:36:16.843 INFO:teuthology.orchestra.run.smithi072.stdout: Birth: 2024-04-26 23:28:36.512585790 +0000 2024-04-26T23:36:16.843 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T23:36:16.922 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records in 2024-04-26T23:36:16.923 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records out 2024-04-26T23:36:16.923 INFO:teuthology.orchestra.run.smithi072.stderr:512 bytes copied, 0.000371324 s, 1.4 MB/s 2024-04-26T23:36:16.926 DEBUG:teuthology.orchestra.run.smithi072:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T23:36:16.982 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-04-26T23:36:16.983 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T23:36:17.001 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T23:36:17.001 DEBUG:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_1 2024-04-26T23:36:17.058 INFO:teuthology.orchestra.run.smithi076.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T23:36:17.058 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T23:36:17.058 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-26T23:36:17.058 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T23:36:17.058 INFO:teuthology.orchestra.run.smithi076.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T23:36:17.058 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2024-04-26 23:36:02.849320129 +0000 2024-04-26T23:36:17.058 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2024-04-26 23:27:27.552530719 +0000 2024-04-26T23:36:17.059 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2024-04-26 23:27:27.552530719 +0000 2024-04-26T23:36:17.059 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: 2024-04-26 23:27:27.552530719 +0000 2024-04-26T23:36:17.059 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T23:36:17.124 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2024-04-26T23:36:17.125 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2024-04-26T23:36:17.125 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes copied, 0.000228515 s, 2.2 MB/s 2024-04-26T23:36:17.126 DEBUG:teuthology.orchestra.run.smithi076:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T23:36:17.183 DEBUG:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_2 2024-04-26T23:36:17.240 INFO:teuthology.orchestra.run.smithi076.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T23:36:17.240 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T23:36:17.240 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-26T23:36:17.240 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T23:36:17.240 INFO:teuthology.orchestra.run.smithi076.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T23:36:17.240 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2024-04-26 23:36:02.850320099 +0000 2024-04-26T23:36:17.240 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2024-04-26 23:27:27.910519861 +0000 2024-04-26T23:36:17.240 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2024-04-26 23:27:27.910519861 +0000 2024-04-26T23:36:17.240 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: 2024-04-26 23:27:27.910519861 +0000 2024-04-26T23:36:17.241 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T23:36:17.304 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2024-04-26T23:36:17.304 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2024-04-26T23:36:17.304 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes copied, 0.000243254 s, 2.1 MB/s 2024-04-26T23:36:17.306 DEBUG:teuthology.orchestra.run.smithi076:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T23:36:17.363 DEBUG:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_3 2024-04-26T23:36:17.419 INFO:teuthology.orchestra.run.smithi076.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T23:36:17.419 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T23:36:17.419 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-26T23:36:17.419 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T23:36:17.419 INFO:teuthology.orchestra.run.smithi076.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T23:36:17.419 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2024-04-26 23:36:02.850320099 +0000 2024-04-26T23:36:17.419 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2024-04-26 23:27:28.249509579 +0000 2024-04-26T23:36:17.419 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2024-04-26 23:27:28.249509579 +0000 2024-04-26T23:36:17.419 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: 2024-04-26 23:27:28.249509579 +0000 2024-04-26T23:36:17.420 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T23:36:17.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:17 smithi076 ceph-mon[32774]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi072=x;smithi076=y;smithi136=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:36:17.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:17 smithi076 ceph-mon[32774]: Saving service mgr spec with placement smithi072=x;smithi076=y;smithi136=z;count:3 2024-04-26T23:36:17.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:17 smithi076 ceph-mon[32774]: Reconfiguring mon.c (monmap changed)... 2024-04-26T23:36:17.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:17 smithi076 ceph-mon[32774]: Reconfiguring daemon mon.c on smithi136 2024-04-26T23:36:17.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:17 smithi076 ceph-mon[32774]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:17.451 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2024-04-26T23:36:17.451 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2024-04-26T23:36:17.451 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes copied, 0.000214734 s, 2.4 MB/s 2024-04-26T23:36:17.453 DEBUG:teuthology.orchestra.run.smithi076:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T23:36:17.508 DEBUG:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_4 2024-04-26T23:36:17.563 INFO:teuthology.orchestra.run.smithi076.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T23:36:17.564 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T23:36:17.564 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-04-26T23:36:17.564 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T23:36:17.564 INFO:teuthology.orchestra.run.smithi076.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T23:36:17.564 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2024-04-26 23:36:02.850320099 +0000 2024-04-26T23:36:17.564 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2024-04-26 23:27:28.590499237 +0000 2024-04-26T23:36:17.564 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2024-04-26 23:27:28.590499237 +0000 2024-04-26T23:36:17.564 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: 2024-04-26 23:27:28.590499237 +0000 2024-04-26T23:36:17.564 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T23:36:17.626 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2024-04-26T23:36:17.626 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2024-04-26T23:36:17.626 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes copied, 0.000202182 s, 2.5 MB/s 2024-04-26T23:36:17.627 DEBUG:teuthology.orchestra.run.smithi076:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T23:36:17.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:17 smithi136 ceph-mon[33050]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi072=x;smithi076=y;smithi136=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:36:17.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:17 smithi136 ceph-mon[33050]: Saving service mgr spec with placement smithi072=x;smithi076=y;smithi136=z;count:3 2024-04-26T23:36:17.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:17 smithi136 ceph-mon[33050]: Reconfiguring mon.c (monmap changed)... 2024-04-26T23:36:17.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:17 smithi136 ceph-mon[33050]: Reconfiguring daemon mon.c on smithi136 2024-04-26T23:36:17.661 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:17 smithi136 ceph-mon[33050]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:17.683 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-04-26T23:36:17.683 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T23:36:17.701 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T23:36:17.701 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_1 2024-04-26T23:36:17.758 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T23:36:17.758 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T23:36:17.758 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-26T23:36:17.758 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T23:36:17.758 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T23:36:17.758 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-04-26 23:35:52.438356090 +0000 2024-04-26T23:36:17.759 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-04-26 23:27:27.555122075 +0000 2024-04-26T23:36:17.759 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-04-26 23:27:27.555122075 +0000 2024-04-26T23:36:17.759 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-04-26 23:27:27.555122075 +0000 2024-04-26T23:36:17.759 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T23:36:17.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:17 smithi072 ceph-mon[28184]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi072=x;smithi076=y;smithi136=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:36:17.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:17 smithi072 ceph-mon[28184]: Saving service mgr spec with placement smithi072=x;smithi076=y;smithi136=z;count:3 2024-04-26T23:36:17.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:17 smithi072 ceph-mon[28184]: Reconfiguring mon.c (monmap changed)... 2024-04-26T23:36:17.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:17 smithi072 ceph-mon[28184]: Reconfiguring daemon mon.c on smithi136 2024-04-26T23:36:17.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:17 smithi072 ceph-mon[28184]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:17.822 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-04-26T23:36:17.822 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-04-26T23:36:17.822 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000207862 s, 2.5 MB/s 2024-04-26T23:36:17.823 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T23:36:17.880 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_2 2024-04-26T23:36:17.936 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T23:36:17.936 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T23:36:17.936 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-04-26T23:36:17.936 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T23:36:17.936 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T23:36:17.936 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-04-26 23:35:52.438356090 +0000 2024-04-26T23:36:17.936 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-04-26 23:27:27.892113982 +0000 2024-04-26T23:36:17.936 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-04-26 23:27:27.892113982 +0000 2024-04-26T23:36:17.936 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-04-26 23:27:27.892113982 +0000 2024-04-26T23:36:17.936 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T23:36:17.998 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-04-26T23:36:17.999 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-04-26T23:36:17.999 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000200648 s, 2.6 MB/s 2024-04-26T23:36:18.000 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T23:36:18.055 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_3 2024-04-26T23:36:18.110 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T23:36:18.111 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T23:36:18.111 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-26T23:36:18.111 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T23:36:18.111 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T23:36:18.111 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-04-26 23:35:52.438356090 +0000 2024-04-26T23:36:18.111 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-04-26 23:27:28.247105458 +0000 2024-04-26T23:36:18.111 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-04-26 23:27:28.247105458 +0000 2024-04-26T23:36:18.111 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-04-26 23:27:28.247105458 +0000 2024-04-26T23:36:18.111 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T23:36:18.173 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-04-26T23:36:18.173 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-04-26T23:36:18.173 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000208239 s, 2.5 MB/s 2024-04-26T23:36:18.174 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T23:36:18.230 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_4 2024-04-26T23:36:18.287 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T23:36:18.287 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T23:36:18.287 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-04-26T23:36:18.287 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T23:36:18.287 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T23:36:18.287 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-04-26 23:35:52.439356067 +0000 2024-04-26T23:36:18.287 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-04-26 23:27:28.587097294 +0000 2024-04-26T23:36:18.287 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-04-26 23:27:28.587097294 +0000 2024-04-26T23:36:18.287 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-04-26 23:27:28.587097294 +0000 2024-04-26T23:36:18.288 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T23:36:18.350 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-04-26T23:36:18.350 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-04-26T23:36:18.350 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000208738 s, 2.5 MB/s 2024-04-26T23:36:18.352 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T23:36:18.408 INFO:tasks.cephadm:Deploying osd.0 on smithi072 with /dev/vg_nvme/lv_4... 2024-04-26T23:36:18.408 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-26T23:36:18.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:18 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:18.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:18 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:18.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:18 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:36:18.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:18 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:36:18.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:18 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:36:18.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:18 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:18.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:18 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T23:36:18.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:18 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T23:36:18.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:18 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T23:36:18.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:18 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:36:18.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:18 smithi072 ceph-mon[28184]: Deploying daemon mgr.z on smithi136 2024-04-26T23:36:18.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:18 smithi072 ceph-mon[28184]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:18.858 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:36:18.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:18 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:18.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:18 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:18.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:18 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:36:18.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:18 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:36:18.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:18 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:36:18.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:18 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:18.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:18 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T23:36:18.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:18 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T23:36:18.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:18 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T23:36:18.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:18 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:36:18.930 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:18 smithi076 ceph-mon[32774]: Deploying daemon mgr.z on smithi136 2024-04-26T23:36:18.930 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:18 smithi076 ceph-mon[32774]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:19.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:18 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:19.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:18 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:19.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:18 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:36:19.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:18 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:36:19.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:18 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:36:19.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:18 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:19.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:18 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T23:36:19.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:18 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T23:36:19.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:18 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T23:36:19.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:18 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:36:19.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:18 smithi136 ceph-mon[33050]: Deploying daemon mgr.z on smithi136 2024-04-26T23:36:19.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:18 smithi136 ceph-mon[33050]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:20.392 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 26 23:36:20 smithi136 systemd[1]: Started Ceph mgr.z for 3c8453ae-0425-11ef-bc93-c7b262605968. 2024-04-26T23:36:20.676 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 26 23:36:20 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-z[34924]: 2024-04-26T23:36:20.388+0000 7ff1f5601200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T23:36:20.676 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 26 23:36:20 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-z[34924]: 2024-04-26T23:36:20.572+0000 7ff1f5601200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T23:36:20.676 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 26 23:36:20 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-z[34924]: 2024-04-26T23:36:20.673+0000 7ff1f5601200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T23:36:21.299 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 26 23:36:20 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-z[34924]: 2024-04-26T23:36:20.979+0000 7ff1f5601200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T23:36:21.299 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 26 23:36:21 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-z[34924]: 2024-04-26T23:36:21.062+0000 7ff1f5601200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T23:36:21.611 INFO:teuthology.orchestra.run.smithi072.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-26T23:36:21.611 INFO:teuthology.orchestra.run.smithi072.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-26T23:36:21.611 INFO:teuthology.orchestra.run.smithi072.stderr: stderr: 10+0 records in 2024-04-26T23:36:21.611 INFO:teuthology.orchestra.run.smithi072.stderr:10+0 records out 2024-04-26T23:36:21.611 INFO:teuthology.orchestra.run.smithi072.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0346272 s, 303 MB/s 2024-04-26T23:36:21.611 INFO:teuthology.orchestra.run.smithi072.stderr:--> Zapping successful for: 2024-04-26T23:36:21.648 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:21 smithi076 ceph-mon[32774]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:21.648 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:21 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:21.648 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:21 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:21.648 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:21 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:21.648 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:21 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T23:36:21.648 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:21 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T23:36:21.648 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:21 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T23:36:21.648 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:21 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:36:21.648 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:21 smithi076 ceph-mon[32774]: Deploying daemon mgr.y on smithi076 2024-04-26T23:36:21.657 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:21 smithi072 ceph-mon[28184]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:21.657 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:21 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:21.657 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:21 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:21.657 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:21 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:21.657 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:21 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T23:36:21.657 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:21 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T23:36:21.657 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:21 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T23:36:21.657 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:21 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:36:21.658 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:21 smithi072 ceph-mon[28184]: Deploying daemon mgr.y on smithi076 2024-04-26T23:36:21.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:21 smithi136 ceph-mon[33050]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:21.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:21 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:21.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:21 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:21.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:21 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:21.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:21 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T23:36:21.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:21 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T23:36:21.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:21 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T23:36:21.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:21 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:36:21.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:21 smithi136 ceph-mon[33050]: Deploying daemon mgr.y on smithi076 2024-04-26T23:36:22.428 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Apr 26 23:36:22 smithi076 systemd[1]: Starting Ceph mgr.y for 3c8453ae-0425-11ef-bc93-c7b262605968... 2024-04-26T23:36:22.430 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi072:vg_nvme/lv_4 2024-04-26T23:36:22.549 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 26 23:36:22 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-z[34924]: 2024-04-26T23:36:22.249+0000 7ff1f5601200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T23:36:22.549 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 26 23:36:22 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-z[34924]: 2024-04-26T23:36:22.418+0000 7ff1f5601200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T23:36:22.830 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Apr 26 23:36:22 smithi076 podman[34152]: 2024-04-26 23:36:22.522960284 +0000 UTC m=+0.112684434 container create d9487d6301b771feed4ac2949c89c78b484986dc91858d334e02cde4cfcf3bec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-26T23:36:22.830 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Apr 26 23:36:22 smithi076 podman[34152]: 2024-04-26 23:36:22.427864357 +0000 UTC m=+0.017588520 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T23:36:22.830 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Apr 26 23:36:22 smithi076 podman[34152]: 2024-04-26 23:36:22.719489696 +0000 UTC m=+0.309213847 container init d9487d6301b771feed4ac2949c89c78b484986dc91858d334e02cde4cfcf3bec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T23:36:22.830 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Apr 26 23:36:22 smithi076 podman[34152]: 2024-04-26 23:36:22.722261202 +0000 UTC m=+0.311985363 container start d9487d6301b771feed4ac2949c89c78b484986dc91858d334e02cde4cfcf3bec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-y, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-26T23:36:22.830 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Apr 26 23:36:22 smithi076 ceph-mgr[34171]: -- 172.21.15.76:0/3519389098 <== mon.2 v2:172.21.15.76:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555d51d705a0 con 0x555d51d44800 2024-04-26T23:36:22.830 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Apr 26 23:36:22 smithi076 bash[34152]: d9487d6301b771feed4ac2949c89c78b484986dc91858d334e02cde4cfcf3bec 2024-04-26T23:36:22.849 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:36:22.876 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 26 23:36:22 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-z[34924]: 2024-04-26T23:36:22.589+0000 7ff1f5601200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T23:36:22.877 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 26 23:36:22 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-z[34924]: 2024-04-26T23:36:22.873+0000 7ff1f5601200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T23:36:23.164 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Apr 26 23:36:22 smithi076 systemd[1]: Started Ceph mgr.y for 3c8453ae-0425-11ef-bc93-c7b262605968. 2024-04-26T23:36:23.164 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Apr 26 23:36:22 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-y[34167]: 2024-04-26T23:36:22.870+0000 7fdc43a20200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T23:36:23.164 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Apr 26 23:36:23 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-y[34167]: 2024-04-26T23:36:23.055+0000 7fdc43a20200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T23:36:23.164 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Apr 26 23:36:23 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-y[34167]: 2024-04-26T23:36:23.160+0000 7fdc43a20200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T23:36:23.177 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 26 23:36:22 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-z[34924]: 2024-04-26T23:36:22.968+0000 7ff1f5601200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T23:36:23.177 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 26 23:36:23 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-z[34924]: 2024-04-26T23:36:23.174+0000 7ff1f5601200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T23:36:23.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:23 smithi076 ceph-mon[32774]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:23.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:23 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:23.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:23 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:23.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:23 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:23.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:23 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:23.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:23 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:36:23.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:23 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:23.549 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 26 23:36:23 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-z[34924]: 2024-04-26T23:36:23.373+0000 7ff1f5601200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T23:36:23.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:23 smithi136 ceph-mon[33050]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:23.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:23 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:23.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:23 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:23.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:23 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:23.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:23 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:23.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:23 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:36:23.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:23 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:23.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:23 smithi072 ceph-mon[28184]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:23.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:23 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:23.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:23 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:23.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:23 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:23.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:23 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:23.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:23 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:36:23.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:23 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:23.928 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Apr 26 23:36:23 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-y[34167]: 2024-04-26T23:36:23.486+0000 7fdc43a20200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T23:36:23.928 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Apr 26 23:36:23 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-y[34167]: 2024-04-26T23:36:23.573+0000 7fdc43a20200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T23:36:24.299 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 26 23:36:24 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-z[34924]: 2024-04-26T23:36:24.017+0000 7ff1f5601200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T23:36:24.757 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 26 23:36:24 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-z[34924]: 2024-04-26T23:36:24.455+0000 7ff1f5601200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T23:36:24.757 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 26 23:36:24 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-z[34924]: 2024-04-26T23:36:24.552+0000 7ff1f5601200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T23:36:24.757 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 26 23:36:24 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-z[34924]: 2024-04-26T23:36:24.753+0000 7ff1f5601200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T23:36:24.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:24.916+0000 7f6c8be00700 1 -- 172.21.15.72:0/1538502763 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c84103f10 msgr2=0x7f6c84106300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:36:24.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:24.916+0000 7f6c8be00700 1 --2- 172.21.15.72:0/1538502763 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c84103f10 0x7f6c84106300 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6c780098f0 tx=0x7f6c78009c00 comp rx=0 tx=0).stop 2024-04-26T23:36:24.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:24.916+0000 7f6c8be00700 1 -- 172.21.15.72:0/1538502763 shutdown_connections 2024-04-26T23:36:24.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:24.916+0000 7f6c8be00700 1 --2- 172.21.15.72:0/1538502763 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6c84109170 0x7f6c8410b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:36:24.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:24.916+0000 7f6c8be00700 1 --2- 172.21.15.72:0/1538502763 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c84103f10 0x7f6c84106300 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:36:24.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:24.916+0000 7f6c8be00700 1 --2- 172.21.15.72:0/1538502763 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6c84106840 0x7f6c84108c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:36:24.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:24.916+0000 7f6c8be00700 1 -- 172.21.15.72:0/1538502763 >> 172.21.15.72:0/1538502763 conn(0x7f6c840fd920 msgr2=0x7f6c840ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:36:24.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:24.916+0000 7f6c8be00700 1 -- 172.21.15.72:0/1538502763 shutdown_connections 2024-04-26T23:36:24.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:24.916+0000 7f6c8be00700 1 -- 172.21.15.72:0/1538502763 wait complete. 2024-04-26T23:36:24.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:24.917+0000 7f6c8be00700 1 Processor -- start 2024-04-26T23:36:24.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:24.917+0000 7f6c8be00700 1 -- start start 2024-04-26T23:36:24.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:24.917+0000 7f6c8be00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6c84103f10 0x7f6c8419b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:36:24.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:24.917+0000 7f6c8be00700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c84106840 0x7f6c8419be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:36:24.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:24.917+0000 7f6c8be00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6c84109170 0x7f6c841a2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:36:24.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:24.917+0000 7f6c8be00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f6c8410ef90 con 0x7f6c84109170 2024-04-26T23:36:24.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:24.917+0000 7f6c8be00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f6c8410ee10 con 0x7f6c84103f10 2024-04-26T23:36:24.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:24.917+0000 7f6c8be00700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f6c8410f110 con 0x7f6c84106840 2024-04-26T23:36:24.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:24.918+0000 7f6c89400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6c84103f10 0x7f6c8419b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:36:24.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:24.918+0000 7f6c89400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6c84103f10 0x7f6c8419b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:49560/0 (socket says 172.21.15.72:49560) 2024-04-26T23:36:24.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:24.918+0000 7f6c88a00700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c84106840 0x7f6c8419be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:36:24.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:24.918+0000 7f6c89400700 1 -- 172.21.15.72:0/288011809 learned_addr learned my addr 172.21.15.72:0/288011809 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:36:24.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:24.918+0000 7f6c89e00700 1 --2- 172.21.15.72:0/288011809 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6c84109170 0x7f6c841a2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:36:24.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:24.918+0000 7f6c89400700 1 -- 172.21.15.72:0/288011809 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c84106840 msgr2=0x7f6c8419be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:36:24.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:24.918+0000 7f6c89400700 1 --2- 172.21.15.72:0/288011809 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c84106840 0x7f6c8419be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:36:24.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:24.918+0000 7f6c89400700 1 -- 172.21.15.72:0/288011809 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6c84109170 msgr2=0x7f6c841a2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:36:24.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:24.918+0000 7f6c89400700 1 --2- 172.21.15.72:0/288011809 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6c84109170 0x7f6c841a2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:36:24.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:24.918+0000 7f6c89400700 1 -- 172.21.15.72:0/288011809 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c841a3520 con 0x7f6c84103f10 2024-04-26T23:36:24.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:24.918+0000 7f6c89e00700 1 --2- 172.21.15.72:0/288011809 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6c84109170 0x7f6c841a2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:36:24.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:24.918+0000 7f6c89400700 1 --2- 172.21.15.72:0/288011809 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6c84103f10 0x7f6c8419b8c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6c78011040 tx=0x7f6c78005270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:36:24.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:24.919+0000 7f6c82000700 1 -- 172.21.15.72:0/288011809 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c78027070 con 0x7f6c84103f10 2024-04-26T23:36:24.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:24.919+0000 7f6c82000700 1 -- 172.21.15.72:0/288011809 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c7800f930 con 0x7f6c84103f10 2024-04-26T23:36:24.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:24.919+0000 7f6c8be00700 1 -- 172.21.15.72:0/288011809 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c841a37b0 con 0x7f6c84103f10 2024-04-26T23:36:24.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:24.919+0000 7f6c82000700 1 -- 172.21.15.72:0/288011809 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c780049c0 con 0x7f6c84103f10 2024-04-26T23:36:24.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:24.919+0000 7f6c8be00700 1 -- 172.21.15.72:0/288011809 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c841a3c00 con 0x7f6c84103f10 2024-04-26T23:36:24.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:24.920+0000 7f6c82000700 1 -- 172.21.15.72:0/288011809 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f6c7802a070 con 0x7f6c84103f10 2024-04-26T23:36:24.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:24.920+0000 7f6c8be00700 1 -- 172.21.15.72:0/288011809 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c8406a140 con 0x7f6c84103f10 2024-04-26T23:36:24.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:24.920+0000 7f6c82000700 1 --2- 172.21.15.72:0/288011809 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6c6c039890 0x7f6c6c03bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:36:24.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:24.921+0000 7f6c82000700 1 -- 172.21.15.72:0/288011809 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f6c7804d620 con 0x7f6c84103f10 2024-04-26T23:36:24.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:24.921+0000 7f6c88a00700 1 --2- 172.21.15.72:0/288011809 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6c6c039890 0x7f6c6c03bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:36:24.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:24.921+0000 7f6c88a00700 1 --2- 172.21.15.72:0/288011809 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6c6c039890 0x7f6c6c03bd50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6c70009750 tx=0x7f6c70009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:36:24.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:24.930+0000 7f6c82000700 1 -- 172.21.15.72:0/288011809 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6c8406a140 con 0x7f6c84103f10 2024-04-26T23:36:25.171 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Apr 26 23:36:24 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-y[34167]: 2024-04-26T23:36:24.832+0000 7fdc43a20200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T23:36:25.171 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Apr 26 23:36:25 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-y[34167]: 2024-04-26T23:36:25.002+0000 7fdc43a20200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T23:36:25.172 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Apr 26 23:36:25 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-y[34167]: 2024-04-26T23:36:25.168+0000 7fdc43a20200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T23:36:25.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:25.171+0000 7f6c8be00700 1 -- 172.21.15.72:0/288011809 --> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi072:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f6c84102690 con 0x7f6c6c039890 2024-04-26T23:36:25.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:25 smithi076 ceph-mon[32774]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:25.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:25 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:25.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:25 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:25.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:25 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:25.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:25 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:25.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:25 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:36:25.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:25 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:36:25.430 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:25 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:25.430 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:25 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T23:36:25.430 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:25 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T23:36:25.430 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:25 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:36:25.430 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:25 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:36:25.430 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:25 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:36:25.430 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:25 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:36:25.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:25 smithi072 ceph-mon[28184]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:25.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:25 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:25.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:25 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:25.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:25 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:25.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:25 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:25.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:25 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:36:25.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:25 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:36:25.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:25 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:25.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:25 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T23:36:25.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:25 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T23:36:25.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:25 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:36:25.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:25 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:36:25.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:25 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:36:25.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:25 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:36:25.549 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 26 23:36:25 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-z[34924]: 2024-04-26T23:36:25.291+0000 7ff1f5601200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T23:36:25.549 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 26 23:36:25 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-z[34924]: 2024-04-26T23:36:25.381+0000 7ff1f5601200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T23:36:25.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:25 smithi136 ceph-mon[33050]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:25.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:25 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:25.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:25 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:25.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:25 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:25.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:25 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:25.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:25 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:36:25.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:25 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:36:25.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:25 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:25.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:25 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T23:36:25.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:25 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T23:36:25.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:25 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:36:25.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:25 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:36:25.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:25 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:36:25.551 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:25 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:36:25.742 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Apr 26 23:36:25 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-y[34167]: 2024-04-26T23:36:25.451+0000 7fdc43a20200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T23:36:25.743 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Apr 26 23:36:25 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-y[34167]: 2024-04-26T23:36:25.541+0000 7fdc43a20200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T23:36:25.743 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Apr 26 23:36:25 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-y[34167]: 2024-04-26T23:36:25.739+0000 7fdc43a20200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T23:36:25.965 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 26 23:36:25 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-z[34924]: 2024-04-26T23:36:25.623+0000 7ff1f5601200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T23:36:25.965 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 26 23:36:25 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-z[34924]: 2024-04-26T23:36:25.755+0000 7ff1f5601200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T23:36:25.965 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 26 23:36:25 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-z[34924]: 2024-04-26T23:36:25.848+0000 7ff1f5601200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T23:36:25.965 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 26 23:36:25 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-z[34924]: 2024-04-26T23:36:25.961+0000 7ff1f5601200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T23:36:26.178 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Apr 26 23:36:25 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-y[34167]: 2024-04-26T23:36:25.926+0000 7fdc43a20200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T23:36:26.549 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 26 23:36:26 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-z[34924]: 2024-04-26T23:36:26.279+0000 7ff1f5601200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T23:36:26.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:26 smithi136 ceph-mon[33050]: Reconfiguring mgr.x (unknown last config time)... 2024-04-26T23:36:26.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:26 smithi136 ceph-mon[33050]: Reconfiguring daemon mgr.x on smithi072 2024-04-26T23:36:26.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:26 smithi136 ceph-mon[33050]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi072:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:36:26.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:26 smithi076 ceph-mon[32774]: Reconfiguring mgr.x (unknown last config time)... 2024-04-26T23:36:26.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:26 smithi076 ceph-mon[32774]: Reconfiguring daemon mgr.x on smithi072 2024-04-26T23:36:26.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:26 smithi076 ceph-mon[32774]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi072:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:36:26.679 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Apr 26 23:36:26 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-y[34167]: 2024-04-26T23:36:26.514+0000 7fdc43a20200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T23:36:26.788 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:26 smithi072 ceph-mon[28184]: Reconfiguring mgr.x (unknown last config time)... 2024-04-26T23:36:26.788 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:26 smithi072 ceph-mon[28184]: Reconfiguring daemon mgr.x on smithi072 2024-04-26T23:36:26.788 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:26 smithi072 ceph-mon[28184]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi072:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:36:27.178 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Apr 26 23:36:26 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-y[34167]: 2024-04-26T23:36:26.924+0000 7fdc43a20200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T23:36:27.178 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Apr 26 23:36:27 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-y[34167]: 2024-04-26T23:36:27.019+0000 7fdc43a20200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T23:36:27.299 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 26 23:36:26 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-z[34924]: 2024-04-26T23:36:26.871+0000 7ff1f5601200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T23:36:27.299 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 26 23:36:27 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-z[34924]: 2024-04-26T23:36:27.085+0000 7ff1f5601200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T23:36:27.678 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Apr 26 23:36:27 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-y[34167]: 2024-04-26T23:36:27.218+0000 7fdc43a20200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T23:36:27.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:27 smithi076 ceph-mon[32774]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:27.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:27 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:27.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:27 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:27.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:27 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:36:27.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:27 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:36:27.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:27 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:36:27.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:27 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:27.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:27 smithi076 ceph-mon[32774]: Standby manager daemon z started 2024-04-26T23:36:27.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:27 smithi076 ceph-mon[32774]: from='mgr.? 172.21.15.136:0/1872101725' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T23:36:27.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:27 smithi076 ceph-mon[32774]: from='mgr.? 172.21.15.136:0/1872101725' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T23:36:27.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:27 smithi076 ceph-mon[32774]: from='mgr.? 172.21.15.136:0/1872101725' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T23:36:27.680 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:27 smithi076 ceph-mon[32774]: from='mgr.? 172.21.15.136:0/1872101725' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T23:36:27.789 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:27 smithi072 ceph-mon[28184]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:27.790 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:27 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:27.790 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:27 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:27.790 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:27 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:36:27.790 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:27 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:36:27.790 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:27 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:36:27.790 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:27 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:27.790 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:27 smithi072 ceph-mon[28184]: Standby manager daemon z started 2024-04-26T23:36:27.790 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:27 smithi072 ceph-mon[28184]: from='mgr.? 172.21.15.136:0/1872101725' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T23:36:27.790 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:27 smithi072 ceph-mon[28184]: from='mgr.? 172.21.15.136:0/1872101725' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T23:36:27.790 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:27 smithi072 ceph-mon[28184]: from='mgr.? 172.21.15.136:0/1872101725' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T23:36:27.790 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:27 smithi072 ceph-mon[28184]: from='mgr.? 172.21.15.136:0/1872101725' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T23:36:27.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:27 smithi136 ceph-mon[33050]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:27.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:27 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:27.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:27 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:27.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:27 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:36:27.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:27 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:36:27.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:27 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:36:27.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:27 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:27.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:27 smithi136 ceph-mon[33050]: Standby manager daemon z started 2024-04-26T23:36:27.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:27 smithi136 ceph-mon[33050]: from='mgr.? 172.21.15.136:0/1872101725' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T23:36:27.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:27 smithi136 ceph-mon[33050]: from='mgr.? 172.21.15.136:0/1872101725' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T23:36:27.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:27 smithi136 ceph-mon[33050]: from='mgr.? 172.21.15.136:0/1872101725' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T23:36:27.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:27 smithi136 ceph-mon[33050]: from='mgr.? 172.21.15.136:0/1872101725' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T23:36:27.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:27.976+0000 7f6c82000700 1 -- 172.21.15.72:0/288011809 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mgrmap(e 14) ==== 92602+0+0 (secure 0 0 0) 0x7f6c7802c960 con 0x7f6c84103f10 2024-04-26T23:36:28.058 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Apr 26 23:36:27 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-y[34167]: 2024-04-26T23:36:27.730+0000 7fdc43a20200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T23:36:28.058 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Apr 26 23:36:27 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-y[34167]: 2024-04-26T23:36:27.819+0000 7fdc43a20200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T23:36:28.392 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Apr 26 23:36:28 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-y[34167]: 2024-04-26T23:36:28.055+0000 7fdc43a20200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T23:36:28.392 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Apr 26 23:36:28 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-y[34167]: 2024-04-26T23:36:28.185+0000 7fdc43a20200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T23:36:28.392 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Apr 26 23:36:28 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-y[34167]: 2024-04-26T23:36:28.277+0000 7fdc43a20200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T23:36:28.392 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Apr 26 23:36:28 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-y[34167]: 2024-04-26T23:36:28.389+0000 7fdc43a20200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T23:36:28.984 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Apr 26 23:36:28 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-y[34167]: 2024-04-26T23:36:28.697+0000 7fdc43a20200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T23:36:29.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:28 smithi072 ceph-mon[28184]: mgrmap e14: x(active, since 89s), standbys: z 2024-04-26T23:36:29.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:28 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T23:36:29.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:28 smithi072 ceph-mon[28184]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:29.295 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:28 smithi076 ceph-mon[32774]: mgrmap e14: x(active, since 89s), standbys: z 2024-04-26T23:36:29.295 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:28 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T23:36:29.295 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:28 smithi076 ceph-mon[32774]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:29.299 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:28 smithi136 ceph-mon[33050]: mgrmap e14: x(active, since 89s), standbys: z 2024-04-26T23:36:29.299 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:28 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T23:36:29.299 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:28 smithi136 ceph-mon[33050]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:29.678 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Apr 26 23:36:29 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-y[34167]: 2024-04-26T23:36:29.292+0000 7fdc43a20200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T23:36:29.678 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Apr 26 23:36:29 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-y[34167]: 2024-04-26T23:36:29.503+0000 7fdc43a20200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T23:36:30.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:29.997+0000 7f6c82000700 1 -- 172.21.15.72:0/288011809 <== mon.2 v2:172.21.15.76:3300/0 8 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f6c7802a360 con 0x7f6c84103f10 2024-04-26T23:36:30.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:30 smithi072 ceph-mon[28184]: from='mgr.? 172.21.15.76:0/1649104551' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T23:36:30.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:30 smithi072 ceph-mon[28184]: Standby manager daemon y started 2024-04-26T23:36:30.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:30 smithi072 ceph-mon[28184]: from='mgr.? 172.21.15.76:0/1649104551' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T23:36:30.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:30 smithi072 ceph-mon[28184]: from='mgr.? 172.21.15.76:0/1649104551' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T23:36:30.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:30 smithi072 ceph-mon[28184]: from='mgr.? 172.21.15.76:0/1649104551' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T23:36:30.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:30 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/224600047' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e20c9ebf-56f6-47c1-a11b-e1e615af042b"}]: dispatch 2024-04-26T23:36:30.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:30 smithi072 ceph-mon[28184]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e20c9ebf-56f6-47c1-a11b-e1e615af042b"}]: dispatch 2024-04-26T23:36:30.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:30 smithi072 ceph-mon[28184]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e20c9ebf-56f6-47c1-a11b-e1e615af042b"}]': finished 2024-04-26T23:36:30.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:30 smithi072 ceph-mon[28184]: osdmap e5: 1 total, 0 up, 1 in 2024-04-26T23:36:30.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:30 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T23:36:30.299 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:30 smithi136 ceph-mon[33050]: from='mgr.? 172.21.15.76:0/1649104551' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T23:36:30.299 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:30 smithi136 ceph-mon[33050]: Standby manager daemon y started 2024-04-26T23:36:30.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:30 smithi136 ceph-mon[33050]: from='mgr.? 172.21.15.76:0/1649104551' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T23:36:30.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:30 smithi136 ceph-mon[33050]: from='mgr.? 172.21.15.76:0/1649104551' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T23:36:30.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:30 smithi136 ceph-mon[33050]: from='mgr.? 172.21.15.76:0/1649104551' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T23:36:30.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:30 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/224600047' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e20c9ebf-56f6-47c1-a11b-e1e615af042b"}]: dispatch 2024-04-26T23:36:30.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:30 smithi136 ceph-mon[33050]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e20c9ebf-56f6-47c1-a11b-e1e615af042b"}]: dispatch 2024-04-26T23:36:30.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:30 smithi136 ceph-mon[33050]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e20c9ebf-56f6-47c1-a11b-e1e615af042b"}]': finished 2024-04-26T23:36:30.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:30 smithi136 ceph-mon[33050]: osdmap e5: 1 total, 0 up, 1 in 2024-04-26T23:36:30.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:30 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T23:36:30.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:30 smithi076 ceph-mon[32774]: from='mgr.? 172.21.15.76:0/1649104551' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T23:36:30.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:30 smithi076 ceph-mon[32774]: Standby manager daemon y started 2024-04-26T23:36:30.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:30 smithi076 ceph-mon[32774]: from='mgr.? 172.21.15.76:0/1649104551' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T23:36:30.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:30 smithi076 ceph-mon[32774]: from='mgr.? 172.21.15.76:0/1649104551' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T23:36:30.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:30 smithi076 ceph-mon[32774]: from='mgr.? 172.21.15.76:0/1649104551' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T23:36:30.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:30 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/224600047' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e20c9ebf-56f6-47c1-a11b-e1e615af042b"}]: dispatch 2024-04-26T23:36:30.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:30 smithi076 ceph-mon[32774]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e20c9ebf-56f6-47c1-a11b-e1e615af042b"}]: dispatch 2024-04-26T23:36:30.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:30 smithi076 ceph-mon[32774]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e20c9ebf-56f6-47c1-a11b-e1e615af042b"}]': finished 2024-04-26T23:36:30.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:30 smithi076 ceph-mon[32774]: osdmap e5: 1 total, 0 up, 1 in 2024-04-26T23:36:30.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:30 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T23:36:31.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:31 smithi072 ceph-mon[28184]: mgrmap e15: x(active, since 91s), standbys: z, y 2024-04-26T23:36:31.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:31 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T23:36:31.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:31 smithi072 ceph-mon[28184]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:31.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:31 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/1982770446' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:36:31.299 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:31 smithi136 ceph-mon[33050]: mgrmap e15: x(active, since 91s), standbys: z, y 2024-04-26T23:36:31.299 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:31 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T23:36:31.299 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:31 smithi136 ceph-mon[33050]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:31.299 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:31 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/1982770446' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:36:31.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:31 smithi076 ceph-mon[32774]: mgrmap e15: x(active, since 91s), standbys: z, y 2024-04-26T23:36:31.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:31 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T23:36:31.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:31 smithi076 ceph-mon[32774]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:31.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:31 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/1982770446' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:36:33.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:33 smithi076 ceph-mon[32774]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:33.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:33 smithi072 ceph-mon[28184]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:33.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:33 smithi136 ceph-mon[33050]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:35.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:35 smithi076 ceph-mon[32774]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:35.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:35 smithi072 ceph-mon[28184]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:35.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:35 smithi136 ceph-mon[33050]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:37.440 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:37 smithi072 ceph-mon[28184]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:37.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:37 smithi076 ceph-mon[32774]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:37.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:37 smithi136 ceph-mon[33050]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:39.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:39 smithi076 ceph-mon[32774]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:39.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:39 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-26T23:36:39.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:39 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:36:39.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:39 smithi072 ceph-mon[28184]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:39.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:39 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-26T23:36:39.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:39 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:36:39.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:39 smithi136 ceph-mon[33050]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:39.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:39 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-26T23:36:39.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:39 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:36:40.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:40 smithi076 ceph-mon[32774]: Deploying daemon osd.0 on smithi072 2024-04-26T23:36:40.701 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:40 smithi136 ceph-mon[33050]: Deploying daemon osd.0 on smithi072 2024-04-26T23:36:40.765 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:40 smithi072 ceph-mon[28184]: Deploying daemon osd.0 on smithi072 2024-04-26T23:36:41.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:41 smithi076 ceph-mon[32774]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:41.704 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:41 smithi072 ceph-mon[28184]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:41.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:41 smithi136 ceph-mon[33050]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:43.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:43 smithi072 ceph-mon[28184]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:43.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:43 smithi076 ceph-mon[32774]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:43.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:43 smithi136 ceph-mon[33050]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:45.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:45 smithi076 ceph-mon[32774]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:45.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:45 smithi072 ceph-mon[28184]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:45.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:45 smithi136 ceph-mon[33050]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:47.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:47 smithi076 ceph-mon[32774]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:47.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:47 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:47.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:47 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:36:47.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:47 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:47.746 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:47 smithi136 ceph-mon[33050]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:47.746 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:47 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:47.746 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:47 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:36:47.746 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:47 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:47.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:47 smithi072 ceph-mon[28184]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:47.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:47 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:47.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:47 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:36:47.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:47 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:49.616 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:49 smithi072 ceph-mon[28184]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:49.685 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:49 smithi076 ceph-mon[32774]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:49.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:49 smithi136 ceph-mon[33050]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:51.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:51 smithi072 ceph-mon[28184]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:51.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:51 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:51.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:51 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:51.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:51 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:36:51.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:51 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:36:51.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:51 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:51.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:51 smithi072 ceph-mon[28184]: from='osd.0 [v2:172.21.15.72:6802/1639298875,v1:172.21.15.72:6803/1639298875]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T23:36:51.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:51 smithi136 ceph-mon[33050]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:51.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:51 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:51.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:51 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:51.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:51 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:36:51.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:51 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:36:51.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:51 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:51.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:51 smithi136 ceph-mon[33050]: from='osd.0 [v2:172.21.15.72:6802/1639298875,v1:172.21.15.72:6803/1639298875]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T23:36:51.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:51.922+0000 7f6c82000700 1 -- 172.21.15.72:0/288011809 <== mgr.14152 v2:172.21.15.72:6800/4281177709 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6c84102690 con 0x7f6c6c039890 2024-04-26T23:36:51.926 INFO:teuthology.orchestra.run.smithi072.stdout:Created osd(s) 0 on host 'smithi072' 2024-04-26T23:36:51.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:51 smithi076 ceph-mon[32774]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:51.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:51 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:51.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:51 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:51.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:51 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:36:51.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:51 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:36:51.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:51 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:51.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:51 smithi076 ceph-mon[32774]: from='osd.0 [v2:172.21.15.72:6802/1639298875,v1:172.21.15.72:6803/1639298875]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T23:36:51.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:51.925+0000 7f6c8be00700 1 -- 172.21.15.72:0/288011809 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6c6c039890 msgr2=0x7f6c6c03bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:36:51.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:51.925+0000 7f6c8be00700 1 --2- 172.21.15.72:0/288011809 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6c6c039890 0x7f6c6c03bd50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6c70009750 tx=0x7f6c70009310 comp rx=0 tx=0).stop 2024-04-26T23:36:51.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:51.925+0000 7f6c8be00700 1 -- 172.21.15.72:0/288011809 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6c84103f10 msgr2=0x7f6c8419b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:36:51.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:51.925+0000 7f6c8be00700 1 --2- 172.21.15.72:0/288011809 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6c84103f10 0x7f6c8419b8c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6c78011040 tx=0x7f6c78005270 comp rx=0 tx=0).stop 2024-04-26T23:36:51.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:51.926+0000 7f6c8be00700 1 -- 172.21.15.72:0/288011809 shutdown_connections 2024-04-26T23:36:51.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:51.926+0000 7f6c8be00700 1 --2- 172.21.15.72:0/288011809 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6c6c039890 0x7f6c6c03bd50 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:36:51.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:51.926+0000 7f6c8be00700 1 --2- 172.21.15.72:0/288011809 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6c84109170 0x7f6c841a2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:36:51.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:51.926+0000 7f6c8be00700 1 --2- 172.21.15.72:0/288011809 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6c84103f10 0x7f6c8419b8c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:36:51.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:51.926+0000 7f6c8be00700 1 --2- 172.21.15.72:0/288011809 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c84106840 0x7f6c8419be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:36:51.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:51.926+0000 7f6c8be00700 1 -- 172.21.15.72:0/288011809 >> 172.21.15.72:0/288011809 conn(0x7f6c840fd920 msgr2=0x7f6c84101f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:36:51.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:51.926+0000 7f6c8be00700 1 -- 172.21.15.72:0/288011809 shutdown_connections 2024-04-26T23:36:51.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:36:51.926+0000 7f6c8be00700 1 -- 172.21.15.72:0/288011809 wait complete. 2024-04-26T23:36:52.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:52 smithi072 ceph-mon[28184]: from='osd.0 [v2:172.21.15.72:6802/1639298875,v1:172.21.15.72:6803/1639298875]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-26T23:36:52.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:52 smithi072 ceph-mon[28184]: osdmap e6: 1 total, 0 up, 1 in 2024-04-26T23:36:52.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:52 smithi072 ceph-mon[28184]: from='osd.0 [v2:172.21.15.72:6802/1639298875,v1:172.21.15.72:6803/1639298875]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]: dispatch 2024-04-26T23:36:52.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:52 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T23:36:52.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:52 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:52.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:52 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:52.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:52 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:36:52.730 DEBUG:teuthology.orchestra.run.smithi072:osd.0> sudo journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@osd.0.service 2024-04-26T23:36:52.735 INFO:tasks.cephadm:Deploying osd.1 on smithi076 with /dev/vg_nvme/lv_4... 2024-04-26T23:36:52.735 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-26T23:36:52.761 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:52 smithi076 ceph-mon[32774]: from='osd.0 [v2:172.21.15.72:6802/1639298875,v1:172.21.15.72:6803/1639298875]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-26T23:36:52.761 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:52 smithi076 ceph-mon[32774]: osdmap e6: 1 total, 0 up, 1 in 2024-04-26T23:36:52.761 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:52 smithi076 ceph-mon[32774]: from='osd.0 [v2:172.21.15.72:6802/1639298875,v1:172.21.15.72:6803/1639298875]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]: dispatch 2024-04-26T23:36:52.761 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:52 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T23:36:52.761 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:52 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:52.762 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:52 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:52.762 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:52 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:36:52.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:52 smithi136 ceph-mon[33050]: from='osd.0 [v2:172.21.15.72:6802/1639298875,v1:172.21.15.72:6803/1639298875]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-26T23:36:52.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:52 smithi136 ceph-mon[33050]: osdmap e6: 1 total, 0 up, 1 in 2024-04-26T23:36:52.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:52 smithi136 ceph-mon[33050]: from='osd.0 [v2:172.21.15.72:6802/1639298875,v1:172.21.15.72:6803/1639298875]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]: dispatch 2024-04-26T23:36:52.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:52 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T23:36:52.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:52 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:52.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:52 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:52.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:52 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:36:52.975 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T23:36:53.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:53 smithi076 ceph-mon[32774]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:53.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:53 smithi076 ceph-mon[32774]: from='osd.0 [v2:172.21.15.72:6802/1639298875,v1:172.21.15.72:6803/1639298875]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]': finished 2024-04-26T23:36:53.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:53 smithi076 ceph-mon[32774]: osdmap e7: 1 total, 0 up, 1 in 2024-04-26T23:36:53.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:53 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T23:36:53.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:53 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T23:36:53.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:53 smithi072 ceph-mon[28184]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:53.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:53 smithi072 ceph-mon[28184]: from='osd.0 [v2:172.21.15.72:6802/1639298875,v1:172.21.15.72:6803/1639298875]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]': finished 2024-04-26T23:36:53.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:53 smithi072 ceph-mon[28184]: osdmap e7: 1 total, 0 up, 1 in 2024-04-26T23:36:53.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:53 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T23:36:53.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:53 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T23:36:53.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:53 smithi136 ceph-mon[33050]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T23:36:53.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:53 smithi136 ceph-mon[33050]: from='osd.0 [v2:172.21.15.72:6802/1639298875,v1:172.21.15.72:6803/1639298875]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]': finished 2024-04-26T23:36:53.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:53 smithi136 ceph-mon[33050]: osdmap e7: 1 total, 0 up, 1 in 2024-04-26T23:36:53.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:53 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T23:36:53.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:53 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T23:36:54.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:54 smithi076 ceph-mon[32774]: purged_snaps scrub starts 2024-04-26T23:36:54.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:54 smithi076 ceph-mon[32774]: purged_snaps scrub ok 2024-04-26T23:36:54.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:54 smithi076 ceph-mon[32774]: osd.0 [v2:172.21.15.72:6802/1639298875,v1:172.21.15.72:6803/1639298875] boot 2024-04-26T23:36:54.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:54 smithi076 ceph-mon[32774]: osdmap e8: 1 total, 1 up, 1 in 2024-04-26T23:36:54.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:54 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T23:36:54.757 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:54 smithi072 ceph-mon[28184]: purged_snaps scrub starts 2024-04-26T23:36:54.757 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:54 smithi072 ceph-mon[28184]: purged_snaps scrub ok 2024-04-26T23:36:54.757 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:54 smithi072 ceph-mon[28184]: osd.0 [v2:172.21.15.72:6802/1639298875,v1:172.21.15.72:6803/1639298875] boot 2024-04-26T23:36:54.757 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:54 smithi072 ceph-mon[28184]: osdmap e8: 1 total, 1 up, 1 in 2024-04-26T23:36:54.758 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:54 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T23:36:54.802 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:54 smithi136 ceph-mon[33050]: purged_snaps scrub starts 2024-04-26T23:36:54.802 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:54 smithi136 ceph-mon[33050]: purged_snaps scrub ok 2024-04-26T23:36:54.802 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:54 smithi136 ceph-mon[33050]: osd.0 [v2:172.21.15.72:6802/1639298875,v1:172.21.15.72:6803/1639298875] boot 2024-04-26T23:36:54.802 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:54 smithi136 ceph-mon[33050]: osdmap e8: 1 total, 1 up, 1 in 2024-04-26T23:36:54.802 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:54 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T23:36:54.859 INFO:teuthology.orchestra.run.smithi076.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-26T23:36:54.859 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-26T23:36:54.859 INFO:teuthology.orchestra.run.smithi076.stderr: stderr: 10+0 records in 2024-04-26T23:36:54.859 INFO:teuthology.orchestra.run.smithi076.stderr:10+0 records out 2024-04-26T23:36:54.859 INFO:teuthology.orchestra.run.smithi076.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0196324 s, 534 MB/s 2024-04-26T23:36:54.859 INFO:teuthology.orchestra.run.smithi076.stderr: stderr: 2024-04-26T23:36:54.860 INFO:teuthology.orchestra.run.smithi076.stderr:--> Zapping successful for: 2024-04-26T23:36:55.494 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi076:vg_nvme/lv_4 2024-04-26T23:36:55.767 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T23:36:55.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:55 smithi072 ceph-mon[28184]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:36:55.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:55 smithi072 ceph-mon[28184]: osdmap e9: 1 total, 1 up, 1 in 2024-04-26T23:36:55.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:55 smithi136 ceph-mon[33050]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:36:55.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:55 smithi136 ceph-mon[33050]: osdmap e9: 1 total, 1 up, 1 in 2024-04-26T23:36:55.889 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:55 smithi076 ceph-mon[32774]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:36:55.889 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:55 smithi076 ceph-mon[32774]: osdmap e9: 1 total, 1 up, 1 in 2024-04-26T23:36:57.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:36:57.289+0000 7f697a4ea700 1 -- 172.21.15.76:0/2407647732 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6974103f10 msgr2=0x7f697410a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:36:57.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:36:57.289+0000 7f697a4ea700 1 --2- 172.21.15.76:0/2407647732 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6974103f10 0x7f697410a7c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f696c00b0d0 tx=0x7f696c00b3e0 comp rx=0 tx=0).stop 2024-04-26T23:36:57.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:36:57.289+0000 7f697a4ea700 1 -- 172.21.15.76:0/2407647732 shutdown_connections 2024-04-26T23:36:57.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:36:57.289+0000 7f697a4ea700 1 --2- 172.21.15.76:0/2407647732 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6974103f10 0x7f697410a7c0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:36:57.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:36:57.289+0000 7f697a4ea700 1 --2- 172.21.15.76:0/2407647732 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6974102370 0x7f6974102790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:36:57.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:36:57.289+0000 7f697a4ea700 1 --2- 172.21.15.76:0/2407647732 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6974103570 0x7f69741039d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:36:57.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:36:57.289+0000 7f697a4ea700 1 -- 172.21.15.76:0/2407647732 >> 172.21.15.76:0/2407647732 conn(0x7f69740fd910 msgr2=0x7f69740ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:36:57.295 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:36:57.289+0000 7f697a4ea700 1 -- 172.21.15.76:0/2407647732 shutdown_connections 2024-04-26T23:36:57.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:36:57.289+0000 7f697a4ea700 1 -- 172.21.15.76:0/2407647732 wait complete. 2024-04-26T23:36:57.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:36:57.290+0000 7f697a4ea700 1 Processor -- start 2024-04-26T23:36:57.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:36:57.290+0000 7f697a4ea700 1 -- start start 2024-04-26T23:36:57.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:36:57.290+0000 7f697a4ea700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6974102370 0x7f697406c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:36:57.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:36:57.290+0000 7f697a4ea700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6974103570 0x7f697419e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:36:57.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:36:57.290+0000 7f697a4ea700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6974103f10 0x7f69741a0950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:36:57.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:36:57.290+0000 7f697a4ea700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f697410deb0 con 0x7f6974102370 2024-04-26T23:36:57.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:36:57.290+0000 7f697a4ea700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f697410dd30 con 0x7f6974103570 2024-04-26T23:36:57.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:36:57.290+0000 7f697a4ea700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f697410e030 con 0x7f6974103f10 2024-04-26T23:36:57.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:36:57.290+0000 7f69737fe700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6974103570 0x7f697419e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:36:57.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:36:57.290+0000 7f69737fe700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6974103570 0x7f697419e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:56054/0 (socket says 172.21.15.76:56054) 2024-04-26T23:36:57.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:36:57.290+0000 7f69737fe700 1 -- 172.21.15.76:0/483524060 learned_addr learned my addr 172.21.15.76:0/483524060 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-26T23:36:57.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:36:57.290+0000 7f6978a87700 1 --2- 172.21.15.76:0/483524060 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6974103f10 0x7f69741a0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:36:57.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:36:57.290+0000 7f69737fe700 1 -- 172.21.15.76:0/483524060 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6974103f10 msgr2=0x7f69741a0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:36:57.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:36:57.290+0000 7f69737fe700 1 --2- 172.21.15.76:0/483524060 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6974103f10 0x7f69741a0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:36:57.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:36:57.290+0000 7f69737fe700 1 -- 172.21.15.76:0/483524060 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6974102370 msgr2=0x7f697406c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:36:57.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:36:57.290+0000 7f69737fe700 1 --2- 172.21.15.76:0/483524060 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6974102370 0x7f697406c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:36:57.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:36:57.290+0000 7f69737fe700 1 -- 172.21.15.76:0/483524060 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69741a1220 con 0x7f6974103570 2024-04-26T23:36:57.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:36:57.290+0000 7f6978a87700 1 --2- 172.21.15.76:0/483524060 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6974103f10 0x7f69741a0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:36:57.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:36:57.291+0000 7f69737fe700 1 --2- 172.21.15.76:0/483524060 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6974103570 0x7f697419e2c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f696800d980 tx=0x7f696800dd40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:36:57.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:36:57.291+0000 7f69717fa700 1 -- 172.21.15.76:0/483524060 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6968017070 con 0x7f6974103570 2024-04-26T23:36:57.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:36:57.291+0000 7f69717fa700 1 -- 172.21.15.76:0/483524060 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6968004830 con 0x7f6974103570 2024-04-26T23:36:57.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:36:57.291+0000 7f69717fa700 1 -- 172.21.15.76:0/483524060 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6968005440 con 0x7f6974103570 2024-04-26T23:36:57.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:36:57.291+0000 7f697a4ea700 1 -- 172.21.15.76:0/483524060 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69741a1510 con 0x7f6974103570 2024-04-26T23:36:57.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:36:57.291+0000 7f697a4ea700 1 -- 172.21.15.76:0/483524060 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69741a1ad0 con 0x7f6974103570 2024-04-26T23:36:57.297 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:36:57.292+0000 7f697a4ea700 1 -- 172.21.15.76:0/483524060 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f697406a110 con 0x7f6974103570 2024-04-26T23:36:57.298 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:36:57.293+0000 7f69717fa700 1 -- 172.21.15.76:0/483524060 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f69680055e0 con 0x7f6974103570 2024-04-26T23:36:57.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:36:57.293+0000 7f69717fa700 1 --2- 172.21.15.76:0/483524060 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f695c0a3c30 0x7f695c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:36:57.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:36:57.294+0000 7f69717fa700 1 -- 172.21.15.76:0/483524060 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f69680ce840 con 0x7f6974103570 2024-04-26T23:36:57.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:36:57.294+0000 7f6973fff700 1 --2- 172.21.15.76:0/483524060 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f695c0a3c30 0x7f695c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:36:57.299 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:36:57.295+0000 7f6973fff700 1 --2- 172.21.15.76:0/483524060 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f695c0a3c30 0x7f695c0a60f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f69741033d0 tx=0x7f6964006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:36:57.301 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:36:57.298+0000 7f69717fa700 1 -- 172.21.15.76:0/483524060 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6968020020 con 0x7f6974103570 2024-04-26T23:36:57.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:36:57.505+0000 7f697a4ea700 1 -- 172.21.15.76:0/483524060 --> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f6974063860 con 0x7f695c0a3c30 2024-04-26T23:36:57.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:57 smithi072 ceph-mon[28184]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:36:57.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:57 smithi136 ceph-mon[33050]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:36:57.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:57 smithi076 ceph-mon[32774]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:36:58.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:58 smithi072 ceph-mon[28184]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:36:58.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:58 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:36:58.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:58 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:36:58.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:58 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:36:58.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:58 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:58.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:58 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:58.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:58 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:36:58.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:58 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:58.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:58 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:36:58.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:58 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:36:58.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:58 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:58.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:58 smithi136 ceph-mon[33050]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:36:58.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:58 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:36:58.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:58 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:36:58.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:58 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:36:58.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:58 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:58.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:58 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:58.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:58 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:36:58.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:58 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:58.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:58 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:36:58.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:58 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:36:58.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:58 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:58.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:58 smithi076 ceph-mon[32774]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:36:58.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:58 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:36:58.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:58 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:36:58.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:58 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:36:58.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:58 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:58.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:58 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:58.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:58 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:36:58.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:58 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:58.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:58 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:36:58.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:58 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:36:58.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:58 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:36:59.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:59 smithi076 ceph-mon[32774]: Detected new or changed devices on smithi072 2024-04-26T23:36:59.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:59 smithi076 ceph-mon[32774]: Adjusting osd_memory_target on smithi072 to 17089M 2024-04-26T23:36:59.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:36:59 smithi076 ceph-mon[32774]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:36:59.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:59 smithi072 ceph-mon[28184]: Detected new or changed devices on smithi072 2024-04-26T23:36:59.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:59 smithi072 ceph-mon[28184]: Adjusting osd_memory_target on smithi072 to 17089M 2024-04-26T23:36:59.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:36:59 smithi072 ceph-mon[28184]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:36:59.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:59 smithi136 ceph-mon[33050]: Detected new or changed devices on smithi072 2024-04-26T23:36:59.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:59 smithi136 ceph-mon[33050]: Adjusting osd_memory_target on smithi072 to 17089M 2024-04-26T23:36:59.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:36:59 smithi136 ceph-mon[33050]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:37:00.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:00 smithi072 ceph-mon[28184]: from='client.? 172.21.15.76:0/1231831022' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e6cac469-780c-4d66-b560-b686072371fb"}]: dispatch 2024-04-26T23:37:00.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:00 smithi072 ceph-mon[28184]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e6cac469-780c-4d66-b560-b686072371fb"}]: dispatch 2024-04-26T23:37:00.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:00 smithi072 ceph-mon[28184]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e6cac469-780c-4d66-b560-b686072371fb"}]': finished 2024-04-26T23:37:00.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:00 smithi072 ceph-mon[28184]: osdmap e10: 2 total, 1 up, 2 in 2024-04-26T23:37:00.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:00 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T23:37:00.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:00 smithi136 ceph-mon[33050]: from='client.? 172.21.15.76:0/1231831022' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e6cac469-780c-4d66-b560-b686072371fb"}]: dispatch 2024-04-26T23:37:00.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:00 smithi136 ceph-mon[33050]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e6cac469-780c-4d66-b560-b686072371fb"}]: dispatch 2024-04-26T23:37:00.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:00 smithi136 ceph-mon[33050]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e6cac469-780c-4d66-b560-b686072371fb"}]': finished 2024-04-26T23:37:00.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:00 smithi136 ceph-mon[33050]: osdmap e10: 2 total, 1 up, 2 in 2024-04-26T23:37:00.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:00 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T23:37:00.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:00 smithi076 ceph-mon[32774]: from='client.? 172.21.15.76:0/1231831022' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e6cac469-780c-4d66-b560-b686072371fb"}]: dispatch 2024-04-26T23:37:00.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:00 smithi076 ceph-mon[32774]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e6cac469-780c-4d66-b560-b686072371fb"}]: dispatch 2024-04-26T23:37:00.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:00 smithi076 ceph-mon[32774]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e6cac469-780c-4d66-b560-b686072371fb"}]': finished 2024-04-26T23:37:00.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:00 smithi076 ceph-mon[32774]: osdmap e10: 2 total, 1 up, 2 in 2024-04-26T23:37:00.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:00 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T23:37:01.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:01 smithi072 ceph-mon[28184]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:37:01.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:01 smithi072 ceph-mon[28184]: from='client.? 172.21.15.76:0/3869604720' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:37:01.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:01 smithi136 ceph-mon[33050]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:37:01.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:01 smithi136 ceph-mon[33050]: from='client.? 172.21.15.76:0/3869604720' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:37:01.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:01 smithi076 ceph-mon[32774]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:37:01.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:01 smithi076 ceph-mon[32774]: from='client.? 172.21.15.76:0/3869604720' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:37:02.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:02 smithi072 ceph-mon[28184]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:37:02.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:02 smithi136 ceph-mon[33050]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:37:02.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:02 smithi076 ceph-mon[32774]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:37:05.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:05 smithi076 ceph-mon[32774]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:37:05.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:05 smithi072 ceph-mon[28184]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:37:05.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:05 smithi136 ceph-mon[33050]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:37:07.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:07 smithi076 ceph-mon[32774]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:37:07.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:07 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-26T23:37:07.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:07 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:37:07.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:07 smithi072 ceph-mon[28184]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:37:07.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:07 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-26T23:37:07.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:07 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:37:07.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:07 smithi136 ceph-mon[33050]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:37:07.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:07 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-26T23:37:07.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:07 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:37:08.557 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:08 smithi076 ceph-mon[32774]: Deploying daemon osd.1 on smithi076 2024-04-26T23:37:08.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:08 smithi072 ceph-mon[28184]: Deploying daemon osd.1 on smithi076 2024-04-26T23:37:08.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:08 smithi136 ceph-mon[33050]: Deploying daemon osd.1 on smithi076 2024-04-26T23:37:09.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:09 smithi076 ceph-mon[32774]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:37:09.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:09 smithi072 ceph-mon[28184]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:37:09.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:09 smithi136 ceph-mon[33050]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:37:11.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:11 smithi076 ceph-mon[32774]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:37:11.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:11 smithi072 ceph-mon[28184]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:37:11.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:11 smithi136 ceph-mon[33050]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:37:13.628 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:13 smithi076 ceph-mon[32774]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:37:13.628 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:13 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:37:13.628 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:13 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:13.628 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:13 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:13.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:13 smithi072 ceph-mon[28184]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:37:13.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:13 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:37:13.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:13 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:13.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:13 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:13.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:13 smithi136 ceph-mon[33050]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:37:13.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:13 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:37:13.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:13 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:13.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:13 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:15.592 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:15 smithi076 ceph-mon[32774]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:37:15.592 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:15 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:15.592 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:15 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:15.592 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:15 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:37:15.592 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:15 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:37:15.593 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:15 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:15.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:15 smithi072 ceph-mon[28184]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:37:15.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:15 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:15.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:15 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:15.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:15 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:37:15.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:15 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:37:15.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:15 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:15.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:15 smithi136 ceph-mon[33050]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:37:15.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:15 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:15.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:15 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:15.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:15 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:37:15.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:15 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:37:15.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:15 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:16.265 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:37:16.262+0000 7f69717fa700 1 -- 172.21.15.76:0/483524060 <== mgr.14152 v2:172.21.15.72:6800/4281177709 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6974063860 con 0x7f695c0a3c30 2024-04-26T23:37:16.266 INFO:teuthology.orchestra.run.smithi076.stdout:Created osd(s) 1 on host 'smithi076' 2024-04-26T23:37:16.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:37:16.264+0000 7f697a4ea700 1 -- 172.21.15.76:0/483524060 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f695c0a3c30 msgr2=0x7f695c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:37:16.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:37:16.264+0000 7f697a4ea700 1 --2- 172.21.15.76:0/483524060 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f695c0a3c30 0x7f695c0a60f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f69741033d0 tx=0x7f6964006d20 comp rx=0 tx=0).stop 2024-04-26T23:37:16.267 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:37:16.264+0000 7f697a4ea700 1 -- 172.21.15.76:0/483524060 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6974103570 msgr2=0x7f697419e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:37:16.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:37:16.264+0000 7f697a4ea700 1 --2- 172.21.15.76:0/483524060 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6974103570 0x7f697419e2c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f696800d980 tx=0x7f696800dd40 comp rx=0 tx=0).stop 2024-04-26T23:37:16.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:37:16.264+0000 7f697a4ea700 1 -- 172.21.15.76:0/483524060 shutdown_connections 2024-04-26T23:37:16.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:37:16.264+0000 7f697a4ea700 1 --2- 172.21.15.76:0/483524060 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f695c0a3c30 0x7f695c0a60f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:37:16.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:37:16.264+0000 7f697a4ea700 1 --2- 172.21.15.76:0/483524060 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6974103f10 0x7f69741a0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:37:16.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:37:16.264+0000 7f697a4ea700 1 --2- 172.21.15.76:0/483524060 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6974102370 0x7f697406c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:37:16.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:37:16.264+0000 7f697a4ea700 1 --2- 172.21.15.76:0/483524060 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6974103570 0x7f697419e2c0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:37:16.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:37:16.264+0000 7f697a4ea700 1 -- 172.21.15.76:0/483524060 >> 172.21.15.76:0/483524060 conn(0x7f69740fd910 msgr2=0x7f6974106870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:37:16.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:37:16.264+0000 7f697a4ea700 1 -- 172.21.15.76:0/483524060 shutdown_connections 2024-04-26T23:37:16.268 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:37:16.264+0000 7f697a4ea700 1 -- 172.21.15.76:0/483524060 wait complete. 2024-04-26T23:37:16.929 DEBUG:teuthology.orchestra.run.smithi076:osd.1> sudo journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@osd.1.service 2024-04-26T23:37:16.931 INFO:tasks.cephadm:Deploying osd.2 on smithi136 with /dev/vg_nvme/lv_4... 2024-04-26T23:37:16.931 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-26T23:37:17.193 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T23:37:17.464 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:17 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:17.464 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:17 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:17.465 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:17 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:37:17.465 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:17 smithi076 ceph-mon[32774]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:37:17.465 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:17 smithi076 ceph-mon[32774]: from='osd.1 [v2:172.21.15.76:6800/4185342193,v1:172.21.15.76:6801/4185342193]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T23:37:17.465 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:17 smithi076 ceph-mon[32774]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T23:37:17.487 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:17 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:17.487 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:17 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:17.487 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:17 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:37:17.487 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:17 smithi136 ceph-mon[33050]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:37:17.487 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:17 smithi136 ceph-mon[33050]: from='osd.1 [v2:172.21.15.76:6800/4185342193,v1:172.21.15.76:6801/4185342193]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T23:37:17.487 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:17 smithi136 ceph-mon[33050]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T23:37:17.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:17 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:17.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:17 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:17.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:17 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:37:17.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:17 smithi072 ceph-mon[28184]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:37:17.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:17 smithi072 ceph-mon[28184]: from='osd.1 [v2:172.21.15.76:6800/4185342193,v1:172.21.15.76:6801/4185342193]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T23:37:17.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:17 smithi072 ceph-mon[28184]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T23:37:18.377 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:18 smithi136 ceph-mon[33050]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-26T23:37:18.378 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:18 smithi136 ceph-mon[33050]: from='osd.1 [v2:172.21.15.76:6800/4185342193,v1:172.21.15.76:6801/4185342193]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-04-26T23:37:18.378 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:18 smithi136 ceph-mon[33050]: osdmap e11: 2 total, 1 up, 2 in 2024-04-26T23:37:18.378 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:18 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T23:37:18.378 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:18 smithi136 ceph-mon[33050]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-04-26T23:37:18.397 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:18 smithi076 ceph-mon[32774]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-26T23:37:18.397 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:18 smithi076 ceph-mon[32774]: from='osd.1 [v2:172.21.15.76:6800/4185342193,v1:172.21.15.76:6801/4185342193]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-04-26T23:37:18.397 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:18 smithi076 ceph-mon[32774]: osdmap e11: 2 total, 1 up, 2 in 2024-04-26T23:37:18.397 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:18 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T23:37:18.398 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:18 smithi076 ceph-mon[32774]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-04-26T23:37:18.398 INFO:journalctl@ceph.osd.1.smithi076.stdout:Apr 26 23:37:18 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-1[37364]: 2024-04-26T23:37:18.264+0000 7f4b6ba5c700 -1 osd.1 0 waiting for initial osdmap 2024-04-26T23:37:18.398 INFO:journalctl@ceph.osd.1.smithi076.stdout:Apr 26 23:37:18 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-1[37364]: 2024-04-26T23:37:18.268+0000 7f4b6784f700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T23:37:18.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:18 smithi072 ceph-mon[28184]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-26T23:37:18.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:18 smithi072 ceph-mon[28184]: from='osd.1 [v2:172.21.15.76:6800/4185342193,v1:172.21.15.76:6801/4185342193]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-04-26T23:37:18.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:18 smithi072 ceph-mon[28184]: osdmap e11: 2 total, 1 up, 2 in 2024-04-26T23:37:18.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:18 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T23:37:18.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:18 smithi072 ceph-mon[28184]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-04-26T23:37:18.908 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-26T23:37:18.908 INFO:teuthology.orchestra.run.smithi136.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-26T23:37:18.908 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10+0 records in 2024-04-26T23:37:18.908 INFO:teuthology.orchestra.run.smithi136.stderr:10+0 records out 2024-04-26T23:37:18.908 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0094333 s, 1.1 GB/s 2024-04-26T23:37:18.908 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping successful for: 2024-04-26T23:37:19.517 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi136:vg_nvme/lv_4 2024-04-26T23:37:19.539 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:19 smithi136 ceph-mon[33050]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-04-26T23:37:19.539 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:19 smithi136 ceph-mon[33050]: osdmap e12: 2 total, 1 up, 2 in 2024-04-26T23:37:19.539 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:19 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T23:37:19.539 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:19 smithi136 ceph-mon[33050]: pgmap v72: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:37:19.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:19 smithi072 ceph-mon[28184]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-04-26T23:37:19.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:19 smithi072 ceph-mon[28184]: osdmap e12: 2 total, 1 up, 2 in 2024-04-26T23:37:19.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:19 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T23:37:19.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:19 smithi072 ceph-mon[28184]: pgmap v72: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:37:19.625 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:19 smithi076 ceph-mon[32774]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-04-26T23:37:19.625 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:19 smithi076 ceph-mon[32774]: osdmap e12: 2 total, 1 up, 2 in 2024-04-26T23:37:19.625 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:19 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T23:37:19.625 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:19 smithi076 ceph-mon[32774]: pgmap v72: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T23:37:19.754 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T23:37:20.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:20 smithi072 ceph-mon[28184]: purged_snaps scrub starts 2024-04-26T23:37:20.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:20 smithi072 ceph-mon[28184]: purged_snaps scrub ok 2024-04-26T23:37:20.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:20 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T23:37:20.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:20 smithi072 ceph-mon[28184]: osd.1 [v2:172.21.15.76:6800/4185342193,v1:172.21.15.76:6801/4185342193] boot 2024-04-26T23:37:20.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:20 smithi072 ceph-mon[28184]: osdmap e13: 2 total, 2 up, 2 in 2024-04-26T23:37:20.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:20 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T23:37:20.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:20 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:20.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:20 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:20.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:20 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:37:20.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:20 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:20.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:20 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:37:20.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:20 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:37:20.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:20 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:20.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:20 smithi136 ceph-mon[33050]: purged_snaps scrub starts 2024-04-26T23:37:20.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:20 smithi136 ceph-mon[33050]: purged_snaps scrub ok 2024-04-26T23:37:20.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:20 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T23:37:20.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:20 smithi136 ceph-mon[33050]: osd.1 [v2:172.21.15.76:6800/4185342193,v1:172.21.15.76:6801/4185342193] boot 2024-04-26T23:37:20.551 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:20 smithi136 ceph-mon[33050]: osdmap e13: 2 total, 2 up, 2 in 2024-04-26T23:37:20.551 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:20 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T23:37:20.551 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:20 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:20.551 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:20 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:20.551 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:20 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:37:20.551 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:20 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:20.551 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:20 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:37:20.551 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:20 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:37:20.551 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:20 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:20.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:20 smithi076 ceph-mon[32774]: purged_snaps scrub starts 2024-04-26T23:37:20.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:20 smithi076 ceph-mon[32774]: purged_snaps scrub ok 2024-04-26T23:37:20.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:20 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T23:37:20.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:20 smithi076 ceph-mon[32774]: osd.1 [v2:172.21.15.76:6800/4185342193,v1:172.21.15.76:6801/4185342193] boot 2024-04-26T23:37:20.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:20 smithi076 ceph-mon[32774]: osdmap e13: 2 total, 2 up, 2 in 2024-04-26T23:37:20.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:20 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T23:37:20.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:20 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:20.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:20 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:20.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:20 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:37:20.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:20 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:20.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:20 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:37:20.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:20 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:37:20.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:20 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:21.329 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:21.324+0000 7f0b41beb700 1 -- 172.21.15.136:0/1581248668 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b3c103f40 msgr2=0x7f0b3c10a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:37:21.329 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:21.324+0000 7f0b41beb700 1 --2- 172.21.15.136:0/1581248668 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b3c103f40 0x7f0b3c10a7f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0b3400b0a0 tx=0x7f0b3400b3b0 comp rx=0 tx=0).stop 2024-04-26T23:37:21.329 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:21.324+0000 7f0b41beb700 1 -- 172.21.15.136:0/1581248668 shutdown_connections 2024-04-26T23:37:21.329 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:21.324+0000 7f0b41beb700 1 --2- 172.21.15.136:0/1581248668 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b3c103f40 0x7f0b3c10a7f0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:37:21.329 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:21.324+0000 7f0b41beb700 1 --2- 172.21.15.136:0/1581248668 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0b3c1023a0 0x7f0b3c1027c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:37:21.329 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:21.324+0000 7f0b41beb700 1 --2- 172.21.15.136:0/1581248668 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0b3c1035a0 0x7f0b3c103a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:37:21.330 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:21.324+0000 7f0b41beb700 1 -- 172.21.15.136:0/1581248668 >> 172.21.15.136:0/1581248668 conn(0x7f0b3c0fd980 msgr2=0x7f0b3c0ffda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:37:21.330 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:21.324+0000 7f0b41beb700 1 -- 172.21.15.136:0/1581248668 shutdown_connections 2024-04-26T23:37:21.330 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:21.324+0000 7f0b41beb700 1 -- 172.21.15.136:0/1581248668 wait complete. 2024-04-26T23:37:21.330 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:21.324+0000 7f0b41beb700 1 Processor -- start 2024-04-26T23:37:21.330 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:21.324+0000 7f0b41beb700 1 -- start start 2024-04-26T23:37:21.330 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:21.325+0000 7f0b41beb700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b3c1023a0 0x7f0b3c1996e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:37:21.330 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:21.325+0000 7f0b41beb700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0b3c1035a0 0x7f0b3c199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:37:21.330 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:21.325+0000 7f0b41beb700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0b3c103f40 0x7f0b3c1a0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:37:21.330 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:21.325+0000 7f0b41beb700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f0b3c10df30 con 0x7f0b3c103f40 2024-04-26T23:37:21.330 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:21.325+0000 7f0b41beb700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f0b3c10ddb0 con 0x7f0b3c1035a0 2024-04-26T23:37:21.330 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:21.325+0000 7f0b41beb700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f0b3c10e0b0 con 0x7f0b3c1023a0 2024-04-26T23:37:21.331 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:21.325+0000 7f0b3b7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b3c1023a0 0x7f0b3c1996e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:37:21.331 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:21.325+0000 7f0b3b7fe700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b3c1023a0 0x7f0b3c1996e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36462/0 (socket says 172.21.15.136:36462) 2024-04-26T23:37:21.331 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:21.325+0000 7f0b3b7fe700 1 -- 172.21.15.136:0/3647616737 learned_addr learned my addr 172.21.15.136:0/3647616737 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-26T23:37:21.331 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:21.325+0000 7f0b3affd700 1 --2- 172.21.15.136:0/3647616737 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0b3c1035a0 0x7f0b3c199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:37:21.331 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:21.325+0000 7f0b3b7fe700 1 -- 172.21.15.136:0/3647616737 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0b3c1035a0 msgr2=0x7f0b3c199c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:37:21.331 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:21.325+0000 7f0b3b7fe700 1 --2- 172.21.15.136:0/3647616737 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0b3c1035a0 0x7f0b3c199c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:37:21.331 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:21.325+0000 7f0b3b7fe700 1 -- 172.21.15.136:0/3647616737 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0b3c103f40 msgr2=0x7f0b3c1a0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:37:21.331 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:21.325+0000 7f0b3b7fe700 1 --2- 172.21.15.136:0/3647616737 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0b3c103f40 0x7f0b3c1a0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:37:21.331 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:21.325+0000 7f0b3b7fe700 1 -- 172.21.15.136:0/3647616737 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b3c1a1340 con 0x7f0b3c1023a0 2024-04-26T23:37:21.331 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:21.325+0000 7f0b3affd700 1 --2- 172.21.15.136:0/3647616737 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0b3c1035a0 0x7f0b3c199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:37:21.332 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:21.325+0000 7f0b3b7fe700 1 --2- 172.21.15.136:0/3647616737 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b3c1023a0 0x7f0b3c1996e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c00b530 tx=0x7f0b2c00b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:37:21.332 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:21.325+0000 7f0b38ff9700 1 -- 172.21.15.136:0/3647616737 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b2c015020 con 0x7f0b3c1023a0 2024-04-26T23:37:21.332 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:21.325+0000 7f0b38ff9700 1 -- 172.21.15.136:0/3647616737 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b2c011ce0 con 0x7f0b3c1023a0 2024-04-26T23:37:21.332 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:21.325+0000 7f0b41beb700 1 -- 172.21.15.136:0/3647616737 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b3c1a1630 con 0x7f0b3c1023a0 2024-04-26T23:37:21.332 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:21.325+0000 7f0b38ff9700 1 -- 172.21.15.136:0/3647616737 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b2c010d60 con 0x7f0b3c1023a0 2024-04-26T23:37:21.332 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:21.325+0000 7f0b41beb700 1 -- 172.21.15.136:0/3647616737 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b3c1a1bf0 con 0x7f0b3c1023a0 2024-04-26T23:37:21.333 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:21.326+0000 7f0b41beb700 1 -- 172.21.15.136:0/3647616737 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b3c06a110 con 0x7f0b3c1023a0 2024-04-26T23:37:21.333 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:21.327+0000 7f0b38ff9700 1 -- 172.21.15.136:0/3647616737 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f0b2c018020 con 0x7f0b3c1023a0 2024-04-26T23:37:21.333 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:21.328+0000 7f0b38ff9700 1 --2- 172.21.15.136:0/3647616737 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f0b240a3bb0 0x7f0b240a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:37:21.333 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:21.328+0000 7f0b38ff9700 1 -- 172.21.15.136:0/3647616737 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f0b2c0ce8e0 con 0x7f0b3c1023a0 2024-04-26T23:37:21.333 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:21.328+0000 7f0b3affd700 1 --2- 172.21.15.136:0/3647616737 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f0b240a3bb0 0x7f0b240a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:37:21.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:21.329+0000 7f0b3affd700 1 --2- 172.21.15.136:0/3647616737 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f0b240a3bb0 0x7f0b240a6070 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0b300099d0 tx=0x7f0b30008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:37:21.335 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:21.332+0000 7f0b38ff9700 1 -- 172.21.15.136:0/3647616737 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f0b2c01f020 con 0x7f0b3c1023a0 2024-04-26T23:37:21.540 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:21.537+0000 7f0b41beb700 1 -- 172.21.15.136:0/3647616737 --> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f0b3c0ffbb0 con 0x7f0b240a3bb0 2024-04-26T23:37:21.541 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:21 smithi136 ceph-mon[33050]: Detected new or changed devices on smithi076 2024-04-26T23:37:21.541 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:21 smithi136 ceph-mon[33050]: Adjusting osd_memory_target on smithi076 to 17088M 2024-04-26T23:37:21.541 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:21 smithi136 ceph-mon[33050]: osdmap e14: 2 total, 2 up, 2 in 2024-04-26T23:37:21.541 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:21 smithi136 ceph-mon[33050]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:37:21.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:21 smithi072 ceph-mon[28184]: Detected new or changed devices on smithi076 2024-04-26T23:37:21.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:21 smithi072 ceph-mon[28184]: Adjusting osd_memory_target on smithi076 to 17088M 2024-04-26T23:37:21.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:21 smithi072 ceph-mon[28184]: osdmap e14: 2 total, 2 up, 2 in 2024-04-26T23:37:21.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:21 smithi072 ceph-mon[28184]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:37:21.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:21 smithi076 ceph-mon[32774]: Detected new or changed devices on smithi076 2024-04-26T23:37:21.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:21 smithi076 ceph-mon[32774]: Adjusting osd_memory_target on smithi076 to 17088M 2024-04-26T23:37:21.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:21 smithi076 ceph-mon[32774]: osdmap e14: 2 total, 2 up, 2 in 2024-04-26T23:37:21.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:21 smithi076 ceph-mon[32774]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:37:22.476 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:22 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:37:22.477 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:22 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:37:22.477 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:22 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:37:22.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:22 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:37:22.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:22 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:37:22.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:22 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:37:22.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:22 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:37:22.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:22 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:37:22.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:22 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:37:23.421 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:23 smithi136 ceph-mon[33050]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:37:23.421 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:23 smithi136 ceph-mon[33050]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:37:23.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:23 smithi076 ceph-mon[32774]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:37:23.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:23 smithi076 ceph-mon[32774]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:37:23.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:23 smithi072 ceph-mon[28184]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:37:23.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:23 smithi072 ceph-mon[28184]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:37:24.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:24 smithi136 ceph-mon[33050]: from='client.? 172.21.15.136:0/4013997130' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "838a49fc-18de-446c-8336-67b3e05da231"}]: dispatch 2024-04-26T23:37:24.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:24 smithi136 ceph-mon[33050]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "838a49fc-18de-446c-8336-67b3e05da231"}]: dispatch 2024-04-26T23:37:24.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:24 smithi136 ceph-mon[33050]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "838a49fc-18de-446c-8336-67b3e05da231"}]': finished 2024-04-26T23:37:24.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:24 smithi136 ceph-mon[33050]: osdmap e15: 3 total, 2 up, 3 in 2024-04-26T23:37:24.575 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:24 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T23:37:24.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:24 smithi076 ceph-mon[32774]: from='client.? 172.21.15.136:0/4013997130' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "838a49fc-18de-446c-8336-67b3e05da231"}]: dispatch 2024-04-26T23:37:24.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:24 smithi076 ceph-mon[32774]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "838a49fc-18de-446c-8336-67b3e05da231"}]: dispatch 2024-04-26T23:37:24.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:24 smithi076 ceph-mon[32774]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "838a49fc-18de-446c-8336-67b3e05da231"}]': finished 2024-04-26T23:37:24.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:24 smithi076 ceph-mon[32774]: osdmap e15: 3 total, 2 up, 3 in 2024-04-26T23:37:24.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:24 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T23:37:24.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:24 smithi072 ceph-mon[28184]: from='client.? 172.21.15.136:0/4013997130' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "838a49fc-18de-446c-8336-67b3e05da231"}]: dispatch 2024-04-26T23:37:24.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:24 smithi072 ceph-mon[28184]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "838a49fc-18de-446c-8336-67b3e05da231"}]: dispatch 2024-04-26T23:37:24.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:24 smithi072 ceph-mon[28184]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "838a49fc-18de-446c-8336-67b3e05da231"}]': finished 2024-04-26T23:37:24.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:24 smithi072 ceph-mon[28184]: osdmap e15: 3 total, 2 up, 3 in 2024-04-26T23:37:24.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:24 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T23:37:25.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:25 smithi076 ceph-mon[32774]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:37:25.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:25 smithi076 ceph-mon[32774]: from='client.? 172.21.15.136:0/3184972592' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:37:25.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:25 smithi072 ceph-mon[28184]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:37:25.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:25 smithi072 ceph-mon[28184]: from='client.? 172.21.15.136:0/3184972592' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:37:25.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:25 smithi136 ceph-mon[33050]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:37:25.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:25 smithi136 ceph-mon[33050]: from='client.? 172.21.15.136:0/3184972592' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:37:27.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:27 smithi076 ceph-mon[32774]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:37:27.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:27 smithi072 ceph-mon[28184]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:37:27.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:27 smithi136 ceph-mon[33050]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:37:29.408 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:29 smithi136 ceph-mon[33050]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:37:29.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:29 smithi076 ceph-mon[32774]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:37:29.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:29 smithi072 ceph-mon[28184]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:37:31.549 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:31 smithi136 ceph-mon[33050]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:37:31.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:31 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-26T23:37:31.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:31 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:37:31.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:31 smithi076 ceph-mon[32774]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:37:31.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:31 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-26T23:37:31.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:31 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:37:31.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:31 smithi072 ceph-mon[28184]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:37:31.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:31 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-26T23:37:31.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:31 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:37:32.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:32 smithi072 ceph-mon[28184]: Deploying daemon osd.2 on smithi136 2024-04-26T23:37:32.549 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:32 smithi136 ceph-mon[33050]: Deploying daemon osd.2 on smithi136 2024-04-26T23:37:32.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:32 smithi076 ceph-mon[32774]: Deploying daemon osd.2 on smithi136 2024-04-26T23:37:33.549 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:33 smithi136 ceph-mon[33050]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:37:33.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:33 smithi076 ceph-mon[32774]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:37:33.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:33 smithi072 ceph-mon[28184]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:37:35.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:35 smithi136 ceph-mon[33050]: pgmap v83: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:37:35.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:35 smithi076 ceph-mon[32774]: pgmap v83: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:37:35.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:35 smithi072 ceph-mon[28184]: pgmap v83: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:37:37.462 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:37 smithi136 ceph-mon[33050]: pgmap v84: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:37:37.463 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:37 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:37.463 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:37 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:37.463 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:37 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:37:37.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:37 smithi076 ceph-mon[32774]: pgmap v84: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:37:37.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:37 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:37.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:37 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:37.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:37 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:37:37.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:37 smithi072 ceph-mon[28184]: pgmap v84: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:37:37.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:37 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:37.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:37 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:37.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:37 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:37:39.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:39 smithi076 ceph-mon[32774]: pgmap v85: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:37:39.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:39 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:39.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:39 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:39.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:39 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:37:39.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:39 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:37:39.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:39 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:39.779 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:39 smithi136 ceph-mon[33050]: pgmap v85: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:37:39.779 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:39 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:39.779 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:39 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:39.780 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:39 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:37:39.780 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:39 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:37:39.780 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:39 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:39.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:39 smithi072 ceph-mon[28184]: pgmap v85: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:37:39.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:39 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:39.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:39 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:39.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:39 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:37:39.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:39 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:37:39.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:39 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:40.473 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:40.469+0000 7f0b38ff9700 1 -- 172.21.15.136:0/3647616737 <== mgr.14152 v2:172.21.15.72:6800/4281177709 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0b3c0ffbb0 con 0x7f0b240a3bb0 2024-04-26T23:37:40.474 INFO:teuthology.orchestra.run.smithi136.stdout:Created osd(s) 2 on host 'smithi136' 2024-04-26T23:37:40.475 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:40.472+0000 7f0b41beb700 1 -- 172.21.15.136:0/3647616737 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f0b240a3bb0 msgr2=0x7f0b240a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:37:40.475 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:40.472+0000 7f0b41beb700 1 --2- 172.21.15.136:0/3647616737 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f0b240a3bb0 0x7f0b240a6070 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0b300099d0 tx=0x7f0b30008040 comp rx=0 tx=0).stop 2024-04-26T23:37:40.475 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:40.472+0000 7f0b41beb700 1 -- 172.21.15.136:0/3647616737 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b3c1023a0 msgr2=0x7f0b3c1996e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:37:40.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:40.472+0000 7f0b41beb700 1 --2- 172.21.15.136:0/3647616737 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b3c1023a0 0x7f0b3c1996e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c00b530 tx=0x7f0b2c00b8f0 comp rx=0 tx=0).stop 2024-04-26T23:37:40.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:40.472+0000 7f0b41beb700 1 -- 172.21.15.136:0/3647616737 shutdown_connections 2024-04-26T23:37:40.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:40.472+0000 7f0b41beb700 1 --2- 172.21.15.136:0/3647616737 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f0b240a3bb0 0x7f0b240a6070 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:37:40.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:40.472+0000 7f0b41beb700 1 --2- 172.21.15.136:0/3647616737 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0b3c103f40 0x7f0b3c1a0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:37:40.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:40.472+0000 7f0b41beb700 1 --2- 172.21.15.136:0/3647616737 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b3c1023a0 0x7f0b3c1996e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:37:40.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:40.472+0000 7f0b41beb700 1 --2- 172.21.15.136:0/3647616737 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0b3c1035a0 0x7f0b3c199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:37:40.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:40.472+0000 7f0b41beb700 1 -- 172.21.15.136:0/3647616737 >> 172.21.15.136:0/3647616737 conn(0x7f0b3c0fd980 msgr2=0x7f0b3c0ff4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:37:40.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:40.472+0000 7f0b41beb700 1 -- 172.21.15.136:0/3647616737 shutdown_connections 2024-04-26T23:37:40.476 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:37:40.472+0000 7f0b41beb700 1 -- 172.21.15.136:0/3647616737 wait complete. 2024-04-26T23:37:41.125 DEBUG:teuthology.orchestra.run.smithi136:osd.2> sudo journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@osd.2.service 2024-04-26T23:37:41.167 INFO:tasks.cephadm:Deploying osd.3 on smithi072 with /dev/vg_nvme/lv_3... 2024-04-26T23:37:41.167 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-26T23:37:41.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:41 smithi072 ceph-mon[28184]: pgmap v86: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:37:41.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:41 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:41.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:41 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:41.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:41 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:37:41.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:41 smithi072 ceph-mon[28184]: from='osd.2 [v2:172.21.15.136:6800/3033860799,v1:172.21.15.136:6801/3033860799]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T23:37:41.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:41 smithi072 ceph-mon[28184]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T23:37:41.607 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:37:41.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:41 smithi136 ceph-mon[33050]: pgmap v86: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:37:41.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:41 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:41.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:41 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:41.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:41 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:37:41.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:41 smithi136 ceph-mon[33050]: from='osd.2 [v2:172.21.15.136:6800/3033860799,v1:172.21.15.136:6801/3033860799]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T23:37:41.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:41 smithi136 ceph-mon[33050]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T23:37:41.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:41 smithi076 ceph-mon[32774]: pgmap v86: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:37:41.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:41 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:41.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:41 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:41.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:41 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:37:41.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:41 smithi076 ceph-mon[32774]: from='osd.2 [v2:172.21.15.136:6800/3033860799,v1:172.21.15.136:6801/3033860799]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T23:37:41.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:41 smithi076 ceph-mon[32774]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T23:37:42.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:42 smithi136 ceph-mon[33050]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-26T23:37:42.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:42 smithi136 ceph-mon[33050]: osdmap e16: 3 total, 2 up, 3 in 2024-04-26T23:37:42.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:42 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T23:37:42.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:42 smithi136 ceph-mon[33050]: from='osd.2 [v2:172.21.15.136:6800/3033860799,v1:172.21.15.136:6801/3033860799]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-04-26T23:37:42.634 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:42 smithi136 ceph-mon[33050]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-04-26T23:37:42.635 INFO:journalctl@ceph.osd.2.smithi136.stdout:Apr 26 23:37:42 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-2[38195]: 2024-04-26T23:37:42.446+0000 7fbd5dc91700 -1 osd.2 0 waiting for initial osdmap 2024-04-26T23:37:42.635 INFO:journalctl@ceph.osd.2.smithi136.stdout:Apr 26 23:37:42 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-2[38195]: 2024-04-26T23:37:42.454+0000 7fbd59a84700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T23:37:42.765 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:42 smithi072 ceph-mon[28184]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-26T23:37:42.765 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:42 smithi072 ceph-mon[28184]: osdmap e16: 3 total, 2 up, 3 in 2024-04-26T23:37:42.765 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:42 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T23:37:42.765 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:42 smithi072 ceph-mon[28184]: from='osd.2 [v2:172.21.15.136:6800/3033860799,v1:172.21.15.136:6801/3033860799]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-04-26T23:37:42.765 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:42 smithi072 ceph-mon[28184]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-04-26T23:37:42.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:42 smithi076 ceph-mon[32774]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-26T23:37:42.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:42 smithi076 ceph-mon[32774]: osdmap e16: 3 total, 2 up, 3 in 2024-04-26T23:37:42.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:42 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T23:37:42.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:42 smithi076 ceph-mon[32774]: from='osd.2 [v2:172.21.15.136:6800/3033860799,v1:172.21.15.136:6801/3033860799]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-04-26T23:37:42.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:42 smithi076 ceph-mon[32774]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-04-26T23:37:43.497 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:43 smithi072 ceph-mon[28184]: pgmap v88: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:37:43.497 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:43 smithi072 ceph-mon[28184]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-04-26T23:37:43.498 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:43 smithi072 ceph-mon[28184]: osdmap e17: 3 total, 2 up, 3 in 2024-04-26T23:37:43.498 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:43 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T23:37:43.498 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:43 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T23:37:43.784 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:43 smithi136 ceph-mon[33050]: pgmap v88: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:37:43.784 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:43 smithi136 ceph-mon[33050]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-04-26T23:37:43.784 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:43 smithi136 ceph-mon[33050]: osdmap e17: 3 total, 2 up, 3 in 2024-04-26T23:37:43.784 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:43 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T23:37:43.784 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:43 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T23:37:43.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:43 smithi076 ceph-mon[32774]: pgmap v88: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:37:43.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:43 smithi076 ceph-mon[32774]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-04-26T23:37:43.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:43 smithi076 ceph-mon[32774]: osdmap e17: 3 total, 2 up, 3 in 2024-04-26T23:37:43.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:43 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T23:37:43.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:43 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T23:37:44.497 INFO:teuthology.orchestra.run.smithi072.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-26T23:37:44.497 INFO:teuthology.orchestra.run.smithi072.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-26T23:37:44.497 INFO:teuthology.orchestra.run.smithi072.stderr: stderr: 10+0 records in 2024-04-26T23:37:44.497 INFO:teuthology.orchestra.run.smithi072.stderr:10+0 records out 2024-04-26T23:37:44.497 INFO:teuthology.orchestra.run.smithi072.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0403128 s, 260 MB/s 2024-04-26T23:37:44.497 INFO:teuthology.orchestra.run.smithi072.stderr:--> Zapping successful for: 2024-04-26T23:37:44.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:44 smithi072 ceph-mon[28184]: purged_snaps scrub starts 2024-04-26T23:37:44.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:44 smithi072 ceph-mon[28184]: purged_snaps scrub ok 2024-04-26T23:37:44.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:44 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T23:37:44.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:44 smithi072 ceph-mon[28184]: osd.2 [v2:172.21.15.136:6800/3033860799,v1:172.21.15.136:6801/3033860799] boot 2024-04-26T23:37:44.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:44 smithi072 ceph-mon[28184]: osdmap e18: 3 total, 3 up, 3 in 2024-04-26T23:37:44.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:44 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T23:37:44.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:44 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:44.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:44 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:44.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:44 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:37:44.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:44 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:44.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:44 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:37:44.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:44 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:37:44.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:44 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:44.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:44 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-26T23:37:44.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:44 smithi136 ceph-mon[33050]: purged_snaps scrub starts 2024-04-26T23:37:44.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:44 smithi136 ceph-mon[33050]: purged_snaps scrub ok 2024-04-26T23:37:44.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:44 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T23:37:44.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:44 smithi136 ceph-mon[33050]: osd.2 [v2:172.21.15.136:6800/3033860799,v1:172.21.15.136:6801/3033860799] boot 2024-04-26T23:37:44.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:44 smithi136 ceph-mon[33050]: osdmap e18: 3 total, 3 up, 3 in 2024-04-26T23:37:44.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:44 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T23:37:44.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:44 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:44.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:44 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:44.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:44 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:37:44.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:44 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:44.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:44 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:37:44.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:44 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:37:44.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:44 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:44.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:44 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-26T23:37:44.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:44 smithi076 ceph-mon[32774]: purged_snaps scrub starts 2024-04-26T23:37:44.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:44 smithi076 ceph-mon[32774]: purged_snaps scrub ok 2024-04-26T23:37:44.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:44 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T23:37:44.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:44 smithi076 ceph-mon[32774]: osd.2 [v2:172.21.15.136:6800/3033860799,v1:172.21.15.136:6801/3033860799] boot 2024-04-26T23:37:44.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:44 smithi076 ceph-mon[32774]: osdmap e18: 3 total, 3 up, 3 in 2024-04-26T23:37:44.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:44 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T23:37:44.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:44 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:44.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:44 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:44.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:44 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:37:44.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:44 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:44.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:44 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:37:44.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:44 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:37:44.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:44 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:44.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:44 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-26T23:37:45.487 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi072:vg_nvme/lv_3 2024-04-26T23:37:45.666 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:45 smithi072 ceph-mon[28184]: Detected new or changed devices on smithi136 2024-04-26T23:37:45.666 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:45 smithi072 ceph-mon[28184]: Adjusting osd_memory_target on smithi136 to 17088M 2024-04-26T23:37:45.666 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:45 smithi072 ceph-mon[28184]: pgmap v91: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:37:45.666 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:45 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-26T23:37:45.666 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:45 smithi072 ceph-mon[28184]: osdmap e19: 3 total, 3 up, 3 in 2024-04-26T23:37:45.666 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:45 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T23:37:45.666 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:45 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-26T23:37:45.667 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:45 smithi072 ceph-mon[28184]: osdmap e20: 3 total, 3 up, 3 in 2024-04-26T23:37:45.667 INFO:journalctl@ceph.osd.0.smithi072.stdout:Apr 26 23:37:45 smithi072 sudo[44773]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T23:37:45.667 INFO:journalctl@ceph.osd.0.smithi072.stdout:Apr 26 23:37:45 smithi072 sudo[44773]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T23:37:45.667 INFO:journalctl@ceph.osd.0.smithi072.stdout:Apr 26 23:37:45 smithi072 sudo[44773]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T23:37:45.667 INFO:journalctl@ceph.osd.0.smithi072.stdout:Apr 26 23:37:45 smithi072 sudo[44773]: pam_unix(sudo:session): session closed for user root 2024-04-26T23:37:45.753 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:45 smithi076 ceph-mon[32774]: Detected new or changed devices on smithi136 2024-04-26T23:37:45.753 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:45 smithi076 ceph-mon[32774]: Adjusting osd_memory_target on smithi136 to 17088M 2024-04-26T23:37:45.753 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:45 smithi076 ceph-mon[32774]: pgmap v91: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:37:45.753 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:45 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-26T23:37:45.753 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:45 smithi076 ceph-mon[32774]: osdmap e19: 3 total, 3 up, 3 in 2024-04-26T23:37:45.753 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:45 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T23:37:45.753 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:45 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-26T23:37:45.754 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:45 smithi076 ceph-mon[32774]: osdmap e20: 3 total, 3 up, 3 in 2024-04-26T23:37:45.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:45 smithi136 ceph-mon[33050]: Detected new or changed devices on smithi136 2024-04-26T23:37:45.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:45 smithi136 ceph-mon[33050]: Adjusting osd_memory_target on smithi136 to 17088M 2024-04-26T23:37:45.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:45 smithi136 ceph-mon[33050]: pgmap v91: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-26T23:37:45.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:45 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-26T23:37:45.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:45 smithi136 ceph-mon[33050]: osdmap e19: 3 total, 3 up, 3 in 2024-04-26T23:37:45.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:45 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T23:37:45.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:45 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-26T23:37:45.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:45 smithi136 ceph-mon[33050]: osdmap e20: 3 total, 3 up, 3 in 2024-04-26T23:37:45.932 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:45 smithi072 sudo[44806]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T23:37:45.933 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:45 smithi072 sudo[44806]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T23:37:45.933 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:45 smithi072 sudo[44806]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T23:37:45.933 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:45 smithi072 sudo[44806]: pam_unix(sudo:session): session closed for user root 2024-04-26T23:37:45.933 INFO:journalctl@ceph.osd.0.smithi072.stdout:Apr 26 23:37:45 smithi072 sudo[44776]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T23:37:45.933 INFO:journalctl@ceph.osd.0.smithi072.stdout:Apr 26 23:37:45 smithi072 sudo[44776]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T23:37:45.933 INFO:journalctl@ceph.osd.0.smithi072.stdout:Apr 26 23:37:45 smithi072 sudo[44776]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T23:37:45.933 INFO:journalctl@ceph.osd.0.smithi072.stdout:Apr 26 23:37:45 smithi072 sudo[44776]: pam_unix(sudo:session): session closed for user root 2024-04-26T23:37:45.944 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:37:46.018 INFO:journalctl@ceph.osd.1.smithi076.stdout:Apr 26 23:37:45 smithi076 sudo[41050]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T23:37:46.018 INFO:journalctl@ceph.osd.1.smithi076.stdout:Apr 26 23:37:45 smithi076 sudo[41050]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T23:37:46.018 INFO:journalctl@ceph.osd.1.smithi076.stdout:Apr 26 23:37:45 smithi076 sudo[41050]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T23:37:46.018 INFO:journalctl@ceph.osd.1.smithi076.stdout:Apr 26 23:37:45 smithi076 sudo[41050]: pam_unix(sudo:session): session closed for user root 2024-04-26T23:37:46.018 INFO:journalctl@ceph.osd.1.smithi076.stdout:Apr 26 23:37:45 smithi076 sudo[41053]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T23:37:46.018 INFO:journalctl@ceph.osd.1.smithi076.stdout:Apr 26 23:37:45 smithi076 sudo[41053]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T23:37:46.018 INFO:journalctl@ceph.osd.1.smithi076.stdout:Apr 26 23:37:45 smithi076 sudo[41053]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T23:37:46.018 INFO:journalctl@ceph.osd.1.smithi076.stdout:Apr 26 23:37:45 smithi076 sudo[41053]: pam_unix(sudo:session): session closed for user root 2024-04-26T23:37:46.264 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:45 smithi072 sudo[44821]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T23:37:46.264 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:45 smithi072 sudo[44821]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T23:37:46.264 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:45 smithi072 sudo[44821]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T23:37:46.264 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:45 smithi072 sudo[44821]: pam_unix(sudo:session): session closed for user root 2024-04-26T23:37:46.300 INFO:journalctl@ceph.osd.2.smithi136.stdout:Apr 26 23:37:45 smithi136 sudo[41807]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T23:37:46.300 INFO:journalctl@ceph.osd.2.smithi136.stdout:Apr 26 23:37:45 smithi136 sudo[41807]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T23:37:46.300 INFO:journalctl@ceph.osd.2.smithi136.stdout:Apr 26 23:37:45 smithi136 sudo[41807]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T23:37:46.300 INFO:journalctl@ceph.osd.2.smithi136.stdout:Apr 26 23:37:45 smithi136 sudo[41807]: pam_unix(sudo:session): session closed for user root 2024-04-26T23:37:46.300 INFO:journalctl@ceph.osd.2.smithi136.stdout:Apr 26 23:37:45 smithi136 sudo[41810]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T23:37:46.300 INFO:journalctl@ceph.osd.2.smithi136.stdout:Apr 26 23:37:45 smithi136 sudo[41810]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T23:37:46.300 INFO:journalctl@ceph.osd.2.smithi136.stdout:Apr 26 23:37:45 smithi136 sudo[41810]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T23:37:46.300 INFO:journalctl@ceph.osd.2.smithi136.stdout:Apr 26 23:37:45 smithi136 sudo[41810]: pam_unix(sudo:session): session closed for user root 2024-04-26T23:37:46.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:45 smithi136 sudo[41813]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T23:37:46.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:45 smithi136 sudo[41813]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T23:37:46.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:45 smithi136 sudo[41813]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T23:37:46.301 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:45 smithi136 sudo[41813]: pam_unix(sudo:session): session closed for user root 2024-04-26T23:37:46.301 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:46 smithi136 sudo[41816]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T23:37:46.301 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:46 smithi136 sudo[41816]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T23:37:46.301 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:46 smithi136 sudo[41816]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T23:37:46.301 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:46 smithi136 sudo[41816]: pam_unix(sudo:session): session closed for user root 2024-04-26T23:37:46.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:46 smithi076 sudo[41056]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T23:37:46.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:46 smithi076 sudo[41056]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T23:37:46.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:46 smithi076 sudo[41056]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T23:37:46.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:46 smithi076 sudo[41056]: pam_unix(sudo:session): session closed for user root 2024-04-26T23:37:46.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:46 smithi076 sudo[41059]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T23:37:46.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:46 smithi076 sudo[41059]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T23:37:46.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:46 smithi076 sudo[41059]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T23:37:46.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:46 smithi076 sudo[41059]: pam_unix(sudo:session): session closed for user root 2024-04-26T23:37:46.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:46 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:46.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:46 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:46.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:46 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:46.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:46 smithi072 ceph-mon[28184]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T23:37:46.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:46 smithi072 ceph-mon[28184]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T23:37:46.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:46 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T23:37:46.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:46 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:37:46.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:46 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:37:46.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:46 smithi072 ceph-mon[28184]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T23:37:46.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:46 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T23:37:46.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:46 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:37:46.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:46 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:37:46.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:46 smithi072 ceph-mon[28184]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T23:37:46.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:46 smithi072 ceph-mon[28184]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T23:37:46.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:46 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T23:37:46.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:46 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:37:46.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:46 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:37:46.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:46 smithi072 ceph-mon[28184]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T23:37:46.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:46 smithi072 ceph-mon[28184]: pgmap v94: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:37:46.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:46 smithi072 ceph-mon[28184]: osdmap e21: 3 total, 3 up, 3 in 2024-04-26T23:37:47.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:46 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:47.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:46 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:47.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:46 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:47.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:46 smithi136 ceph-mon[33050]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T23:37:47.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:46 smithi136 ceph-mon[33050]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T23:37:47.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:46 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T23:37:47.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:46 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:37:47.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:46 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:37:47.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:46 smithi136 ceph-mon[33050]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T23:37:47.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:46 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T23:37:47.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:46 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:37:47.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:46 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:37:47.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:46 smithi136 ceph-mon[33050]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T23:37:47.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:46 smithi136 ceph-mon[33050]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T23:37:47.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:46 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T23:37:47.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:46 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:37:47.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:46 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:37:47.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:46 smithi136 ceph-mon[33050]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T23:37:47.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:46 smithi136 ceph-mon[33050]: pgmap v94: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:37:47.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:46 smithi136 ceph-mon[33050]: osdmap e21: 3 total, 3 up, 3 in 2024-04-26T23:37:47.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:46 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:47.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:46 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:47.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:46 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:37:47.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:46 smithi076 ceph-mon[32774]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T23:37:47.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:46 smithi076 ceph-mon[32774]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T23:37:47.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:46 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T23:37:47.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:46 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:37:47.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:46 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:37:47.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:46 smithi076 ceph-mon[32774]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T23:37:47.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:46 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T23:37:47.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:46 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:37:47.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:46 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:37:47.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:46 smithi076 ceph-mon[32774]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T23:37:47.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:46 smithi076 ceph-mon[32774]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T23:37:47.180 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:46 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T23:37:47.180 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:46 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T23:37:47.180 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:46 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T23:37:47.180 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:46 smithi076 ceph-mon[32774]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T23:37:47.180 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:46 smithi076 ceph-mon[32774]: pgmap v94: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:37:47.180 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:46 smithi076 ceph-mon[32774]: osdmap e21: 3 total, 3 up, 3 in 2024-04-26T23:37:48.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:37:48.084+0000 7f5fd5000700 1 -- 172.21.15.72:0/3367208365 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5fd0106840 msgr2=0x7f5fd0108c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:37:48.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:37:48.084+0000 7f5fd5000700 1 --2- 172.21.15.72:0/3367208365 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5fd0106840 0x7f5fd0108c30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5fb8009990 tx=0x7f5fb8009ca0 comp rx=0 tx=0).stop 2024-04-26T23:37:48.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:37:48.084+0000 7f5fd5000700 1 -- 172.21.15.72:0/3367208365 shutdown_connections 2024-04-26T23:37:48.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:37:48.084+0000 7f5fd5000700 1 --2- 172.21.15.72:0/3367208365 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fd0109170 0x7f5fd010b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:37:48.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:37:48.084+0000 7f5fd5000700 1 --2- 172.21.15.72:0/3367208365 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fd0103f10 0x7f5fd0106300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:37:48.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:37:48.084+0000 7f5fd5000700 1 --2- 172.21.15.72:0/3367208365 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5fd0106840 0x7f5fd0108c30 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:37:48.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:37:48.084+0000 7f5fd5000700 1 -- 172.21.15.72:0/3367208365 >> 172.21.15.72:0/3367208365 conn(0x7f5fd00fd940 msgr2=0x7f5fd00ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:37:48.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:37:48.084+0000 7f5fd5000700 1 -- 172.21.15.72:0/3367208365 shutdown_connections 2024-04-26T23:37:48.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:37:48.084+0000 7f5fd5000700 1 -- 172.21.15.72:0/3367208365 wait complete. 2024-04-26T23:37:48.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:37:48.085+0000 7f5fd5000700 1 Processor -- start 2024-04-26T23:37:48.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:37:48.085+0000 7f5fd5000700 1 -- start start 2024-04-26T23:37:48.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:37:48.086+0000 7f5fd5000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5fd0103f10 0x7f5fd0103870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:37:48.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:37:48.086+0000 7f5fd5000700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fd0106840 0x7f5fd0101ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:37:48.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:37:48.086+0000 7f5fd5000700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fd0109170 0x7f5fd0102400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:37:48.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:37:48.086+0000 7f5fd5000700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f5fd010ef90 con 0x7f5fd0109170 2024-04-26T23:37:48.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:37:48.086+0000 7f5fd5000700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f5fd010ee10 con 0x7f5fd0103f10 2024-04-26T23:37:48.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:37:48.086+0000 7f5fd5000700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f5fd010f110 con 0x7f5fd0106840 2024-04-26T23:37:48.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:37:48.086+0000 7f5fcde00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5fd0103f10 0x7f5fd0103870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:37:48.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:37:48.086+0000 7f5fce800700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fd0109170 0x7f5fd0102400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:37:48.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:37:48.086+0000 7f5fcde00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5fd0103f10 0x7f5fd0103870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:41028/0 (socket says 172.21.15.72:41028) 2024-04-26T23:37:48.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:37:48.086+0000 7f5fcde00700 1 -- 172.21.15.72:0/48158193 learned_addr learned my addr 172.21.15.72:0/48158193 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:37:48.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:37:48.086+0000 7f5fcd400700 1 --2- 172.21.15.72:0/48158193 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fd0106840 0x7f5fd0101ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:37:48.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:37:48.086+0000 7f5fcde00700 1 -- 172.21.15.72:0/48158193 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fd0106840 msgr2=0x7f5fd0101ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:37:48.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:37:48.086+0000 7f5fcde00700 1 --2- 172.21.15.72:0/48158193 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fd0106840 0x7f5fd0101ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:37:48.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:37:48.086+0000 7f5fcde00700 1 -- 172.21.15.72:0/48158193 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fd0109170 msgr2=0x7f5fd0102400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:37:48.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:37:48.086+0000 7f5fcde00700 1 --2- 172.21.15.72:0/48158193 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fd0109170 0x7f5fd0102400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:37:48.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:37:48.086+0000 7f5fcde00700 1 -- 172.21.15.72:0/48158193 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fd0102940 con 0x7f5fd0103f10 2024-04-26T23:37:48.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:37:48.087+0000 7f5fcde00700 1 --2- 172.21.15.72:0/48158193 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5fd0103f10 0x7f5fd0103870 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5fc000d980 tx=0x7f5fc000dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:37:48.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:37:48.087+0000 7f5fc6a00700 1 -- 172.21.15.72:0/48158193 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc0017070 con 0x7f5fd0103f10 2024-04-26T23:37:48.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:37:48.087+0000 7f5fc6a00700 1 -- 172.21.15.72:0/48158193 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5fc0005020 con 0x7f5fd0103f10 2024-04-26T23:37:48.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:37:48.087+0000 7f5fc6a00700 1 -- 172.21.15.72:0/48158193 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc000fa10 con 0x7f5fd0103f10 2024-04-26T23:37:48.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:37:48.087+0000 7f5fd5000700 1 -- 172.21.15.72:0/48158193 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fd0102c30 con 0x7f5fd0103f10 2024-04-26T23:37:48.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:37:48.087+0000 7f5fd5000700 1 -- 172.21.15.72:0/48158193 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fd01a8080 con 0x7f5fd0103f10 2024-04-26T23:37:48.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:37:48.088+0000 7f5fd5000700 1 -- 172.21.15.72:0/48158193 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fd006a110 con 0x7f5fd0103f10 2024-04-26T23:37:48.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:37:48.090+0000 7f5fc6a00700 1 -- 172.21.15.72:0/48158193 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f5fc0004a30 con 0x7f5fd0103f10 2024-04-26T23:37:48.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:37:48.091+0000 7f5fc6a00700 1 --2- 172.21.15.72:0/48158193 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f5fb40a3c40 0x7f5fb40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:37:48.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:37:48.091+0000 7f5fc6a00700 1 -- 172.21.15.72:0/48158193 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3237+0+0 (secure 0 0 0) 0x7f5fc00cef50 con 0x7f5fd0103f10 2024-04-26T23:37:48.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:37:48.091+0000 7f5fcd400700 1 --2- 172.21.15.72:0/48158193 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f5fb40a3c40 0x7f5fb40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:37:48.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:37:48.092+0000 7f5fcd400700 1 --2- 172.21.15.72:0/48158193 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f5fb40a3c40 0x7f5fb40a6100 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5fb800b2e0 tx=0x7f5fb800b7c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:37:48.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:37:48.096+0000 7f5fc6a00700 1 -- 172.21.15.72:0/48158193 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5fc0020020 con 0x7f5fd0103f10 2024-04-26T23:37:48.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:47 smithi076 ceph-mon[32774]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-26T23:37:48.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:47 smithi072 ceph-mon[28184]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-26T23:37:48.299 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:47 smithi136 ceph-mon[33050]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-26T23:37:48.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:37:48.331+0000 7f5fd5000700 1 -- 172.21.15.72:0/48158193 --> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi072:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f5fd0063860 con 0x7f5fb40a3c40 2024-04-26T23:37:49.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:48 smithi076 ceph-mon[32774]: pgmap v96: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:37:49.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:48 smithi076 ceph-mon[32774]: from='client.24191 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi072:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:37:49.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:48 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:37:49.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:48 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:37:49.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:48 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:37:49.261 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:48 smithi072 ceph-mon[28184]: pgmap v96: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:37:49.262 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:48 smithi072 ceph-mon[28184]: from='client.24191 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi072:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:37:49.262 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:48 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:37:49.262 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:48 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:37:49.262 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:48 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:37:49.299 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:48 smithi136 ceph-mon[33050]: pgmap v96: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:37:49.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:48 smithi136 ceph-mon[33050]: from='client.24191 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi072:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:37:49.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:48 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:37:49.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:48 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:37:49.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:48 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:37:51.670 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:51 smithi072 ceph-mon[28184]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:37:51.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:51 smithi076 ceph-mon[32774]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:37:51.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:51 smithi136 ceph-mon[33050]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:37:52.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:52 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/101182116' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e6da89be-5ba7-48a4-9306-f75498660e0e"}]: dispatch 2024-04-26T23:37:52.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:52 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/101182116' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e6da89be-5ba7-48a4-9306-f75498660e0e"}]': finished 2024-04-26T23:37:52.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:52 smithi076 ceph-mon[32774]: osdmap e22: 4 total, 3 up, 4 in 2024-04-26T23:37:52.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:52 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T23:37:52.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:52 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/101182116' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e6da89be-5ba7-48a4-9306-f75498660e0e"}]: dispatch 2024-04-26T23:37:52.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:52 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/101182116' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e6da89be-5ba7-48a4-9306-f75498660e0e"}]': finished 2024-04-26T23:37:52.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:52 smithi072 ceph-mon[28184]: osdmap e22: 4 total, 3 up, 4 in 2024-04-26T23:37:52.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:52 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T23:37:52.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:52 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/101182116' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e6da89be-5ba7-48a4-9306-f75498660e0e"}]: dispatch 2024-04-26T23:37:52.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:52 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/101182116' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e6da89be-5ba7-48a4-9306-f75498660e0e"}]': finished 2024-04-26T23:37:52.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:52 smithi136 ceph-mon[33050]: osdmap e22: 4 total, 3 up, 4 in 2024-04-26T23:37:52.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:52 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T23:37:53.549 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:53 smithi136 ceph-mon[33050]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:37:53.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:53 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/539167136' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:37:53.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:53 smithi076 ceph-mon[32774]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:37:53.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:53 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/539167136' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:37:53.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:53 smithi072 ceph-mon[28184]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:37:53.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:53 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/539167136' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:37:55.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:55 smithi076 ceph-mon[32774]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:37:55.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:55 smithi072 ceph-mon[28184]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:37:55.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:55 smithi136 ceph-mon[33050]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:37:57.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:57 smithi076 ceph-mon[32774]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:37:57.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:57 smithi072 ceph-mon[28184]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:37:57.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:57 smithi136 ceph-mon[33050]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:37:59.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:37:59 smithi076 ceph-mon[32774]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:37:59.697 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:37:59 smithi072 ceph-mon[28184]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:37:59.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:37:59 smithi136 ceph-mon[33050]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:38:01.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:01 smithi076 ceph-mon[32774]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:38:01.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:01 smithi072 ceph-mon[28184]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:38:01.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:01 smithi136 ceph-mon[33050]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:38:02.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:02 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-26T23:38:02.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:02 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:38:02.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:02 smithi076 ceph-mon[32774]: Deploying daemon osd.3 on smithi072 2024-04-26T23:38:02.700 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:02 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-26T23:38:02.700 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:02 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:38:02.700 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:02 smithi072 ceph-mon[28184]: Deploying daemon osd.3 on smithi072 2024-04-26T23:38:02.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:02 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-26T23:38:02.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:02 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:38:02.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:02 smithi136 ceph-mon[33050]: Deploying daemon osd.3 on smithi072 2024-04-26T23:38:03.672 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:03 smithi072 ceph-mon[28184]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:38:03.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:03 smithi076 ceph-mon[32774]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:38:03.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:03 smithi136 ceph-mon[33050]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:38:05.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:05 smithi072 ceph-mon[28184]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:38:05.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:05 smithi076 ceph-mon[32774]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:38:05.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:05 smithi136 ceph-mon[33050]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:38:07.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:07 smithi076 ceph-mon[32774]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:38:07.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:07 smithi072 ceph-mon[28184]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:38:07.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:07 smithi136 ceph-mon[33050]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:38:09.589 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:09 smithi072 ceph-mon[28184]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:38:09.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:09 smithi076 ceph-mon[32774]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:38:09.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:09 smithi136 ceph-mon[33050]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:38:11.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:10 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:11.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:10 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:11.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:10 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:38:11.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:10 smithi072 ceph-mon[28184]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:38:11.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:10 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:11.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:10 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:11.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:10 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:38:11.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:10 smithi076 ceph-mon[32774]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:38:11.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:10 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:11.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:10 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:11.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:10 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:38:11.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:10 smithi136 ceph-mon[33050]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:38:13.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:13 smithi072 ceph-mon[28184]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:38:13.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:13 smithi076 ceph-mon[32774]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:38:13.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:13 smithi136 ceph-mon[33050]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:38:14.683 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:14 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:14.683 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:14 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:14.683 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:14 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:38:14.683 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:14 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:38:14.683 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:14 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:14.683 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:14 smithi072 ceph-mon[28184]: from='osd.3 [v2:172.21.15.72:6810/2071107815,v1:172.21.15.72:6811/2071107815]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T23:38:14.683 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:14 smithi072 ceph-mon[28184]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T23:38:14.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:38:14.784+0000 7f5fc6a00700 1 -- 172.21.15.72:0/48158193 <== mgr.14152 v2:172.21.15.72:6800/4281177709 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f5fd0063860 con 0x7f5fb40a3c40 2024-04-26T23:38:14.789 INFO:teuthology.orchestra.run.smithi072.stdout:Created osd(s) 3 on host 'smithi072' 2024-04-26T23:38:14.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:38:14.788+0000 7f5fd5000700 1 -- 172.21.15.72:0/48158193 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f5fb40a3c40 msgr2=0x7f5fb40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:38:14.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:38:14.788+0000 7f5fd5000700 1 --2- 172.21.15.72:0/48158193 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f5fb40a3c40 0x7f5fb40a6100 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5fb800b2e0 tx=0x7f5fb800b7c0 comp rx=0 tx=0).stop 2024-04-26T23:38:14.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:38:14.788+0000 7f5fd5000700 1 -- 172.21.15.72:0/48158193 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5fd0103f10 msgr2=0x7f5fd0103870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:38:14.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:38:14.788+0000 7f5fd5000700 1 --2- 172.21.15.72:0/48158193 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5fd0103f10 0x7f5fd0103870 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5fc000d980 tx=0x7f5fc000dc90 comp rx=0 tx=0).stop 2024-04-26T23:38:14.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:38:14.788+0000 7f5fd5000700 1 -- 172.21.15.72:0/48158193 shutdown_connections 2024-04-26T23:38:14.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:38:14.788+0000 7f5fd5000700 1 --2- 172.21.15.72:0/48158193 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f5fb40a3c40 0x7f5fb40a6100 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:38:14.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:38:14.788+0000 7f5fd5000700 1 --2- 172.21.15.72:0/48158193 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fd0109170 0x7f5fd0102400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:38:14.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:38:14.788+0000 7f5fd5000700 1 --2- 172.21.15.72:0/48158193 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5fd0103f10 0x7f5fd0103870 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:38:14.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:38:14.788+0000 7f5fd5000700 1 --2- 172.21.15.72:0/48158193 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fd0106840 0x7f5fd0101ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:38:14.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:38:14.788+0000 7f5fd5000700 1 -- 172.21.15.72:0/48158193 >> 172.21.15.72:0/48158193 conn(0x7f5fd00fd940 msgr2=0x7f5fd0104990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:38:14.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:38:14.788+0000 7f5fd5000700 1 -- 172.21.15.72:0/48158193 shutdown_connections 2024-04-26T23:38:14.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:38:14.788+0000 7f5fd5000700 1 -- 172.21.15.72:0/48158193 wait complete. 2024-04-26T23:38:14.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:14 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:14.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:14 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:14.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:14 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:38:14.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:14 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:38:14.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:14 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:14.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:14 smithi136 ceph-mon[33050]: from='osd.3 [v2:172.21.15.72:6810/2071107815,v1:172.21.15.72:6811/2071107815]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T23:38:14.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:14 smithi136 ceph-mon[33050]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T23:38:14.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:14 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:14.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:14 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:14.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:14 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:38:14.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:14 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:38:14.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:14 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:14.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:14 smithi076 ceph-mon[32774]: from='osd.3 [v2:172.21.15.72:6810/2071107815,v1:172.21.15.72:6811/2071107815]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T23:38:14.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:14 smithi076 ceph-mon[32774]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T23:38:15.635 DEBUG:teuthology.orchestra.run.smithi072:osd.3> sudo journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@osd.3.service 2024-04-26T23:38:15.640 INFO:tasks.cephadm:Deploying osd.4 on smithi076 with /dev/vg_nvme/lv_3... 2024-04-26T23:38:15.640 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-26T23:38:15.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:15 smithi072 ceph-mon[28184]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:38:15.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:15 smithi072 ceph-mon[28184]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-26T23:38:15.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:15 smithi072 ceph-mon[28184]: from='osd.3 [v2:172.21.15.72:6810/2071107815,v1:172.21.15.72:6811/2071107815]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]: dispatch 2024-04-26T23:38:15.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:15 smithi072 ceph-mon[28184]: osdmap e23: 4 total, 3 up, 4 in 2024-04-26T23:38:15.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:15 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T23:38:15.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:15 smithi072 ceph-mon[28184]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]: dispatch 2024-04-26T23:38:15.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:15 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:15.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:15 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:15.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:15 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:38:15.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:15 smithi136 ceph-mon[33050]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:38:15.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:15 smithi136 ceph-mon[33050]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-26T23:38:15.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:15 smithi136 ceph-mon[33050]: from='osd.3 [v2:172.21.15.72:6810/2071107815,v1:172.21.15.72:6811/2071107815]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]: dispatch 2024-04-26T23:38:15.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:15 smithi136 ceph-mon[33050]: osdmap e23: 4 total, 3 up, 4 in 2024-04-26T23:38:15.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:15 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T23:38:15.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:15 smithi136 ceph-mon[33050]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]: dispatch 2024-04-26T23:38:15.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:15 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:15.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:15 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:15.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:15 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:38:15.888 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T23:38:15.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:15 smithi076 ceph-mon[32774]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:38:15.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:15 smithi076 ceph-mon[32774]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-26T23:38:15.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:15 smithi076 ceph-mon[32774]: from='osd.3 [v2:172.21.15.72:6810/2071107815,v1:172.21.15.72:6811/2071107815]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]: dispatch 2024-04-26T23:38:15.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:15 smithi076 ceph-mon[32774]: osdmap e23: 4 total, 3 up, 4 in 2024-04-26T23:38:15.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:15 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T23:38:15.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:15 smithi076 ceph-mon[32774]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]: dispatch 2024-04-26T23:38:15.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:15 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:15.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:15 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:15.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:15 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:38:16.514 INFO:journalctl@ceph.osd.3.smithi072.stdout:Apr 26 23:38:16 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-3[47161]: 2024-04-26T23:38:16.165+0000 7fe515600700 -1 osd.3 0 waiting for initial osdmap 2024-04-26T23:38:16.514 INFO:journalctl@ceph.osd.3.smithi072.stdout:Apr 26 23:38:16 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-3[47161]: 2024-04-26T23:38:16.184+0000 7fe50b400700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T23:38:16.515 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:16 smithi072 ceph-mon[28184]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]': finished 2024-04-26T23:38:16.515 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:16 smithi072 ceph-mon[28184]: osdmap e24: 4 total, 3 up, 4 in 2024-04-26T23:38:16.515 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:16 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T23:38:16.516 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:16 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T23:38:16.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:16 smithi076 ceph-mon[32774]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]': finished 2024-04-26T23:38:16.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:16 smithi076 ceph-mon[32774]: osdmap e24: 4 total, 3 up, 4 in 2024-04-26T23:38:16.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:16 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T23:38:16.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:16 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T23:38:16.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:16 smithi076 ceph-mon[32774]: from='osd.3 ' entity='osd.3' 2024-04-26T23:38:16.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:16 smithi072 ceph-mon[28184]: from='osd.3 ' entity='osd.3' 2024-04-26T23:38:16.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:16 smithi136 ceph-mon[33050]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]': finished 2024-04-26T23:38:16.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:16 smithi136 ceph-mon[33050]: osdmap e24: 4 total, 3 up, 4 in 2024-04-26T23:38:16.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:16 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T23:38:16.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:16 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T23:38:16.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:16 smithi136 ceph-mon[33050]: from='osd.3 ' entity='osd.3' 2024-04-26T23:38:17.526 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:17 smithi076 ceph-mon[32774]: purged_snaps scrub starts 2024-04-26T23:38:17.526 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:17 smithi076 ceph-mon[32774]: purged_snaps scrub ok 2024-04-26T23:38:17.526 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:17 smithi076 ceph-mon[32774]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:38:17.526 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:17 smithi076 ceph-mon[32774]: osd.3 [v2:172.21.15.72:6810/2071107815,v1:172.21.15.72:6811/2071107815] boot 2024-04-26T23:38:17.526 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:17 smithi076 ceph-mon[32774]: osdmap e25: 4 total, 4 up, 4 in 2024-04-26T23:38:17.526 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:17 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T23:38:17.618 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:17 smithi072 ceph-mon[28184]: purged_snaps scrub starts 2024-04-26T23:38:17.618 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:17 smithi072 ceph-mon[28184]: purged_snaps scrub ok 2024-04-26T23:38:17.618 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:17 smithi072 ceph-mon[28184]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:38:17.618 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:17 smithi072 ceph-mon[28184]: osd.3 [v2:172.21.15.72:6810/2071107815,v1:172.21.15.72:6811/2071107815] boot 2024-04-26T23:38:17.618 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:17 smithi072 ceph-mon[28184]: osdmap e25: 4 total, 4 up, 4 in 2024-04-26T23:38:17.618 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:17 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T23:38:17.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:17 smithi136 ceph-mon[33050]: purged_snaps scrub starts 2024-04-26T23:38:17.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:17 smithi136 ceph-mon[33050]: purged_snaps scrub ok 2024-04-26T23:38:17.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:17 smithi136 ceph-mon[33050]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-04-26T23:38:17.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:17 smithi136 ceph-mon[33050]: osd.3 [v2:172.21.15.72:6810/2071107815,v1:172.21.15.72:6811/2071107815] boot 2024-04-26T23:38:17.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:17 smithi136 ceph-mon[33050]: osdmap e25: 4 total, 4 up, 4 in 2024-04-26T23:38:17.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:17 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T23:38:17.978 INFO:teuthology.orchestra.run.smithi076.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-26T23:38:17.978 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-26T23:38:17.978 INFO:teuthology.orchestra.run.smithi076.stderr: stderr: 10+0 records in 2024-04-26T23:38:17.978 INFO:teuthology.orchestra.run.smithi076.stderr:10+0 records out 2024-04-26T23:38:17.978 INFO:teuthology.orchestra.run.smithi076.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.011479 s, 913 MB/s 2024-04-26T23:38:17.978 INFO:teuthology.orchestra.run.smithi076.stderr:--> Zapping successful for: 2024-04-26T23:38:18.596 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:18 smithi076 ceph-mon[32774]: osdmap e26: 4 total, 4 up, 4 in 2024-04-26T23:38:18.627 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi076:vg_nvme/lv_3 2024-04-26T23:38:18.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:18 smithi072 ceph-mon[28184]: osdmap e26: 4 total, 4 up, 4 in 2024-04-26T23:38:18.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:18 smithi136 ceph-mon[33050]: osdmap e26: 4 total, 4 up, 4 in 2024-04-26T23:38:18.876 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T23:38:19.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:19 smithi072 ceph-mon[28184]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 357 GiB / 358 GiB avail 2024-04-26T23:38:19.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:19 smithi072 ceph-mon[28184]: osdmap e27: 4 total, 4 up, 4 in 2024-04-26T23:38:19.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:19 smithi136 ceph-mon[33050]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 357 GiB / 358 GiB avail 2024-04-26T23:38:19.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:19 smithi136 ceph-mon[33050]: osdmap e27: 4 total, 4 up, 4 in 2024-04-26T23:38:19.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:19 smithi076 ceph-mon[32774]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 147 MiB used, 357 GiB / 358 GiB avail 2024-04-26T23:38:19.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:19 smithi076 ceph-mon[32774]: osdmap e27: 4 total, 4 up, 4 in 2024-04-26T23:38:20.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:20.536+0000 7f87b04d9700 1 -- 172.21.15.76:0/1380547956 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f87a81068a0 msgr2=0x7f87a8108c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:38:20.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:20.536+0000 7f87b04d9700 1 --2- 172.21.15.76:0/1380547956 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f87a81068a0 0x7f87a8108c90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8798009990 tx=0x7f8798009ca0 comp rx=0 tx=0).stop 2024-04-26T23:38:20.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:20.536+0000 7f87b04d9700 1 -- 172.21.15.76:0/1380547956 shutdown_connections 2024-04-26T23:38:20.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:20.536+0000 7f87b04d9700 1 --2- 172.21.15.76:0/1380547956 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87a81091d0 0x7f87a810b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:38:20.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:20.536+0000 7f87b04d9700 1 --2- 172.21.15.76:0/1380547956 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f87a8103f70 0x7f87a8106360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:38:20.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:20.536+0000 7f87b04d9700 1 --2- 172.21.15.76:0/1380547956 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f87a81068a0 0x7f87a8108c90 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:38:20.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:20.536+0000 7f87b04d9700 1 -- 172.21.15.76:0/1380547956 >> 172.21.15.76:0/1380547956 conn(0x7f87a80fd920 msgr2=0x7f87a80ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:38:20.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:20.536+0000 7f87b04d9700 1 -- 172.21.15.76:0/1380547956 shutdown_connections 2024-04-26T23:38:20.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:20.536+0000 7f87b04d9700 1 -- 172.21.15.76:0/1380547956 wait complete. 2024-04-26T23:38:20.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:20.536+0000 7f87b04d9700 1 Processor -- start 2024-04-26T23:38:20.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:20.536+0000 7f87b04d9700 1 -- start start 2024-04-26T23:38:20.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:20.537+0000 7f87b04d9700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f87a8103f70 0x7f87a81996a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:38:20.541 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:20.537+0000 7f87b04d9700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f87a81068a0 0x7f87a8199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:38:20.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:20.537+0000 7f87b04d9700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87a81091d0 0x7f87a81a0c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:38:20.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:20.537+0000 7f87b04d9700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f87a810efd0 con 0x7f87a81091d0 2024-04-26T23:38:20.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:20.537+0000 7f87b04d9700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f87a810ee50 con 0x7f87a81068a0 2024-04-26T23:38:20.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:20.537+0000 7f87b04d9700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f87a810f150 con 0x7f87a8103f70 2024-04-26T23:38:20.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:20.537+0000 7f87ada74700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f87a81068a0 0x7f87a8199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:38:20.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:20.537+0000 7f87ada74700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f87a81068a0 0x7f87a8199be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:55202/0 (socket says 172.21.15.76:55202) 2024-04-26T23:38:20.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:20.537+0000 7f87ada74700 1 -- 172.21.15.76:0/689445221 learned_addr learned my addr 172.21.15.76:0/689445221 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-26T23:38:20.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:20.537+0000 7f87ae275700 1 --2- 172.21.15.76:0/689445221 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f87a8103f70 0x7f87a81996a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:38:20.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:20.537+0000 7f87ada74700 1 -- 172.21.15.76:0/689445221 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f87a8103f70 msgr2=0x7f87a81996a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:38:20.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:20.537+0000 7f87ada74700 1 --2- 172.21.15.76:0/689445221 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f87a8103f70 0x7f87a81996a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:38:20.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:20.537+0000 7f87ada74700 1 -- 172.21.15.76:0/689445221 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87a81091d0 msgr2=0x7f87a81a0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:38:20.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:20.537+0000 7f87ada74700 1 --2- 172.21.15.76:0/689445221 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87a81091d0 0x7f87a81a0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:38:20.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:20.537+0000 7f87ada74700 1 -- 172.21.15.76:0/689445221 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87a81a1300 con 0x7f87a81068a0 2024-04-26T23:38:20.542 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:20.537+0000 7f87ada74700 1 --2- 172.21.15.76:0/689445221 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f87a81068a0 0x7f87a8199be0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f87980038a0 tx=0x7f87980047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:38:20.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:20.537+0000 7f879f7fe700 1 -- 172.21.15.76:0/689445221 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8798027070 con 0x7f87a81068a0 2024-04-26T23:38:20.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:20.537+0000 7f87b04d9700 1 -- 172.21.15.76:0/689445221 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87a81a1590 con 0x7f87a81068a0 2024-04-26T23:38:20.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:20.537+0000 7f879f7fe700 1 -- 172.21.15.76:0/689445221 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8798004080 con 0x7f87a81068a0 2024-04-26T23:38:20.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:20.537+0000 7f879f7fe700 1 -- 172.21.15.76:0/689445221 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f879800f460 con 0x7f87a81068a0 2024-04-26T23:38:20.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:20.537+0000 7f87b04d9700 1 -- 172.21.15.76:0/689445221 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87a81a1a60 con 0x7f87a81068a0 2024-04-26T23:38:20.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:20.538+0000 7f87b04d9700 1 -- 172.21.15.76:0/689445221 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87a802e490 con 0x7f87a81068a0 2024-04-26T23:38:20.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:20.540+0000 7f879f7fe700 1 -- 172.21.15.76:0/689445221 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8798004230 con 0x7f87a81068a0 2024-04-26T23:38:20.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:20.540+0000 7f879f7fe700 1 --2- 172.21.15.76:0/689445221 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f87940a3c40 0x7f87940a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:38:20.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:20.540+0000 7f879f7fe700 1 -- 172.21.15.76:0/689445221 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 3669+0+0 (secure 0 0 0) 0x7f87980d00a0 con 0x7f87a81068a0 2024-04-26T23:38:20.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:20.541+0000 7f87ae275700 1 --2- 172.21.15.76:0/689445221 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f87940a3c40 0x7f87940a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:38:20.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:20.542+0000 7f87ae275700 1 --2- 172.21.15.76:0/689445221 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f87940a3c40 0x7f87940a6100 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f87a4009d10 tx=0x7f87a4009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:38:20.548 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:20.545+0000 7f879f7fe700 1 -- 172.21.15.76:0/689445221 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8798013070 con 0x7f87a81068a0 2024-04-26T23:38:20.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:20.752+0000 7f87b04d9700 1 -- 172.21.15.76:0/689445221 --> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f87a8063860 con 0x7f87940a3c40 2024-04-26T23:38:20.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:20 smithi136 ceph-mon[33050]: osdmap e28: 4 total, 4 up, 4 in 2024-04-26T23:38:20.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:20 smithi076 ceph-mon[32774]: osdmap e28: 4 total, 4 up, 4 in 2024-04-26T23:38:21.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:20 smithi072 ceph-mon[28184]: osdmap e28: 4 total, 4 up, 4 in 2024-04-26T23:38:21.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:21 smithi076 ceph-mon[32774]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 147 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T23:38:21.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:21 smithi076 ceph-mon[32774]: osdmap e29: 4 total, 4 up, 4 in 2024-04-26T23:38:21.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:21 smithi076 ceph-mon[32774]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:38:21.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:21 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:38:21.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:21 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:38:21.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:21 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:38:21.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:21 smithi072 ceph-mon[28184]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 147 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T23:38:21.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:21 smithi072 ceph-mon[28184]: osdmap e29: 4 total, 4 up, 4 in 2024-04-26T23:38:21.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:21 smithi072 ceph-mon[28184]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:38:21.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:21 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:38:21.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:21 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:38:21.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:21 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:38:21.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:21 smithi136 ceph-mon[33050]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 147 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T23:38:21.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:21 smithi136 ceph-mon[33050]: osdmap e29: 4 total, 4 up, 4 in 2024-04-26T23:38:21.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:21 smithi136 ceph-mon[33050]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:38:21.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:21 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:38:21.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:21 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:38:21.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:21 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:38:23.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:23 smithi076 ceph-mon[32774]: Detected new or changed devices on smithi072 2024-04-26T23:38:23.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:23 smithi076 ceph-mon[32774]: pgmap v121: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-04-26T23:38:23.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:23 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:23.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:23 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:23.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:23 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:38:23.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:23 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:38:23.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:23 smithi076 ceph-mon[32774]: Adjusting osd_memory_target on smithi072 to 8544M 2024-04-26T23:38:23.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:23 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:23.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:23 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:38:23.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:23 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:38:23.680 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:23 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:23.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:23 smithi072 ceph-mon[28184]: Detected new or changed devices on smithi072 2024-04-26T23:38:23.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:23 smithi072 ceph-mon[28184]: pgmap v121: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-04-26T23:38:23.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:23 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:23.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:23 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:23.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:23 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:38:23.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:23 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:38:23.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:23 smithi072 ceph-mon[28184]: Adjusting osd_memory_target on smithi072 to 8544M 2024-04-26T23:38:23.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:23 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:23.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:23 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:38:23.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:23 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:38:23.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:23 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:23.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:23 smithi136 ceph-mon[33050]: Detected new or changed devices on smithi072 2024-04-26T23:38:23.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:23 smithi136 ceph-mon[33050]: pgmap v121: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-04-26T23:38:23.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:23 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:23.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:23 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:23.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:23 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:38:23.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:23 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:38:23.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:23 smithi136 ceph-mon[33050]: Adjusting osd_memory_target on smithi072 to 8544M 2024-04-26T23:38:23.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:23 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:23.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:23 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:38:23.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:23 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:38:23.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:23 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:24.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:24 smithi076 ceph-mon[32774]: from='client.? 172.21.15.76:0/3106053871' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b17c491-d798-488f-892d-521ef2fbc51e"}]: dispatch 2024-04-26T23:38:24.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:24 smithi076 ceph-mon[32774]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b17c491-d798-488f-892d-521ef2fbc51e"}]: dispatch 2024-04-26T23:38:24.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:24 smithi076 ceph-mon[32774]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0b17c491-d798-488f-892d-521ef2fbc51e"}]': finished 2024-04-26T23:38:24.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:24 smithi076 ceph-mon[32774]: osdmap e30: 5 total, 4 up, 5 in 2024-04-26T23:38:24.680 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:24 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T23:38:24.680 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:24 smithi076 ceph-mon[32774]: from='client.? 172.21.15.76:0/2597891399' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:38:24.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:24 smithi072 ceph-mon[28184]: from='client.? 172.21.15.76:0/3106053871' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b17c491-d798-488f-892d-521ef2fbc51e"}]: dispatch 2024-04-26T23:38:24.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:24 smithi072 ceph-mon[28184]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b17c491-d798-488f-892d-521ef2fbc51e"}]: dispatch 2024-04-26T23:38:24.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:24 smithi072 ceph-mon[28184]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0b17c491-d798-488f-892d-521ef2fbc51e"}]': finished 2024-04-26T23:38:24.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:24 smithi072 ceph-mon[28184]: osdmap e30: 5 total, 4 up, 5 in 2024-04-26T23:38:24.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:24 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T23:38:24.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:24 smithi072 ceph-mon[28184]: from='client.? 172.21.15.76:0/2597891399' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:38:24.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:24 smithi136 ceph-mon[33050]: from='client.? 172.21.15.76:0/3106053871' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b17c491-d798-488f-892d-521ef2fbc51e"}]: dispatch 2024-04-26T23:38:24.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:24 smithi136 ceph-mon[33050]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b17c491-d798-488f-892d-521ef2fbc51e"}]: dispatch 2024-04-26T23:38:24.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:24 smithi136 ceph-mon[33050]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0b17c491-d798-488f-892d-521ef2fbc51e"}]': finished 2024-04-26T23:38:24.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:24 smithi136 ceph-mon[33050]: osdmap e30: 5 total, 4 up, 5 in 2024-04-26T23:38:24.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:24 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T23:38:24.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:24 smithi136 ceph-mon[33050]: from='client.? 172.21.15.76:0/2597891399' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:38:25.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:25 smithi076 ceph-mon[32774]: pgmap v123: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-04-26T23:38:25.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:25 smithi072 ceph-mon[28184]: pgmap v123: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-04-26T23:38:25.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:25 smithi136 ceph-mon[33050]: pgmap v123: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-04-26T23:38:27.572 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:27 smithi076 ceph-mon[32774]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-04-26T23:38:27.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:27 smithi072 ceph-mon[28184]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-04-26T23:38:27.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:27 smithi136 ceph-mon[33050]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-04-26T23:38:29.645 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:29 smithi076 ceph-mon[32774]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T23:38:29.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:29 smithi072 ceph-mon[28184]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T23:38:29.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:29 smithi136 ceph-mon[33050]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T23:38:31.664 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:31 smithi076 ceph-mon[32774]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T23:38:31.664 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:31 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-26T23:38:31.664 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:31 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:38:31.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:31 smithi072 ceph-mon[28184]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T23:38:31.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:31 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-26T23:38:31.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:31 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:38:31.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:31 smithi136 ceph-mon[33050]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T23:38:31.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:31 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-26T23:38:31.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:31 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:38:32.639 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:32 smithi076 ceph-mon[32774]: Deploying daemon osd.4 on smithi076 2024-04-26T23:38:32.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:32 smithi072 ceph-mon[28184]: Deploying daemon osd.4 on smithi076 2024-04-26T23:38:32.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:32 smithi136 ceph-mon[33050]: Deploying daemon osd.4 on smithi076 2024-04-26T23:38:33.625 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:33 smithi076 ceph-mon[32774]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T23:38:33.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:33 smithi072 ceph-mon[28184]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T23:38:33.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:33 smithi136 ceph-mon[33050]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T23:38:35.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:35 smithi076 ceph-mon[32774]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T23:38:35.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:35 smithi072 ceph-mon[28184]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T23:38:35.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:35 smithi136 ceph-mon[33050]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T23:38:37.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:37 smithi076 ceph-mon[32774]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T23:38:37.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:37 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:37.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:37 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:37.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:37 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:38:37.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:37 smithi072 ceph-mon[28184]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T23:38:37.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:37 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:37.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:37 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:37.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:37 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:38:37.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:37 smithi136 ceph-mon[33050]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T23:38:37.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:37 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:37.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:37 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:37.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:37 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:38:39.617 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:39 smithi076 ceph-mon[32774]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T23:38:39.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:39 smithi072 ceph-mon[28184]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T23:38:39.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:39 smithi136 ceph-mon[33050]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T23:38:40.384 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:40.380+0000 7f879f7fe700 1 -- 172.21.15.76:0/689445221 <== mgr.14152 v2:172.21.15.72:6800/4281177709 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f87a8063860 con 0x7f87940a3c40 2024-04-26T23:38:40.384 INFO:teuthology.orchestra.run.smithi076.stdout:Created osd(s) 4 on host 'smithi076' 2024-04-26T23:38:40.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:40.383+0000 7f87b04d9700 1 -- 172.21.15.76:0/689445221 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f87940a3c40 msgr2=0x7f87940a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:38:40.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:40.383+0000 7f87b04d9700 1 --2- 172.21.15.76:0/689445221 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f87940a3c40 0x7f87940a6100 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f87a4009d10 tx=0x7f87a4009040 comp rx=0 tx=0).stop 2024-04-26T23:38:40.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:40.383+0000 7f87b04d9700 1 -- 172.21.15.76:0/689445221 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f87a81068a0 msgr2=0x7f87a8199be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:38:40.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:40.383+0000 7f87b04d9700 1 --2- 172.21.15.76:0/689445221 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f87a81068a0 0x7f87a8199be0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f87980038a0 tx=0x7f87980047b0 comp rx=0 tx=0).stop 2024-04-26T23:38:40.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:40.383+0000 7f87b04d9700 1 -- 172.21.15.76:0/689445221 shutdown_connections 2024-04-26T23:38:40.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:40.383+0000 7f87b04d9700 1 --2- 172.21.15.76:0/689445221 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f87940a3c40 0x7f87940a6100 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:38:40.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:40.383+0000 7f87b04d9700 1 --2- 172.21.15.76:0/689445221 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87a81091d0 0x7f87a81a0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:38:40.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:40.383+0000 7f87b04d9700 1 --2- 172.21.15.76:0/689445221 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f87a8103f70 0x7f87a81996a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:38:40.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:40.383+0000 7f87b04d9700 1 --2- 172.21.15.76:0/689445221 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f87a81068a0 0x7f87a8199be0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:38:40.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:40.383+0000 7f87b04d9700 1 -- 172.21.15.76:0/689445221 >> 172.21.15.76:0/689445221 conn(0x7f87a80fd920 msgr2=0x7f87a8104830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:38:40.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:40.383+0000 7f87b04d9700 1 -- 172.21.15.76:0/689445221 shutdown_connections 2024-04-26T23:38:40.387 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:38:40.383+0000 7f87b04d9700 1 -- 172.21.15.76:0/689445221 wait complete. 2024-04-26T23:38:40.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:40 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:40.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:40 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:40.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:40 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:38:40.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:40 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:38:40.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:40 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:40.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:40 smithi076 ceph-mon[32774]: from='osd.4 [v2:172.21.15.76:6808/147449238,v1:172.21.15.76:6809/147449238]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T23:38:40.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:40 smithi076 ceph-mon[32774]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T23:38:40.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:40 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:40.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:40 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:40.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:40 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:38:40.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:40 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:40.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:40 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:40.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:40 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:38:40.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:40 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:38:40.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:40 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:40.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:40 smithi072 ceph-mon[28184]: from='osd.4 [v2:172.21.15.76:6808/147449238,v1:172.21.15.76:6809/147449238]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T23:38:40.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:40 smithi072 ceph-mon[28184]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T23:38:40.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:40 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:40.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:40 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:40.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:40 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:38:40.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:40 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:40.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:40 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:40.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:40 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:38:40.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:40 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:38:40.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:40 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:40.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:40 smithi136 ceph-mon[33050]: from='osd.4 [v2:172.21.15.76:6808/147449238,v1:172.21.15.76:6809/147449238]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T23:38:40.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:40 smithi136 ceph-mon[33050]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T23:38:40.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:40 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:40.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:40 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:40.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:40 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:38:41.117 DEBUG:teuthology.orchestra.run.smithi076:osd.4> sudo journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@osd.4.service 2024-04-26T23:38:41.120 INFO:tasks.cephadm:Deploying osd.5 on smithi136 with /dev/vg_nvme/lv_3... 2024-04-26T23:38:41.120 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-26T23:38:41.364 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T23:38:41.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:41 smithi076 ceph-mon[32774]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T23:38:41.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:41 smithi076 ceph-mon[32774]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-26T23:38:41.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:41 smithi076 ceph-mon[32774]: from='osd.4 [v2:172.21.15.76:6808/147449238,v1:172.21.15.76:6809/147449238]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-04-26T23:38:41.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:41 smithi076 ceph-mon[32774]: osdmap e31: 5 total, 4 up, 5 in 2024-04-26T23:38:41.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:41 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T23:38:41.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:41 smithi076 ceph-mon[32774]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-04-26T23:38:41.680 INFO:journalctl@ceph.osd.4.smithi076.stdout:Apr 26 23:38:41 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-4[43878]: 2024-04-26T23:38:41.514+0000 7f087c716700 -1 osd.4 0 waiting for initial osdmap 2024-04-26T23:38:41.680 INFO:journalctl@ceph.osd.4.smithi076.stdout:Apr 26 23:38:41 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-4[43878]: 2024-04-26T23:38:41.524+0000 7f0878509700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T23:38:41.764 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:41 smithi136 ceph-mon[33050]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T23:38:41.764 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:41 smithi136 ceph-mon[33050]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-26T23:38:41.764 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:41 smithi136 ceph-mon[33050]: from='osd.4 [v2:172.21.15.76:6808/147449238,v1:172.21.15.76:6809/147449238]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-04-26T23:38:41.764 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:41 smithi136 ceph-mon[33050]: osdmap e31: 5 total, 4 up, 5 in 2024-04-26T23:38:41.764 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:41 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T23:38:41.765 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:41 smithi136 ceph-mon[33050]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-04-26T23:38:41.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:41 smithi072 ceph-mon[28184]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T23:38:41.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:41 smithi072 ceph-mon[28184]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-26T23:38:41.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:41 smithi072 ceph-mon[28184]: from='osd.4 [v2:172.21.15.76:6808/147449238,v1:172.21.15.76:6809/147449238]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-04-26T23:38:41.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:41 smithi072 ceph-mon[28184]: osdmap e31: 5 total, 4 up, 5 in 2024-04-26T23:38:41.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:41 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T23:38:41.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:41 smithi072 ceph-mon[28184]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-04-26T23:38:42.665 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:42 smithi076 ceph-mon[32774]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-04-26T23:38:42.665 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:42 smithi076 ceph-mon[32774]: osdmap e32: 5 total, 4 up, 5 in 2024-04-26T23:38:42.666 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:42 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T23:38:42.666 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:42 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T23:38:42.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:42 smithi072 ceph-mon[28184]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-04-26T23:38:42.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:42 smithi072 ceph-mon[28184]: osdmap e32: 5 total, 4 up, 5 in 2024-04-26T23:38:42.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:42 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T23:38:42.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:42 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T23:38:42.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:42 smithi136 ceph-mon[33050]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-04-26T23:38:42.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:42 smithi136 ceph-mon[33050]: osdmap e32: 5 total, 4 up, 5 in 2024-04-26T23:38:42.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:42 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T23:38:42.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:42 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T23:38:43.494 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-26T23:38:43.494 INFO:teuthology.orchestra.run.smithi136.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-26T23:38:43.494 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10+0 records in 2024-04-26T23:38:43.494 INFO:teuthology.orchestra.run.smithi136.stderr:10+0 records out 2024-04-26T23:38:43.495 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0129441 s, 810 MB/s 2024-04-26T23:38:43.495 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping successful for: 2024-04-26T23:38:43.788 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:43 smithi076 ceph-mon[32774]: purged_snaps scrub starts 2024-04-26T23:38:43.788 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:43 smithi076 ceph-mon[32774]: purged_snaps scrub ok 2024-04-26T23:38:43.788 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:43 smithi076 ceph-mon[32774]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T23:38:43.788 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:43 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T23:38:43.788 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:43 smithi076 ceph-mon[32774]: osd.4 [v2:172.21.15.76:6808/147449238,v1:172.21.15.76:6809/147449238] boot 2024-04-26T23:38:43.788 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:43 smithi076 ceph-mon[32774]: osdmap e33: 5 total, 5 up, 5 in 2024-04-26T23:38:43.788 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:43 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T23:38:43.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:43 smithi072 ceph-mon[28184]: purged_snaps scrub starts 2024-04-26T23:38:43.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:43 smithi072 ceph-mon[28184]: purged_snaps scrub ok 2024-04-26T23:38:43.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:43 smithi072 ceph-mon[28184]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T23:38:43.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:43 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T23:38:43.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:43 smithi072 ceph-mon[28184]: osd.4 [v2:172.21.15.76:6808/147449238,v1:172.21.15.76:6809/147449238] boot 2024-04-26T23:38:43.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:43 smithi072 ceph-mon[28184]: osdmap e33: 5 total, 5 up, 5 in 2024-04-26T23:38:43.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:43 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T23:38:43.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:43 smithi136 ceph-mon[33050]: purged_snaps scrub starts 2024-04-26T23:38:43.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:43 smithi136 ceph-mon[33050]: purged_snaps scrub ok 2024-04-26T23:38:43.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:43 smithi136 ceph-mon[33050]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-26T23:38:43.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:43 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T23:38:43.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:43 smithi136 ceph-mon[33050]: osd.4 [v2:172.21.15.76:6808/147449238,v1:172.21.15.76:6809/147449238] boot 2024-04-26T23:38:43.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:43 smithi136 ceph-mon[33050]: osdmap e33: 5 total, 5 up, 5 in 2024-04-26T23:38:43.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:43 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T23:38:44.095 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi136:vg_nvme/lv_3 2024-04-26T23:38:44.329 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T23:38:44.785 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:44 smithi136 ceph-mon[33050]: osdmap e34: 5 total, 5 up, 5 in 2024-04-26T23:38:44.785 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:44 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:44.785 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:44 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:44.785 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:44 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:38:44.786 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:44 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:38:44.786 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:44 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:44.786 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:44 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:38:44.786 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:44 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:38:44.786 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:44 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:44.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:44 smithi076 ceph-mon[32774]: osdmap e34: 5 total, 5 up, 5 in 2024-04-26T23:38:44.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:44 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:44.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:44 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:44.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:44 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:38:44.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:44 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:38:44.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:44 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:44.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:44 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:38:44.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:44 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:38:44.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:44 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:45.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:44 smithi072 ceph-mon[28184]: osdmap e34: 5 total, 5 up, 5 in 2024-04-26T23:38:45.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:44 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:45.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:44 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:45.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:44 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:38:45.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:44 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:38:45.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:44 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:45.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:44 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:38:45.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:44 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:38:45.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:44 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:38:45.741 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:45 smithi136 ceph-mon[33050]: Detected new or changed devices on smithi076 2024-04-26T23:38:45.742 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:45 smithi136 ceph-mon[33050]: Adjusting osd_memory_target on smithi076 to 8544M 2024-04-26T23:38:45.742 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:45 smithi136 ceph-mon[33050]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:38:45.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:45 smithi076 ceph-mon[32774]: Detected new or changed devices on smithi076 2024-04-26T23:38:45.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:45 smithi076 ceph-mon[32774]: Adjusting osd_memory_target on smithi076 to 8544M 2024-04-26T23:38:45.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:45 smithi076 ceph-mon[32774]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:38:45.955 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:38:45.951+0000 7f3c348cf700 1 -- 172.21.15.136:0/1578013587 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c301091d0 msgr2=0x7f3c3010b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:38:45.955 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:38:45.951+0000 7f3c348cf700 1 --2- 172.21.15.136:0/1578013587 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c301091d0 0x7f3c3010b5e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3c2800b0a0 tx=0x7f3c2800b3b0 comp rx=0 tx=0).stop 2024-04-26T23:38:45.955 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:38:45.951+0000 7f3c348cf700 1 -- 172.21.15.136:0/1578013587 shutdown_connections 2024-04-26T23:38:45.955 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:38:45.951+0000 7f3c348cf700 1 --2- 172.21.15.136:0/1578013587 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c301091d0 0x7f3c3010b5e0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:38:45.955 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:38:45.951+0000 7f3c348cf700 1 --2- 172.21.15.136:0/1578013587 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3c30103f70 0x7f3c30106360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:38:45.955 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:38:45.951+0000 7f3c348cf700 1 --2- 172.21.15.136:0/1578013587 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3c301068a0 0x7f3c30108c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:38:45.955 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:38:45.951+0000 7f3c348cf700 1 -- 172.21.15.136:0/1578013587 >> 172.21.15.136:0/1578013587 conn(0x7f3c300fd940 msgr2=0x7f3c300ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:38:45.955 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:38:45.951+0000 7f3c348cf700 1 -- 172.21.15.136:0/1578013587 shutdown_connections 2024-04-26T23:38:45.955 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:38:45.951+0000 7f3c348cf700 1 -- 172.21.15.136:0/1578013587 wait complete. 2024-04-26T23:38:45.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:38:45.952+0000 7f3c348cf700 1 Processor -- start 2024-04-26T23:38:45.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:38:45.952+0000 7f3c348cf700 1 -- start start 2024-04-26T23:38:45.958 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:38:45.952+0000 7f3c348cf700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3c30103f70 0x7f3c301996b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:38:45.958 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:38:45.952+0000 7f3c348cf700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c301068a0 0x7f3c30199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:38:45.958 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:38:45.952+0000 7f3c348cf700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3c301091d0 0x7f3c301a0c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:38:45.958 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:38:45.952+0000 7f3c348cf700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f3c3010efe0 con 0x7f3c30103f70 2024-04-26T23:38:45.958 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:38:45.952+0000 7f3c348cf700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3c3010ee60 con 0x7f3c301091d0 2024-04-26T23:38:45.958 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:38:45.952+0000 7f3c348cf700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f3c3010f160 con 0x7f3c301068a0 2024-04-26T23:38:45.958 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:38:45.952+0000 7f3c2dd9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c301068a0 0x7f3c30199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:38:45.958 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:38:45.952+0000 7f3c2dd9b700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c301068a0 0x7f3c30199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:55730/0 (socket says 172.21.15.136:55730) 2024-04-26T23:38:45.958 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:38:45.952+0000 7f3c2dd9b700 1 -- 172.21.15.136:0/318373739 learned_addr learned my addr 172.21.15.136:0/318373739 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-26T23:38:45.958 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:38:45.953+0000 7f3c2ed9d700 1 --2- 172.21.15.136:0/318373739 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3c301091d0 0x7f3c301a0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:38:45.958 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:38:45.953+0000 7f3c2dd9b700 1 -- 172.21.15.136:0/318373739 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3c301091d0 msgr2=0x7f3c301a0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:38:45.959 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:38:45.953+0000 7f3c2dd9b700 1 --2- 172.21.15.136:0/318373739 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3c301091d0 0x7f3c301a0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:38:45.959 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:38:45.953+0000 7f3c2dd9b700 1 -- 172.21.15.136:0/318373739 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3c30103f70 msgr2=0x7f3c301996b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:38:45.959 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:38:45.953+0000 7f3c2dd9b700 1 --2- 172.21.15.136:0/318373739 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3c30103f70 0x7f3c301996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:38:45.959 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:38:45.953+0000 7f3c2dd9b700 1 -- 172.21.15.136:0/318373739 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c301a1310 con 0x7f3c301068a0 2024-04-26T23:38:45.959 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:38:45.953+0000 7f3c2ed9d700 1 --2- 172.21.15.136:0/318373739 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3c301091d0 0x7f3c301a0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:38:45.959 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:38:45.953+0000 7f3c2dd9b700 1 --2- 172.21.15.136:0/318373739 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c301068a0 0x7f3c30199bf0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3c2400b560 tx=0x7f3c2400b920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:38:45.959 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:38:45.953+0000 7f3c1f7fe700 1 -- 172.21.15.136:0/318373739 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c24015020 con 0x7f3c301068a0 2024-04-26T23:38:45.959 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:38:45.953+0000 7f3c1f7fe700 1 -- 172.21.15.136:0/318373739 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c24011ce0 con 0x7f3c301068a0 2024-04-26T23:38:45.959 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:38:45.953+0000 7f3c348cf700 1 -- 172.21.15.136:0/318373739 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c301a1600 con 0x7f3c301068a0 2024-04-26T23:38:45.959 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:38:45.953+0000 7f3c1f7fe700 1 -- 172.21.15.136:0/318373739 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c24010da0 con 0x7f3c301068a0 2024-04-26T23:38:45.959 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:38:45.953+0000 7f3c348cf700 1 -- 172.21.15.136:0/318373739 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c301a1b70 con 0x7f3c301068a0 2024-04-26T23:38:45.959 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:38:45.954+0000 7f3c348cf700 1 -- 172.21.15.136:0/318373739 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c3006a110 con 0x7f3c301068a0 2024-04-26T23:38:45.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:38:45.955+0000 7f3c1f7fe700 1 -- 172.21.15.136:0/318373739 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3c24018020 con 0x7f3c301068a0 2024-04-26T23:38:45.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:38:45.956+0000 7f3c1f7fe700 1 --2- 172.21.15.136:0/318373739 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f3c180a3c40 0x7f3c180a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:38:45.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:38:45.956+0000 7f3c1f7fe700 1 -- 172.21.15.136:0/318373739 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7f3c240cf120 con 0x7f3c301068a0 2024-04-26T23:38:45.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:38:45.956+0000 7f3c2e59c700 1 --2- 172.21.15.136:0/318373739 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f3c180a3c40 0x7f3c180a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:38:45.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:38:45.957+0000 7f3c2e59c700 1 --2- 172.21.15.136:0/318373739 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f3c180a3c40 0x7f3c180a6100 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3c200097b0 tx=0x7f3c20006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:38:45.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:38:45.960+0000 7f3c1f7fe700 1 -- 172.21.15.136:0/318373739 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f3c2401f020 con 0x7f3c301068a0 2024-04-26T23:38:46.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:45 smithi072 ceph-mon[28184]: Detected new or changed devices on smithi076 2024-04-26T23:38:46.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:45 smithi072 ceph-mon[28184]: Adjusting osd_memory_target on smithi076 to 8544M 2024-04-26T23:38:46.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:45 smithi072 ceph-mon[28184]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:38:46.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:38:46.165+0000 7f3c348cf700 1 -- 172.21.15.136:0/318373739 --> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f3c30063860 con 0x7f3c180a3c40 2024-04-26T23:38:46.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:46 smithi136 ceph-mon[33050]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:38:46.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:46 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:38:46.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:46 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:38:46.826 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:46 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:38:46.827 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:46 smithi136 ceph-mon[33050]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:38:46.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:46 smithi076 ceph-mon[32774]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:38:46.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:46 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:38:46.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:46 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:38:46.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:46 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:38:46.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:46 smithi076 ceph-mon[32774]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:38:47.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:46 smithi072 ceph-mon[28184]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:38:47.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:46 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:38:47.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:46 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:38:47.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:46 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:38:47.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:46 smithi072 ceph-mon[28184]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:38:49.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:49 smithi136 ceph-mon[33050]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:38:49.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:49 smithi136 ceph-mon[33050]: from='client.? 172.21.15.136:0/3307687979' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c7e887b3-ab38-4608-9bfd-1ed49956e8af"}]: dispatch 2024-04-26T23:38:49.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:49 smithi136 ceph-mon[33050]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c7e887b3-ab38-4608-9bfd-1ed49956e8af"}]: dispatch 2024-04-26T23:38:49.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:49 smithi136 ceph-mon[33050]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c7e887b3-ab38-4608-9bfd-1ed49956e8af"}]': finished 2024-04-26T23:38:49.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:49 smithi136 ceph-mon[33050]: osdmap e35: 6 total, 5 up, 6 in 2024-04-26T23:38:49.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:49 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T23:38:49.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:49 smithi076 ceph-mon[32774]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:38:49.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:49 smithi076 ceph-mon[32774]: from='client.? 172.21.15.136:0/3307687979' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c7e887b3-ab38-4608-9bfd-1ed49956e8af"}]: dispatch 2024-04-26T23:38:49.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:49 smithi076 ceph-mon[32774]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c7e887b3-ab38-4608-9bfd-1ed49956e8af"}]: dispatch 2024-04-26T23:38:49.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:49 smithi076 ceph-mon[32774]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c7e887b3-ab38-4608-9bfd-1ed49956e8af"}]': finished 2024-04-26T23:38:49.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:49 smithi076 ceph-mon[32774]: osdmap e35: 6 total, 5 up, 6 in 2024-04-26T23:38:49.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:49 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T23:38:49.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:49 smithi072 ceph-mon[28184]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:38:49.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:49 smithi072 ceph-mon[28184]: from='client.? 172.21.15.136:0/3307687979' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c7e887b3-ab38-4608-9bfd-1ed49956e8af"}]: dispatch 2024-04-26T23:38:49.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:49 smithi072 ceph-mon[28184]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c7e887b3-ab38-4608-9bfd-1ed49956e8af"}]: dispatch 2024-04-26T23:38:49.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:49 smithi072 ceph-mon[28184]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c7e887b3-ab38-4608-9bfd-1ed49956e8af"}]': finished 2024-04-26T23:38:49.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:49 smithi072 ceph-mon[28184]: osdmap e35: 6 total, 5 up, 6 in 2024-04-26T23:38:49.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:49 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T23:38:50.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:50 smithi076 ceph-mon[32774]: from='client.? 172.21.15.136:0/883707541' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:38:50.695 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:50 smithi072 ceph-mon[28184]: from='client.? 172.21.15.136:0/883707541' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:38:50.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:50 smithi136 ceph-mon[33050]: from='client.? 172.21.15.136:0/883707541' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:38:51.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:51 smithi076 ceph-mon[32774]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:38:51.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:51 smithi072 ceph-mon[28184]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:38:51.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:51 smithi136 ceph-mon[33050]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:38:53.549 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:53 smithi136 ceph-mon[33050]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:38:53.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:53 smithi076 ceph-mon[32774]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:38:53.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:53 smithi072 ceph-mon[28184]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:38:55.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:55 smithi136 ceph-mon[33050]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:38:55.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:55 smithi076 ceph-mon[32774]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:38:55.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:55 smithi072 ceph-mon[28184]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:38:56.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:56 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-26T23:38:56.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:56 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:38:56.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:56 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-26T23:38:56.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:56 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:38:56.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:56 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-26T23:38:56.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:56 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:38:57.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:57 smithi136 ceph-mon[33050]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:38:57.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:57 smithi136 ceph-mon[33050]: Deploying daemon osd.5 on smithi136 2024-04-26T23:38:57.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:57 smithi076 ceph-mon[32774]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:38:57.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:57 smithi076 ceph-mon[32774]: Deploying daemon osd.5 on smithi136 2024-04-26T23:38:57.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:57 smithi072 ceph-mon[28184]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:38:57.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:57 smithi072 ceph-mon[28184]: Deploying daemon osd.5 on smithi136 2024-04-26T23:38:59.645 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:38:59 smithi136 ceph-mon[33050]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:38:59.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:38:59 smithi076 ceph-mon[32774]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:38:59.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:38:59 smithi072 ceph-mon[28184]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:39:01.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:01 smithi076 ceph-mon[32774]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:39:01.739 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:01 smithi072 ceph-mon[28184]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:39:01.744 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:01 smithi136 ceph-mon[33050]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:39:03.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:02 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:03.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:02 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:39:03.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:02 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:03.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:02 smithi072 ceph-mon[28184]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:39:03.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:02 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:03.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:02 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:39:03.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:02 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:03.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:02 smithi136 ceph-mon[33050]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:39:03.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:02 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:03.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:02 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:39:03.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:02 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:03.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:02 smithi076 ceph-mon[32774]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:39:05.452 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:39:05.448+0000 7f3c1f7fe700 1 -- 172.21.15.136:0/318373739 <== mgr.14152 v2:172.21.15.72:6800/4281177709 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f3c30063860 con 0x7f3c180a3c40 2024-04-26T23:39:05.453 INFO:teuthology.orchestra.run.smithi136.stdout:Created osd(s) 5 on host 'smithi136' 2024-04-26T23:39:05.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:39:05.451+0000 7f3c348cf700 1 -- 172.21.15.136:0/318373739 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f3c180a3c40 msgr2=0x7f3c180a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:39:05.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:39:05.451+0000 7f3c348cf700 1 --2- 172.21.15.136:0/318373739 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f3c180a3c40 0x7f3c180a6100 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3c200097b0 tx=0x7f3c20006d20 comp rx=0 tx=0).stop 2024-04-26T23:39:05.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:39:05.451+0000 7f3c348cf700 1 -- 172.21.15.136:0/318373739 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c301068a0 msgr2=0x7f3c30199bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:39:05.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:39:05.451+0000 7f3c348cf700 1 --2- 172.21.15.136:0/318373739 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c301068a0 0x7f3c30199bf0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3c2400b560 tx=0x7f3c2400b920 comp rx=0 tx=0).stop 2024-04-26T23:39:05.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:39:05.451+0000 7f3c348cf700 1 -- 172.21.15.136:0/318373739 shutdown_connections 2024-04-26T23:39:05.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:39:05.451+0000 7f3c348cf700 1 --2- 172.21.15.136:0/318373739 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f3c180a3c40 0x7f3c180a6100 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:39:05.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:39:05.451+0000 7f3c348cf700 1 --2- 172.21.15.136:0/318373739 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3c301091d0 0x7f3c301a0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:39:05.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:39:05.451+0000 7f3c348cf700 1 --2- 172.21.15.136:0/318373739 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3c30103f70 0x7f3c301996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:39:05.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:39:05.451+0000 7f3c348cf700 1 --2- 172.21.15.136:0/318373739 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3c301068a0 0x7f3c30199bf0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:39:05.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:39:05.451+0000 7f3c348cf700 1 -- 172.21.15.136:0/318373739 >> 172.21.15.136:0/318373739 conn(0x7f3c300fd940 msgr2=0x7f3c301048b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:39:05.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:39:05.451+0000 7f3c348cf700 1 -- 172.21.15.136:0/318373739 shutdown_connections 2024-04-26T23:39:05.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:39:05.451+0000 7f3c348cf700 1 -- 172.21.15.136:0/318373739 wait complete. 2024-04-26T23:39:05.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:05 smithi076 ceph-mon[32774]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:39:05.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:05 smithi072 ceph-mon[28184]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:39:05.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:05 smithi136 ceph-mon[33050]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:39:06.154 DEBUG:teuthology.orchestra.run.smithi136:osd.5> sudo journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@osd.5.service 2024-04-26T23:39:06.156 INFO:tasks.cephadm:Deploying osd.6 on smithi072 with /dev/vg_nvme/lv_2... 2024-04-26T23:39:06.156 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-26T23:39:06.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:06 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:06.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:06 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:06.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:06 smithi072 ceph-mon[28184]: from='osd.5 [v2:172.21.15.136:6808/2320149212,v1:172.21.15.136:6809/2320149212]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T23:39:06.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:06 smithi072 ceph-mon[28184]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T23:39:06.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:06 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:06.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:06 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:06.601 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:39:06.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:06 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:06.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:06 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:06.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:06 smithi076 ceph-mon[32774]: from='osd.5 [v2:172.21.15.136:6808/2320149212,v1:172.21.15.136:6809/2320149212]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T23:39:06.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:06 smithi076 ceph-mon[32774]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T23:39:06.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:06 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:06.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:06 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:06.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:06 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:06.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:06 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:06.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:06 smithi136 ceph-mon[33050]: from='osd.5 [v2:172.21.15.136:6808/2320149212,v1:172.21.15.136:6809/2320149212]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T23:39:06.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:06 smithi136 ceph-mon[33050]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T23:39:06.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:06 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:06.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:06 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:07.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:07 smithi072 ceph-mon[28184]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:39:07.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:07 smithi072 ceph-mon[28184]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-26T23:39:07.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:07 smithi072 ceph-mon[28184]: from='osd.5 [v2:172.21.15.136:6808/2320149212,v1:172.21.15.136:6809/2320149212]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-04-26T23:39:07.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:07 smithi072 ceph-mon[28184]: osdmap e36: 6 total, 5 up, 6 in 2024-04-26T23:39:07.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:07 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T23:39:07.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:07 smithi072 ceph-mon[28184]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-04-26T23:39:07.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:07 smithi136 ceph-mon[33050]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:39:07.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:07 smithi136 ceph-mon[33050]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-26T23:39:07.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:07 smithi136 ceph-mon[33050]: from='osd.5 [v2:172.21.15.136:6808/2320149212,v1:172.21.15.136:6809/2320149212]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-04-26T23:39:07.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:07 smithi136 ceph-mon[33050]: osdmap e36: 6 total, 5 up, 6 in 2024-04-26T23:39:07.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:07 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T23:39:07.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:07 smithi136 ceph-mon[33050]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-04-26T23:39:07.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:07 smithi076 ceph-mon[32774]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:39:07.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:07 smithi076 ceph-mon[32774]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-26T23:39:07.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:07 smithi076 ceph-mon[32774]: from='osd.5 [v2:172.21.15.136:6808/2320149212,v1:172.21.15.136:6809/2320149212]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-04-26T23:39:07.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:07 smithi076 ceph-mon[32774]: osdmap e36: 6 total, 5 up, 6 in 2024-04-26T23:39:07.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:07 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T23:39:07.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:07 smithi076 ceph-mon[32774]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-04-26T23:39:08.127 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 26 23:39:07 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-5[44688]: 2024-04-26T23:39:07.867+0000 7f6f3de89700 -1 osd.5 0 waiting for initial osdmap 2024-04-26T23:39:08.127 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 26 23:39:07 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-5[44688]: 2024-04-26T23:39:07.875+0000 7f6f35473700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T23:39:08.691 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:08 smithi072 ceph-mon[28184]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-04-26T23:39:08.691 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:08 smithi072 ceph-mon[28184]: osdmap e37: 6 total, 5 up, 6 in 2024-04-26T23:39:08.691 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:08 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T23:39:08.691 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:08 smithi072 ceph-mon[28184]: from='osd.5 ' entity='osd.5' 2024-04-26T23:39:08.691 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:08 smithi072 ceph-mon[28184]: osd.5 [v2:172.21.15.136:6808/2320149212,v1:172.21.15.136:6809/2320149212] boot 2024-04-26T23:39:08.691 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:08 smithi072 ceph-mon[28184]: osdmap e38: 6 total, 6 up, 6 in 2024-04-26T23:39:08.691 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:08 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T23:39:08.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:08 smithi136 ceph-mon[33050]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-04-26T23:39:08.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:08 smithi136 ceph-mon[33050]: osdmap e37: 6 total, 5 up, 6 in 2024-04-26T23:39:08.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:08 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T23:39:08.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:08 smithi136 ceph-mon[33050]: from='osd.5 ' entity='osd.5' 2024-04-26T23:39:08.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:08 smithi136 ceph-mon[33050]: osd.5 [v2:172.21.15.136:6808/2320149212,v1:172.21.15.136:6809/2320149212] boot 2024-04-26T23:39:08.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:08 smithi136 ceph-mon[33050]: osdmap e38: 6 total, 6 up, 6 in 2024-04-26T23:39:08.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:08 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T23:39:08.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:08 smithi076 ceph-mon[32774]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-04-26T23:39:08.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:08 smithi076 ceph-mon[32774]: osdmap e37: 6 total, 5 up, 6 in 2024-04-26T23:39:08.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:08 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T23:39:08.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:08 smithi076 ceph-mon[32774]: from='osd.5 ' entity='osd.5' 2024-04-26T23:39:08.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:08 smithi076 ceph-mon[32774]: osd.5 [v2:172.21.15.136:6808/2320149212,v1:172.21.15.136:6809/2320149212] boot 2024-04-26T23:39:08.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:08 smithi076 ceph-mon[32774]: osdmap e38: 6 total, 6 up, 6 in 2024-04-26T23:39:08.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:08 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T23:39:09.702 INFO:teuthology.orchestra.run.smithi072.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-26T23:39:09.702 INFO:teuthology.orchestra.run.smithi072.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-26T23:39:09.702 INFO:teuthology.orchestra.run.smithi072.stderr: stderr: 10+0 records in 2024-04-26T23:39:09.702 INFO:teuthology.orchestra.run.smithi072.stderr:10+0 records out 2024-04-26T23:39:09.702 INFO:teuthology.orchestra.run.smithi072.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0365396 s, 287 MB/s 2024-04-26T23:39:09.702 INFO:teuthology.orchestra.run.smithi072.stderr:--> Zapping successful for: 2024-04-26T23:39:09.725 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:09 smithi136 ceph-mon[33050]: purged_snaps scrub starts 2024-04-26T23:39:09.725 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:09 smithi136 ceph-mon[33050]: purged_snaps scrub ok 2024-04-26T23:39:09.725 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:09 smithi136 ceph-mon[33050]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:39:09.725 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:09 smithi136 ceph-mon[33050]: osdmap e39: 6 total, 6 up, 6 in 2024-04-26T23:39:09.748 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:09 smithi072 ceph-mon[28184]: purged_snaps scrub starts 2024-04-26T23:39:09.748 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:09 smithi072 ceph-mon[28184]: purged_snaps scrub ok 2024-04-26T23:39:09.748 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:09 smithi072 ceph-mon[28184]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:39:09.748 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:09 smithi072 ceph-mon[28184]: osdmap e39: 6 total, 6 up, 6 in 2024-04-26T23:39:09.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:09 smithi076 ceph-mon[32774]: purged_snaps scrub starts 2024-04-26T23:39:09.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:09 smithi076 ceph-mon[32774]: purged_snaps scrub ok 2024-04-26T23:39:09.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:09 smithi076 ceph-mon[32774]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-26T23:39:09.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:09 smithi076 ceph-mon[32774]: osdmap e39: 6 total, 6 up, 6 in 2024-04-26T23:39:10.648 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi072:vg_nvme/lv_2 2024-04-26T23:39:11.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:10 smithi072 ceph-mon[28184]: Detected new or changed devices on smithi136 2024-04-26T23:39:11.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:10 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:11.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:10 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:11.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:10 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:39:11.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:10 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:39:11.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:10 smithi072 ceph-mon[28184]: Adjusting osd_memory_target on smithi136 to 8544M 2024-04-26T23:39:11.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:10 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:11.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:10 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:39:11.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:10 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:39:11.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:10 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:11.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:10 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:39:11.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:10 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:39:11.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:10 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:39:11.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:10 smithi072 ceph-mon[28184]: osdmap e40: 6 total, 6 up, 6 in 2024-04-26T23:39:11.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:10 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:11.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:10 smithi072 ceph-mon[28184]: pgmap v156: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 205 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T23:39:11.080 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:39:11.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:10 smithi076 ceph-mon[32774]: Detected new or changed devices on smithi136 2024-04-26T23:39:11.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:10 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:11.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:10 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:11.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:10 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:39:11.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:10 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:39:11.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:10 smithi076 ceph-mon[32774]: Adjusting osd_memory_target on smithi136 to 8544M 2024-04-26T23:39:11.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:10 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:11.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:10 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:39:11.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:10 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:39:11.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:10 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:11.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:10 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:39:11.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:10 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:39:11.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:10 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:39:11.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:10 smithi076 ceph-mon[32774]: osdmap e40: 6 total, 6 up, 6 in 2024-04-26T23:39:11.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:10 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:11.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:10 smithi076 ceph-mon[32774]: pgmap v156: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 205 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T23:39:11.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:10 smithi136 ceph-mon[33050]: Detected new or changed devices on smithi136 2024-04-26T23:39:11.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:10 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:11.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:10 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:11.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:10 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:39:11.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:10 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:39:11.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:10 smithi136 ceph-mon[33050]: Adjusting osd_memory_target on smithi136 to 8544M 2024-04-26T23:39:11.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:10 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:11.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:10 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:39:11.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:10 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:39:11.301 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:10 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:11.301 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:10 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:39:11.301 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:10 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:39:11.301 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:10 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:39:11.301 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:10 smithi136 ceph-mon[33050]: osdmap e40: 6 total, 6 up, 6 in 2024-04-26T23:39:11.301 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:10 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:11.301 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:10 smithi136 ceph-mon[33050]: pgmap v156: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 205 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T23:39:12.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:11 smithi072 ceph-mon[28184]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-26T23:39:12.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:11 smithi076 ceph-mon[32774]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-26T23:39:12.201 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:11 smithi136 ceph-mon[33050]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-26T23:39:12.929 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:12 smithi072 ceph-mon[28184]: osdmap e41: 6 total, 6 up, 6 in 2024-04-26T23:39:12.930 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:12 smithi072 ceph-mon[28184]: pgmap v158: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 205 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T23:39:12.930 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:12 smithi072 ceph-mon[28184]: osdmap e42: 6 total, 6 up, 6 in 2024-04-26T23:39:13.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:12 smithi076 ceph-mon[32774]: osdmap e41: 6 total, 6 up, 6 in 2024-04-26T23:39:13.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:12 smithi076 ceph-mon[32774]: pgmap v158: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 205 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T23:39:13.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:12 smithi076 ceph-mon[32774]: osdmap e42: 6 total, 6 up, 6 in 2024-04-26T23:39:13.217 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:12 smithi136 ceph-mon[33050]: osdmap e41: 6 total, 6 up, 6 in 2024-04-26T23:39:13.217 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:12 smithi136 ceph-mon[33050]: pgmap v158: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 205 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T23:39:13.217 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:12 smithi136 ceph-mon[33050]: osdmap e42: 6 total, 6 up, 6 in 2024-04-26T23:39:13.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:13.440+0000 7f8b02200700 1 -- 172.21.15.72:0/1070108262 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8afc103f20 msgr2=0x7f8afc10a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:39:13.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:13.440+0000 7f8b02200700 1 --2- 172.21.15.72:0/1070108262 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8afc103f20 0x7f8afc10a7d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8af400b0d0 tx=0x7f8af400b3e0 comp rx=0 tx=0).stop 2024-04-26T23:39:13.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:13.440+0000 7f8b02200700 1 -- 172.21.15.72:0/1070108262 shutdown_connections 2024-04-26T23:39:13.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:13.440+0000 7f8b02200700 1 --2- 172.21.15.72:0/1070108262 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8afc103f20 0x7f8afc10a7d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:39:13.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:13.440+0000 7f8b02200700 1 --2- 172.21.15.72:0/1070108262 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8afc102380 0x7f8afc1027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:39:13.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:13.440+0000 7f8b02200700 1 --2- 172.21.15.72:0/1070108262 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8afc103580 0x7f8afc1039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:39:13.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:13.440+0000 7f8b02200700 1 -- 172.21.15.72:0/1070108262 >> 172.21.15.72:0/1070108262 conn(0x7f8afc0fd920 msgr2=0x7f8afc0ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:39:13.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:13.440+0000 7f8b02200700 1 -- 172.21.15.72:0/1070108262 shutdown_connections 2024-04-26T23:39:13.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:13.440+0000 7f8b02200700 1 -- 172.21.15.72:0/1070108262 wait complete. 2024-04-26T23:39:13.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:13.441+0000 7f8b02200700 1 Processor -- start 2024-04-26T23:39:13.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:13.441+0000 7f8b02200700 1 -- start start 2024-04-26T23:39:13.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:13.442+0000 7f8b02200700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8afc102380 0x7f8afc1062c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:39:13.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:13.442+0000 7f8b02200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8afc103580 0x7f8afc104960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:39:13.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:13.442+0000 7f8b02200700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8afc103f20 0x7f8afc104ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:39:13.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:13.442+0000 7f8b02200700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f8afc10de40 con 0x7f8afc103f20 2024-04-26T23:39:13.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:13.442+0000 7f8b02200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8afc10dcc0 con 0x7f8afc103580 2024-04-26T23:39:13.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:13.442+0000 7f8b02200700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f8afc10dfc0 con 0x7f8afc102380 2024-04-26T23:39:13.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:13.442+0000 7f8afb400700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8afc102380 0x7f8afc1062c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:39:13.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:13.442+0000 7f8afbe00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8afc103f20 0x7f8afc104ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:39:13.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:13.442+0000 7f8afb400700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8afc102380 0x7f8afc1062c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.72:39292/0 (socket says 172.21.15.72:39292) 2024-04-26T23:39:13.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:13.442+0000 7f8afb400700 1 -- 172.21.15.72:0/885071015 learned_addr learned my addr 172.21.15.72:0/885071015 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:39:13.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:13.442+0000 7f8afaa00700 1 --2- 172.21.15.72:0/885071015 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8afc103580 0x7f8afc104960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:39:13.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:13.443+0000 7f8afb400700 1 -- 172.21.15.72:0/885071015 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8afc103580 msgr2=0x7f8afc104960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:39:13.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:13.443+0000 7f8afb400700 1 --2- 172.21.15.72:0/885071015 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8afc103580 0x7f8afc104960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:39:13.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:13.443+0000 7f8afb400700 1 -- 172.21.15.72:0/885071015 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8afc103f20 msgr2=0x7f8afc104ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:39:13.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:13.443+0000 7f8afb400700 1 --2- 172.21.15.72:0/885071015 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8afc103f20 0x7f8afc104ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:39:13.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:13.443+0000 7f8afb400700 1 -- 172.21.15.72:0/885071015 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8afc105770 con 0x7f8afc102380 2024-04-26T23:39:13.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:13.443+0000 7f8afb400700 1 --2- 172.21.15.72:0/885071015 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8afc102380 0x7f8afc1062c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8aec00ea50 tx=0x7f8aec00ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:39:13.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:13.443+0000 7f8af3e00700 1 -- 172.21.15.72:0/885071015 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aec00cbd0 con 0x7f8afc102380 2024-04-26T23:39:13.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:13.443+0000 7f8af3e00700 1 -- 172.21.15.72:0/885071015 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8aec00cd70 con 0x7f8afc102380 2024-04-26T23:39:13.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:13.443+0000 7f8b02200700 1 -- 172.21.15.72:0/885071015 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8afc1a3730 con 0x7f8afc102380 2024-04-26T23:39:13.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:13.443+0000 7f8af3e00700 1 -- 172.21.15.72:0/885071015 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aec018680 con 0x7f8afc102380 2024-04-26T23:39:13.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:13.443+0000 7f8b02200700 1 -- 172.21.15.72:0/885071015 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8afc1a3c90 con 0x7f8afc102380 2024-04-26T23:39:13.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:13.444+0000 7f8b02200700 1 -- 172.21.15.72:0/885071015 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8afc06a110 con 0x7f8afc102380 2024-04-26T23:39:13.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:13.446+0000 7f8af3e00700 1 -- 172.21.15.72:0/885071015 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8aec018820 con 0x7f8afc102380 2024-04-26T23:39:13.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:13.447+0000 7f8af3e00700 1 --2- 172.21.15.72:0/885071015 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f8ae00a3cc0 0x7f8ae00a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:39:13.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:13.448+0000 7f8af3e00700 1 -- 172.21.15.72:0/885071015 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 4533+0+0 (secure 0 0 0) 0x7f8aec0cf8b0 con 0x7f8afc102380 2024-04-26T23:39:13.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:13.448+0000 7f8afaa00700 1 --2- 172.21.15.72:0/885071015 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f8ae00a3cc0 0x7f8ae00a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:39:13.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:13.448+0000 7f8afaa00700 1 --2- 172.21.15.72:0/885071015 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f8ae00a3cc0 0x7f8ae00a6180 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8ae8007be0 tx=0x7f8ae80073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:39:13.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:13.452+0000 7f8af3e00700 1 -- 172.21.15.72:0/885071015 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8aec054060 con 0x7f8afc102380 2024-04-26T23:39:13.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:13.691+0000 7f8b02200700 1 -- 172.21.15.72:0/885071015 --> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi072:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f8afc0ffb20 con 0x7f8ae00a3cc0 2024-04-26T23:39:14.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:13 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:39:14.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:13 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:39:14.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:13 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:39:14.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:13 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:39:14.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:13 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:39:14.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:13 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:39:14.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:13 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:39:14.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:13 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:39:14.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:13 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:39:14.962 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:14 smithi072 ceph-mon[28184]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi072:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:39:14.962 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:14 smithi072 ceph-mon[28184]: pgmap v160: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T23:39:15.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:14 smithi076 ceph-mon[32774]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi072:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:39:15.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:14 smithi076 ceph-mon[32774]: pgmap v160: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T23:39:15.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:14 smithi136 ceph-mon[33050]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi072:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:39:15.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:14 smithi136 ceph-mon[33050]: pgmap v160: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-26T23:39:16.621 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:16 smithi072 ceph-mon[28184]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-26T23:39:16.621 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:16 smithi072 ceph-mon[28184]: Cluster is now healthy 2024-04-26T23:39:16.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:16 smithi136 ceph-mon[33050]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-26T23:39:16.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:16 smithi136 ceph-mon[33050]: Cluster is now healthy 2024-04-26T23:39:16.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:16 smithi076 ceph-mon[32774]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-26T23:39:16.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:16 smithi076 ceph-mon[32774]: Cluster is now healthy 2024-04-26T23:39:17.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:17 smithi072 ceph-mon[28184]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-04-26T23:39:17.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:17 smithi136 ceph-mon[33050]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-04-26T23:39:17.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:17 smithi076 ceph-mon[32774]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-04-26T23:39:18.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:18 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/3547395201' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f0bd604b-0f57-4a38-9295-eb200264f1ff"}]: dispatch 2024-04-26T23:39:18.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:18 smithi072 ceph-mon[28184]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f0bd604b-0f57-4a38-9295-eb200264f1ff"}]: dispatch 2024-04-26T23:39:18.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:18 smithi072 ceph-mon[28184]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f0bd604b-0f57-4a38-9295-eb200264f1ff"}]': finished 2024-04-26T23:39:18.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:18 smithi072 ceph-mon[28184]: osdmap e43: 7 total, 6 up, 7 in 2024-04-26T23:39:18.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:18 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T23:39:18.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:18 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/3547395201' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f0bd604b-0f57-4a38-9295-eb200264f1ff"}]: dispatch 2024-04-26T23:39:18.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:18 smithi136 ceph-mon[33050]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f0bd604b-0f57-4a38-9295-eb200264f1ff"}]: dispatch 2024-04-26T23:39:18.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:18 smithi136 ceph-mon[33050]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f0bd604b-0f57-4a38-9295-eb200264f1ff"}]': finished 2024-04-26T23:39:18.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:18 smithi136 ceph-mon[33050]: osdmap e43: 7 total, 6 up, 7 in 2024-04-26T23:39:18.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:18 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T23:39:18.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:18 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/3547395201' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f0bd604b-0f57-4a38-9295-eb200264f1ff"}]: dispatch 2024-04-26T23:39:18.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:18 smithi076 ceph-mon[32774]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f0bd604b-0f57-4a38-9295-eb200264f1ff"}]: dispatch 2024-04-26T23:39:18.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:18 smithi076 ceph-mon[32774]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f0bd604b-0f57-4a38-9295-eb200264f1ff"}]': finished 2024-04-26T23:39:18.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:18 smithi076 ceph-mon[32774]: osdmap e43: 7 total, 6 up, 7 in 2024-04-26T23:39:18.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:18 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T23:39:19.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:19 smithi072 ceph-mon[28184]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-04-26T23:39:19.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:19 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/951964962' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:39:19.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:19 smithi136 ceph-mon[33050]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-04-26T23:39:19.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:19 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/951964962' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:39:19.830 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:19 smithi076 ceph-mon[32774]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-04-26T23:39:19.830 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:19 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/951964962' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:39:21.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:21 smithi072 ceph-mon[28184]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T23:39:21.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:21 smithi136 ceph-mon[33050]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T23:39:21.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:21 smithi076 ceph-mon[32774]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T23:39:23.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:23 smithi072 ceph-mon[28184]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-04-26T23:39:23.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:23 smithi136 ceph-mon[33050]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-04-26T23:39:23.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:23 smithi076 ceph-mon[32774]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-04-26T23:39:25.779 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:25 smithi072 ceph-mon[28184]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T23:39:25.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:25 smithi136 ceph-mon[33050]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T23:39:25.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:25 smithi076 ceph-mon[32774]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T23:39:27.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:27 smithi072 ceph-mon[28184]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:39:27.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:27 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-26T23:39:27.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:27 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:39:27.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:27 smithi136 ceph-mon[33050]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:39:27.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:27 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-26T23:39:27.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:27 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:39:27.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:27 smithi076 ceph-mon[32774]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:39:27.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:27 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-26T23:39:27.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:27 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:39:28.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:28 smithi072 ceph-mon[28184]: Deploying daemon osd.6 on smithi072 2024-04-26T23:39:28.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:28 smithi136 ceph-mon[33050]: Deploying daemon osd.6 on smithi072 2024-04-26T23:39:28.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:28 smithi076 ceph-mon[32774]: Deploying daemon osd.6 on smithi072 2024-04-26T23:39:29.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:29 smithi072 ceph-mon[28184]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:39:29.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:29 smithi136 ceph-mon[33050]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:39:29.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:29 smithi076 ceph-mon[32774]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:39:31.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:31 smithi072 ceph-mon[28184]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:39:31.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:31 smithi136 ceph-mon[33050]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:39:31.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:31 smithi076 ceph-mon[32774]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:39:33.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:33 smithi072 ceph-mon[28184]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:39:33.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:33 smithi136 ceph-mon[33050]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:39:33.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:33 smithi076 ceph-mon[32774]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:39:35.689 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:35 smithi072 ceph-mon[28184]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:39:35.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:35 smithi136 ceph-mon[33050]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:39:35.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:35 smithi076 ceph-mon[32774]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:39:37.223 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:36 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:37.223 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:36 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:37.224 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:36 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:39:37.224 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:36 smithi072 ceph-mon[28184]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:39:37.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:36 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:37.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:36 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:37.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:36 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:39:37.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:36 smithi136 ceph-mon[33050]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:39:37.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:36 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:37.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:36 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:37.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:36 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:39:37.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:36 smithi076 ceph-mon[32774]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:39:39.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:39 smithi076 ceph-mon[32774]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:39:39.767 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:39 smithi072 ceph-mon[28184]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:39:39.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:39 smithi136 ceph-mon[33050]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:39:41.001 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:40 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:41.002 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:40 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:41.002 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:40 smithi072 ceph-mon[28184]: from='osd.6 [v2:172.21.15.72:6818/389499207,v1:172.21.15.72:6819/389499207]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T23:39:41.002 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:40 smithi072 ceph-mon[28184]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T23:39:41.002 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:40 smithi072 ceph-mon[28184]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:39:41.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:40.997+0000 7f8af3e00700 1 -- 172.21.15.72:0/885071015 <== mgr.14152 v2:172.21.15.72:6800/4281177709 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f8afc0ffb20 con 0x7f8ae00a3cc0 2024-04-26T23:39:41.002 INFO:teuthology.orchestra.run.smithi072.stdout:Created osd(s) 6 on host 'smithi072' 2024-04-26T23:39:41.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:41.000+0000 7f8b02200700 1 -- 172.21.15.72:0/885071015 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f8ae00a3cc0 msgr2=0x7f8ae00a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:39:41.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:41.000+0000 7f8b02200700 1 --2- 172.21.15.72:0/885071015 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f8ae00a3cc0 0x7f8ae00a6180 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8ae8007be0 tx=0x7f8ae80073d0 comp rx=0 tx=0).stop 2024-04-26T23:39:41.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:41.000+0000 7f8b02200700 1 -- 172.21.15.72:0/885071015 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8afc102380 msgr2=0x7f8afc1062c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:39:41.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:41.000+0000 7f8b02200700 1 --2- 172.21.15.72:0/885071015 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8afc102380 0x7f8afc1062c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8aec00ea50 tx=0x7f8aec00ed60 comp rx=0 tx=0).stop 2024-04-26T23:39:41.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:41.000+0000 7f8b02200700 1 -- 172.21.15.72:0/885071015 shutdown_connections 2024-04-26T23:39:41.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:41.000+0000 7f8b02200700 1 --2- 172.21.15.72:0/885071015 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f8ae00a3cc0 0x7f8ae00a6180 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:39:41.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:41.000+0000 7f8b02200700 1 --2- 172.21.15.72:0/885071015 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8afc103f20 0x7f8afc104ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:39:41.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:41.000+0000 7f8b02200700 1 --2- 172.21.15.72:0/885071015 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8afc102380 0x7f8afc1062c0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:39:41.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:41.000+0000 7f8b02200700 1 --2- 172.21.15.72:0/885071015 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8afc103580 0x7f8afc104960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:39:41.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:41.000+0000 7f8b02200700 1 -- 172.21.15.72:0/885071015 >> 172.21.15.72:0/885071015 conn(0x7f8afc0fd920 msgr2=0x7f8afc0ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:39:41.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:41.001+0000 7f8b02200700 1 -- 172.21.15.72:0/885071015 shutdown_connections 2024-04-26T23:39:41.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:39:41.001+0000 7f8b02200700 1 -- 172.21.15.72:0/885071015 wait complete. 2024-04-26T23:39:41.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:40 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:41.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:40 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:41.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:40 smithi136 ceph-mon[33050]: from='osd.6 [v2:172.21.15.72:6818/389499207,v1:172.21.15.72:6819/389499207]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T23:39:41.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:40 smithi136 ceph-mon[33050]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T23:39:41.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:40 smithi136 ceph-mon[33050]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:39:41.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:40 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:41.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:40 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:41.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:40 smithi076 ceph-mon[32774]: from='osd.6 [v2:172.21.15.72:6818/389499207,v1:172.21.15.72:6819/389499207]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T23:39:41.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:40 smithi076 ceph-mon[32774]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T23:39:41.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:40 smithi076 ceph-mon[32774]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:39:41.927 DEBUG:teuthology.orchestra.run.smithi072:osd.6> sudo journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@osd.6.service 2024-04-26T23:39:41.932 INFO:tasks.cephadm:Deploying osd.7 on smithi076 with /dev/vg_nvme/lv_2... 2024-04-26T23:39:41.932 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-26T23:39:42.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:41 smithi072 ceph-mon[28184]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T23:39:42.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:41 smithi072 ceph-mon[28184]: osdmap e44: 7 total, 6 up, 7 in 2024-04-26T23:39:42.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:41 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T23:39:42.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:41 smithi072 ceph-mon[28184]: from='osd.6 [v2:172.21.15.72:6818/389499207,v1:172.21.15.72:6819/389499207]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]: dispatch 2024-04-26T23:39:42.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:41 smithi072 ceph-mon[28184]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]: dispatch 2024-04-26T23:39:42.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:41 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:39:42.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:41 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:39:42.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:41 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:42.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:41 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:42.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:41 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:42.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:41 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:39:42.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:41 smithi136 ceph-mon[33050]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T23:39:42.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:41 smithi136 ceph-mon[33050]: osdmap e44: 7 total, 6 up, 7 in 2024-04-26T23:39:42.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:41 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T23:39:42.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:41 smithi136 ceph-mon[33050]: from='osd.6 [v2:172.21.15.72:6818/389499207,v1:172.21.15.72:6819/389499207]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]: dispatch 2024-04-26T23:39:42.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:41 smithi136 ceph-mon[33050]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]: dispatch 2024-04-26T23:39:42.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:41 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:39:42.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:41 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:39:42.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:41 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:42.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:41 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:42.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:41 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:42.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:41 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:39:42.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:41 smithi076 ceph-mon[32774]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T23:39:42.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:41 smithi076 ceph-mon[32774]: osdmap e44: 7 total, 6 up, 7 in 2024-04-26T23:39:42.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:41 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T23:39:42.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:41 smithi076 ceph-mon[32774]: from='osd.6 [v2:172.21.15.72:6818/389499207,v1:172.21.15.72:6819/389499207]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]: dispatch 2024-04-26T23:39:42.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:41 smithi076 ceph-mon[32774]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]: dispatch 2024-04-26T23:39:42.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:41 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:39:42.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:41 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:39:42.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:41 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:42.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:41 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:42.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:41 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:42.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:41 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:39:42.180 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T23:39:42.455 INFO:journalctl@ceph.osd.6.smithi072.stdout:Apr 26 23:39:42 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-6[54140]: 2024-04-26T23:39:42.406+0000 7fa4a6a00700 -1 osd.6 0 waiting for initial osdmap 2024-04-26T23:39:42.455 INFO:journalctl@ceph.osd.6.smithi072.stdout:Apr 26 23:39:42 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-6[54140]: 2024-04-26T23:39:42.450+0000 7fa49c800700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T23:39:42.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:42 smithi076 ceph-mon[32774]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]': finished 2024-04-26T23:39:42.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:42 smithi076 ceph-mon[32774]: osdmap e45: 7 total, 6 up, 7 in 2024-04-26T23:39:42.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:42 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T23:39:42.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:42 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T23:39:42.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:42 smithi076 ceph-mon[32774]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:39:42.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:42 smithi076 ceph-mon[32774]: from='osd.6 ' entity='osd.6' 2024-04-26T23:39:43.035 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:42 smithi072 ceph-mon[28184]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]': finished 2024-04-26T23:39:43.035 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:42 smithi072 ceph-mon[28184]: osdmap e45: 7 total, 6 up, 7 in 2024-04-26T23:39:43.035 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:42 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T23:39:43.035 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:42 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T23:39:43.035 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:42 smithi072 ceph-mon[28184]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:39:43.036 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:42 smithi072 ceph-mon[28184]: from='osd.6 ' entity='osd.6' 2024-04-26T23:39:43.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:42 smithi136 ceph-mon[33050]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]': finished 2024-04-26T23:39:43.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:42 smithi136 ceph-mon[33050]: osdmap e45: 7 total, 6 up, 7 in 2024-04-26T23:39:43.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:42 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T23:39:43.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:42 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T23:39:43.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:42 smithi136 ceph-mon[33050]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-26T23:39:43.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:42 smithi136 ceph-mon[33050]: from='osd.6 ' entity='osd.6' 2024-04-26T23:39:43.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:43 smithi076 ceph-mon[32774]: purged_snaps scrub starts 2024-04-26T23:39:43.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:43 smithi076 ceph-mon[32774]: purged_snaps scrub ok 2024-04-26T23:39:43.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:43 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T23:39:43.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:43 smithi076 ceph-mon[32774]: osd.6 [v2:172.21.15.72:6818/389499207,v1:172.21.15.72:6819/389499207] boot 2024-04-26T23:39:43.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:43 smithi076 ceph-mon[32774]: osdmap e46: 7 total, 7 up, 7 in 2024-04-26T23:39:43.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:43 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T23:39:44.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:43 smithi072 ceph-mon[28184]: purged_snaps scrub starts 2024-04-26T23:39:44.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:43 smithi072 ceph-mon[28184]: purged_snaps scrub ok 2024-04-26T23:39:44.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:43 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T23:39:44.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:43 smithi072 ceph-mon[28184]: osd.6 [v2:172.21.15.72:6818/389499207,v1:172.21.15.72:6819/389499207] boot 2024-04-26T23:39:44.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:43 smithi072 ceph-mon[28184]: osdmap e46: 7 total, 7 up, 7 in 2024-04-26T23:39:44.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:43 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T23:39:44.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:43 smithi136 ceph-mon[33050]: purged_snaps scrub starts 2024-04-26T23:39:44.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:43 smithi136 ceph-mon[33050]: purged_snaps scrub ok 2024-04-26T23:39:44.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:43 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T23:39:44.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:43 smithi136 ceph-mon[33050]: osd.6 [v2:172.21.15.72:6818/389499207,v1:172.21.15.72:6819/389499207] boot 2024-04-26T23:39:44.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:43 smithi136 ceph-mon[33050]: osdmap e46: 7 total, 7 up, 7 in 2024-04-26T23:39:44.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:43 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T23:39:44.129 INFO:teuthology.orchestra.run.smithi076.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-26T23:39:44.129 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-26T23:39:44.129 INFO:teuthology.orchestra.run.smithi076.stderr: stderr: 10+0 records in 2024-04-26T23:39:44.129 INFO:teuthology.orchestra.run.smithi076.stderr:10+0 records out 2024-04-26T23:39:44.129 INFO:teuthology.orchestra.run.smithi076.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113844 s, 921 MB/s 2024-04-26T23:39:44.129 INFO:teuthology.orchestra.run.smithi076.stderr:--> Zapping successful for: 2024-04-26T23:39:44.842 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:44 smithi076 ceph-mon[32774]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:39:44.868 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi076:vg_nvme/lv_2 2024-04-26T23:39:45.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:44 smithi072 ceph-mon[28184]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:39:45.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:44 smithi136 ceph-mon[33050]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:39:45.115 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T23:39:46.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:45 smithi072 ceph-mon[28184]: osdmap e47: 7 total, 7 up, 7 in 2024-04-26T23:39:46.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:45 smithi136 ceph-mon[33050]: osdmap e47: 7 total, 7 up, 7 in 2024-04-26T23:39:46.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:45 smithi076 ceph-mon[32774]: osdmap e47: 7 total, 7 up, 7 in 2024-04-26T23:39:46.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:39:46.849+0000 7ff1b26c3700 1 -- 172.21.15.76:0/2631378884 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff1ac1051d0 msgr2=0x7ff1ac10ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:39:46.854 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:39:46.849+0000 7ff1b26c3700 1 --2- 172.21.15.76:0/2631378884 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff1ac1051d0 0x7ff1ac10ba80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff1a400b0d0 tx=0x7ff1a400b3e0 comp rx=0 tx=0).stop 2024-04-26T23:39:46.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:39:46.849+0000 7ff1b26c3700 1 -- 172.21.15.76:0/2631378884 shutdown_connections 2024-04-26T23:39:46.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:39:46.849+0000 7ff1b26c3700 1 --2- 172.21.15.76:0/2631378884 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff1ac1051d0 0x7ff1ac10ba80 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:39:46.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:39:46.849+0000 7ff1b26c3700 1 --2- 172.21.15.76:0/2631378884 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1ac1017b0 0x7ff1ac101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:39:46.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:39:46.849+0000 7ff1b26c3700 1 --2- 172.21.15.76:0/2631378884 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1ac104830 0x7ff1ac104c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:39:46.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:39:46.849+0000 7ff1b26c3700 1 -- 172.21.15.76:0/2631378884 >> 172.21.15.76:0/2631378884 conn(0x7ff1ac0fd0f0 msgr2=0x7ff1ac0ff530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:39:46.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:39:46.849+0000 7ff1b26c3700 1 -- 172.21.15.76:0/2631378884 shutdown_connections 2024-04-26T23:39:46.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:39:46.849+0000 7ff1b26c3700 1 -- 172.21.15.76:0/2631378884 wait complete. 2024-04-26T23:39:46.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:39:46.849+0000 7ff1b26c3700 1 Processor -- start 2024-04-26T23:39:46.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:39:46.849+0000 7ff1b26c3700 1 -- start start 2024-04-26T23:39:46.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:39:46.850+0000 7ff1b26c3700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1ac1017b0 0x7ff1ac1995a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:39:46.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:39:46.850+0000 7ff1b26c3700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1ac104830 0x7ff1ac199ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:39:46.855 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:39:46.850+0000 7ff1b26c3700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff1ac1051d0 0x7ff1ac1a0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:39:46.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:39:46.850+0000 7ff1b26c3700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ac10efc0 con 0x7ff1ac104830 2024-04-26T23:39:46.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:39:46.850+0000 7ff1b26c3700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ac10ee40 con 0x7ff1ac1051d0 2024-04-26T23:39:46.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:39:46.850+0000 7ff1b26c3700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ac10f140 con 0x7ff1ac1017b0 2024-04-26T23:39:46.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:39:46.850+0000 7ff1b0c60700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff1ac1051d0 0x7ff1ac1a0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:39:46.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:39:46.850+0000 7ff1b0c60700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff1ac1051d0 0x7ff1ac1a0b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:39228/0 (socket says 172.21.15.76:39228) 2024-04-26T23:39:46.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:39:46.850+0000 7ff1b0c60700 1 -- 172.21.15.76:0/4189346159 learned_addr learned my addr 172.21.15.76:0/4189346159 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-26T23:39:46.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:39:46.850+0000 7ff1abfff700 1 --2- 172.21.15.76:0/4189346159 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1ac1017b0 0x7ff1ac1995a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:39:46.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:39:46.850+0000 7ff1b0c60700 1 -- 172.21.15.76:0/4189346159 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1ac1017b0 msgr2=0x7ff1ac1995a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:39:46.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:39:46.850+0000 7ff1b0c60700 1 --2- 172.21.15.76:0/4189346159 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1ac1017b0 0x7ff1ac1995a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:39:46.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:39:46.850+0000 7ff1b0c60700 1 -- 172.21.15.76:0/4189346159 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1ac104830 msgr2=0x7ff1ac199ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:39:46.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:39:46.850+0000 7ff1b0c60700 1 --2- 172.21.15.76:0/4189346159 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1ac104830 0x7ff1ac199ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:39:46.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:39:46.850+0000 7ff1b0c60700 1 -- 172.21.15.76:0/4189346159 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1ac1a1200 con 0x7ff1ac1051d0 2024-04-26T23:39:46.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:39:46.850+0000 7ff1b0c60700 1 --2- 172.21.15.76:0/4189346159 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff1ac1051d0 0x7ff1ac1a0b60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff1a4009040 tx=0x7ff1a4004d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:39:46.856 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:39:46.850+0000 7ff1a97fa700 1 -- 172.21.15.76:0/4189346159 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a401f030 con 0x7ff1ac1051d0 2024-04-26T23:39:46.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:39:46.850+0000 7ff1b26c3700 1 -- 172.21.15.76:0/4189346159 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1ac1a1490 con 0x7ff1ac1051d0 2024-04-26T23:39:46.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:39:46.850+0000 7ff1a97fa700 1 -- 172.21.15.76:0/4189346159 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1a4003a00 con 0x7ff1ac1051d0 2024-04-26T23:39:46.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:39:46.850+0000 7ff1a97fa700 1 -- 172.21.15.76:0/4189346159 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a40047b0 con 0x7ff1ac1051d0 2024-04-26T23:39:46.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:39:46.850+0000 7ff1b26c3700 1 -- 172.21.15.76:0/4189346159 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1ac1a1960 con 0x7ff1ac1051d0 2024-04-26T23:39:46.857 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:39:46.851+0000 7ff1b26c3700 1 -- 172.21.15.76:0/4189346159 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1ac02e490 con 0x7ff1ac1051d0 2024-04-26T23:39:46.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:39:46.853+0000 7ff1a97fa700 1 -- 172.21.15.76:0/4189346159 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ff1a4003bb0 con 0x7ff1ac1051d0 2024-04-26T23:39:46.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:39:46.853+0000 7ff1a97fa700 1 --2- 172.21.15.76:0/4189346159 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff1940a3c40 0x7ff1940a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:39:46.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:39:46.853+0000 7ff1a97fa700 1 -- 172.21.15.76:0/4189346159 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7ff1a40d0880 con 0x7ff1ac1051d0 2024-04-26T23:39:46.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:39:46.854+0000 7ff1abfff700 1 --2- 172.21.15.76:0/4189346159 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff1940a3c40 0x7ff1940a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:39:46.858 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:39:46.854+0000 7ff1abfff700 1 --2- 172.21.15.76:0/4189346159 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff1940a3c40 0x7ff1940a6100 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac06b8b0 tx=0x7ff19c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:39:46.860 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:39:46.857+0000 7ff1a97fa700 1 -- 172.21.15.76:0/4189346159 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff1a409e000 con 0x7ff1ac1051d0 2024-04-26T23:39:46.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:46 smithi076 ceph-mon[32774]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:39:47.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:46 smithi072 ceph-mon[28184]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:39:47.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:46 smithi136 ceph-mon[33050]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:39:47.066 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:39:47.062+0000 7ff1b26c3700 1 -- 172.21.15.76:0/4189346159 --> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7ff1ac063860 con 0x7ff1940a3c40 2024-04-26T23:39:47.972 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:47 smithi076 ceph-mon[32774]: from='client.24287 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:39:47.972 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:47 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:39:47.972 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:47 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:39:47.972 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:47 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:39:47.972 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:47 smithi076 ceph-mon[32774]: Detected new or changed devices on smithi072 2024-04-26T23:39:47.972 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:47 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:47.972 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:47 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:47.972 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:47 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:39:47.972 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:47 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:39:47.972 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:47 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:39:47.972 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:47 smithi076 ceph-mon[32774]: Adjusting osd_memory_target on smithi072 to 5696M 2024-04-26T23:39:47.972 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:47 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:47.973 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:47 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:39:47.973 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:47 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:39:47.973 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:47 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:48.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:47 smithi072 ceph-mon[28184]: from='client.24287 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:39:48.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:47 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:39:48.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:47 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:39:48.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:47 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:39:48.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:47 smithi072 ceph-mon[28184]: Detected new or changed devices on smithi072 2024-04-26T23:39:48.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:47 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:48.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:47 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:48.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:47 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:39:48.045 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:47 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:39:48.045 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:47 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:39:48.045 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:47 smithi072 ceph-mon[28184]: Adjusting osd_memory_target on smithi072 to 5696M 2024-04-26T23:39:48.045 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:47 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:48.045 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:47 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:39:48.045 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:47 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:39:48.045 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:47 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:48.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:47 smithi136 ceph-mon[33050]: from='client.24287 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:39:48.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:47 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:39:48.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:47 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:39:48.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:47 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:39:48.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:47 smithi136 ceph-mon[33050]: Detected new or changed devices on smithi072 2024-04-26T23:39:48.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:47 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:48.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:47 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:48.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:47 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:39:48.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:47 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:39:48.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:47 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:39:48.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:47 smithi136 ceph-mon[33050]: Adjusting osd_memory_target on smithi072 to 5696M 2024-04-26T23:39:48.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:47 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:48.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:47 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:39:48.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:47 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:39:48.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:47 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:39:49.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:48 smithi072 ceph-mon[28184]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:39:49.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:48 smithi136 ceph-mon[33050]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:39:49.062 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:48 smithi076 ceph-mon[32774]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:39:50.285 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:49 smithi076 ceph-mon[32774]: from='client.? 172.21.15.76:0/3048160179' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1879d4e4-038e-4cd6-a3ec-3a940bcc3797"}]: dispatch 2024-04-26T23:39:50.285 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:49 smithi076 ceph-mon[32774]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1879d4e4-038e-4cd6-a3ec-3a940bcc3797"}]: dispatch 2024-04-26T23:39:50.285 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:49 smithi076 ceph-mon[32774]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1879d4e4-038e-4cd6-a3ec-3a940bcc3797"}]': finished 2024-04-26T23:39:50.285 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:49 smithi076 ceph-mon[32774]: osdmap e48: 8 total, 7 up, 8 in 2024-04-26T23:39:50.285 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:49 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T23:39:50.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:49 smithi072 ceph-mon[28184]: from='client.? 172.21.15.76:0/3048160179' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1879d4e4-038e-4cd6-a3ec-3a940bcc3797"}]: dispatch 2024-04-26T23:39:50.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:49 smithi072 ceph-mon[28184]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1879d4e4-038e-4cd6-a3ec-3a940bcc3797"}]: dispatch 2024-04-26T23:39:50.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:49 smithi072 ceph-mon[28184]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1879d4e4-038e-4cd6-a3ec-3a940bcc3797"}]': finished 2024-04-26T23:39:50.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:49 smithi072 ceph-mon[28184]: osdmap e48: 8 total, 7 up, 8 in 2024-04-26T23:39:50.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:49 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T23:39:50.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:49 smithi136 ceph-mon[33050]: from='client.? 172.21.15.76:0/3048160179' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1879d4e4-038e-4cd6-a3ec-3a940bcc3797"}]: dispatch 2024-04-26T23:39:50.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:49 smithi136 ceph-mon[33050]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1879d4e4-038e-4cd6-a3ec-3a940bcc3797"}]: dispatch 2024-04-26T23:39:50.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:49 smithi136 ceph-mon[33050]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1879d4e4-038e-4cd6-a3ec-3a940bcc3797"}]': finished 2024-04-26T23:39:50.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:49 smithi136 ceph-mon[33050]: osdmap e48: 8 total, 7 up, 8 in 2024-04-26T23:39:50.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:49 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T23:39:51.197 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:50 smithi072 ceph-mon[28184]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:39:51.197 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:50 smithi072 ceph-mon[28184]: from='client.? 172.21.15.76:0/651267345' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:39:51.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:50 smithi136 ceph-mon[33050]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:39:51.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:50 smithi136 ceph-mon[33050]: from='client.? 172.21.15.76:0/651267345' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:39:51.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:50 smithi076 ceph-mon[32774]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:39:51.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:50 smithi076 ceph-mon[32774]: from='client.? 172.21.15.76:0/651267345' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:39:53.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:53 smithi076 ceph-mon[32774]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:39:53.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:53 smithi072 ceph-mon[28184]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:39:53.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:53 smithi136 ceph-mon[33050]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:39:55.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:55 smithi076 ceph-mon[32774]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:39:55.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:55 smithi072 ceph-mon[28184]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:39:55.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:55 smithi136 ceph-mon[33050]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:39:57.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:57 smithi076 ceph-mon[32774]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:39:57.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:57 smithi072 ceph-mon[28184]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:39:57.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:57 smithi136 ceph-mon[33050]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:39:58.518 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:58 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-26T23:39:58.518 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:58 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:39:58.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:58 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-26T23:39:58.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:58 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:39:58.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:58 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-26T23:39:58.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:58 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:39:59.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:59 smithi076 ceph-mon[32774]: Deploying daemon osd.7 on smithi076 2024-04-26T23:39:59.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:39:59 smithi076 ceph-mon[32774]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:39:59.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:59 smithi072 ceph-mon[28184]: Deploying daemon osd.7 on smithi076 2024-04-26T23:39:59.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:39:59 smithi072 ceph-mon[28184]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:39:59.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:59 smithi136 ceph-mon[33050]: Deploying daemon osd.7 on smithi076 2024-04-26T23:39:59.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:39:59 smithi136 ceph-mon[33050]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:40:00.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:00 smithi076 ceph-mon[32774]: overall HEALTH_OK 2024-04-26T23:40:00.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:00 smithi072 ceph-mon[28184]: overall HEALTH_OK 2024-04-26T23:40:00.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:00 smithi136 ceph-mon[33050]: overall HEALTH_OK 2024-04-26T23:40:01.759 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:01 smithi076 ceph-mon[32774]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:40:01.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:01 smithi072 ceph-mon[28184]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:40:01.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:01 smithi136 ceph-mon[33050]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:40:03.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:03 smithi076 ceph-mon[32774]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:40:03.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:03 smithi072 ceph-mon[28184]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:40:03.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:03 smithi136 ceph-mon[33050]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:40:04.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:04 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:04.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:04 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:40:04.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:04 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:04.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:04 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:04.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:04 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:40:04.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:04 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:04.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:04 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:04.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:04 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:40:04.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:04 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:05.783 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:05 smithi076 ceph-mon[32774]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:40:05.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:05 smithi072 ceph-mon[28184]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:40:05.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:05 smithi136 ceph-mon[33050]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:40:07.502 INFO:teuthology.orchestra.run.smithi076.stdout:Created osd(s) 7 on host 'smithi076' 2024-04-26T23:40:07.503 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:40:07.497+0000 7ff1a97fa700 1 -- 172.21.15.76:0/4189346159 <== mgr.14152 v2:172.21.15.72:6800/4281177709 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff1ac063860 con 0x7ff1940a3c40 2024-04-26T23:40:07.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:40:07.500+0000 7ff1b26c3700 1 -- 172.21.15.76:0/4189346159 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff1940a3c40 msgr2=0x7ff1940a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:40:07.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:40:07.500+0000 7ff1b26c3700 1 --2- 172.21.15.76:0/4189346159 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff1940a3c40 0x7ff1940a6100 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac06b8b0 tx=0x7ff19c00b410 comp rx=0 tx=0).stop 2024-04-26T23:40:07.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:40:07.500+0000 7ff1b26c3700 1 -- 172.21.15.76:0/4189346159 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff1ac1051d0 msgr2=0x7ff1ac1a0b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:40:07.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:40:07.500+0000 7ff1b26c3700 1 --2- 172.21.15.76:0/4189346159 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff1ac1051d0 0x7ff1ac1a0b60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff1a4009040 tx=0x7ff1a4004d50 comp rx=0 tx=0).stop 2024-04-26T23:40:07.505 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:40:07.500+0000 7ff1b26c3700 1 -- 172.21.15.76:0/4189346159 shutdown_connections 2024-04-26T23:40:07.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:40:07.500+0000 7ff1b26c3700 1 --2- 172.21.15.76:0/4189346159 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff1940a3c40 0x7ff1940a6100 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:40:07.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:40:07.500+0000 7ff1b26c3700 1 --2- 172.21.15.76:0/4189346159 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff1ac1051d0 0x7ff1ac1a0b60 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:40:07.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:40:07.500+0000 7ff1b26c3700 1 --2- 172.21.15.76:0/4189346159 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1ac1017b0 0x7ff1ac1995a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:40:07.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:40:07.500+0000 7ff1b26c3700 1 --2- 172.21.15.76:0/4189346159 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1ac104830 0x7ff1ac199ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:40:07.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:40:07.500+0000 7ff1b26c3700 1 -- 172.21.15.76:0/4189346159 >> 172.21.15.76:0/4189346159 conn(0x7ff1ac0fd0f0 msgr2=0x7ff1ac107ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:40:07.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:40:07.500+0000 7ff1b26c3700 1 -- 172.21.15.76:0/4189346159 shutdown_connections 2024-04-26T23:40:07.506 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:40:07.500+0000 7ff1b26c3700 1 -- 172.21.15.76:0/4189346159 wait complete. 2024-04-26T23:40:07.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:07 smithi076 ceph-mon[32774]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:40:07.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:07 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:07.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:07 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:07.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:07 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:40:07.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:07 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:40:07.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:07 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:07.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:07 smithi076 ceph-mon[32774]: from='osd.7 [v2:172.21.15.76:6816/375888575,v1:172.21.15.76:6817/375888575]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T23:40:07.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:07 smithi076 ceph-mon[32774]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T23:40:07.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:07 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:07.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:07 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:07.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:07 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:40:07.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:07 smithi072 ceph-mon[28184]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:40:07.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:07 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:07.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:07 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:07.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:07 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:40:07.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:07 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:40:07.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:07 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:07.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:07 smithi072 ceph-mon[28184]: from='osd.7 [v2:172.21.15.76:6816/375888575,v1:172.21.15.76:6817/375888575]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T23:40:07.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:07 smithi072 ceph-mon[28184]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T23:40:07.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:07 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:07.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:07 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:07.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:07 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:40:07.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:07 smithi136 ceph-mon[33050]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:40:07.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:07 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:07.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:07 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:07.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:07 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:40:07.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:07 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:40:07.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:07 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:07.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:07 smithi136 ceph-mon[33050]: from='osd.7 [v2:172.21.15.76:6816/375888575,v1:172.21.15.76:6817/375888575]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T23:40:07.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:07 smithi136 ceph-mon[33050]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T23:40:07.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:07 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:07.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:07 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:07.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:07 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:40:08.213 DEBUG:teuthology.orchestra.run.smithi076:osd.7> sudo journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@osd.7.service 2024-04-26T23:40:08.215 INFO:tasks.cephadm:Deploying osd.8 on smithi136 with /dev/vg_nvme/lv_2... 2024-04-26T23:40:08.215 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-26T23:40:08.462 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T23:40:08.911 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:08 smithi136 ceph-mon[33050]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-26T23:40:08.911 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:08 smithi136 ceph-mon[33050]: from='osd.7 [v2:172.21.15.76:6816/375888575,v1:172.21.15.76:6817/375888575]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-04-26T23:40:08.911 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:08 smithi136 ceph-mon[33050]: osdmap e49: 8 total, 7 up, 8 in 2024-04-26T23:40:08.911 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:08 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T23:40:08.911 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:08 smithi136 ceph-mon[33050]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-04-26T23:40:08.911 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:08 smithi136 ceph-mon[33050]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:40:08.930 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:08 smithi076 ceph-mon[32774]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-26T23:40:08.930 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:08 smithi076 ceph-mon[32774]: from='osd.7 [v2:172.21.15.76:6816/375888575,v1:172.21.15.76:6817/375888575]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-04-26T23:40:08.930 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:08 smithi076 ceph-mon[32774]: osdmap e49: 8 total, 7 up, 8 in 2024-04-26T23:40:08.930 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:08 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T23:40:08.930 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:08 smithi076 ceph-mon[32774]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-04-26T23:40:08.930 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:08 smithi076 ceph-mon[32774]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:40:09.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:08 smithi072 ceph-mon[28184]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-26T23:40:09.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:08 smithi072 ceph-mon[28184]: from='osd.7 [v2:172.21.15.76:6816/375888575,v1:172.21.15.76:6817/375888575]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-04-26T23:40:09.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:08 smithi072 ceph-mon[28184]: osdmap e49: 8 total, 7 up, 8 in 2024-04-26T23:40:09.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:08 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T23:40:09.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:08 smithi072 ceph-mon[28184]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-04-26T23:40:09.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:08 smithi072 ceph-mon[28184]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-04-26T23:40:09.419 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 26 23:40:09 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-26T23:40:09.138+0000 7fb9ae422700 -1 osd.7 0 waiting for initial osdmap 2024-04-26T23:40:09.419 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 26 23:40:09 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-26T23:40:09.147+0000 7fb9a620d700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T23:40:09.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:09 smithi136 ceph-mon[33050]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-04-26T23:40:09.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:09 smithi136 ceph-mon[33050]: osdmap e50: 8 total, 7 up, 8 in 2024-04-26T23:40:09.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:09 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T23:40:09.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:09 smithi136 ceph-mon[33050]: from='osd.7 ' entity='osd.7' 2024-04-26T23:40:09.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:09 smithi136 ceph-mon[33050]: osd.7 [v2:172.21.15.76:6816/375888575,v1:172.21.15.76:6817/375888575] boot 2024-04-26T23:40:09.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:09 smithi136 ceph-mon[33050]: osdmap e51: 8 total, 8 up, 8 in 2024-04-26T23:40:09.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:09 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T23:40:10.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:09 smithi072 ceph-mon[28184]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-04-26T23:40:10.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:09 smithi072 ceph-mon[28184]: osdmap e50: 8 total, 7 up, 8 in 2024-04-26T23:40:10.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:09 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T23:40:10.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:09 smithi072 ceph-mon[28184]: from='osd.7 ' entity='osd.7' 2024-04-26T23:40:10.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:09 smithi072 ceph-mon[28184]: osd.7 [v2:172.21.15.76:6816/375888575,v1:172.21.15.76:6817/375888575] boot 2024-04-26T23:40:10.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:09 smithi072 ceph-mon[28184]: osdmap e51: 8 total, 8 up, 8 in 2024-04-26T23:40:10.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:09 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T23:40:10.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:09 smithi076 ceph-mon[32774]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-04-26T23:40:10.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:09 smithi076 ceph-mon[32774]: osdmap e50: 8 total, 7 up, 8 in 2024-04-26T23:40:10.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:09 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T23:40:10.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:09 smithi076 ceph-mon[32774]: from='osd.7 ' entity='osd.7' 2024-04-26T23:40:10.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:09 smithi076 ceph-mon[32774]: osd.7 [v2:172.21.15.76:6816/375888575,v1:172.21.15.76:6817/375888575] boot 2024-04-26T23:40:10.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:09 smithi076 ceph-mon[32774]: osdmap e51: 8 total, 8 up, 8 in 2024-04-26T23:40:10.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:09 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T23:40:10.393 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-26T23:40:10.393 INFO:teuthology.orchestra.run.smithi136.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-26T23:40:10.393 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10+0 records in 2024-04-26T23:40:10.393 INFO:teuthology.orchestra.run.smithi136.stderr:10+0 records out 2024-04-26T23:40:10.393 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00947242 s, 1.1 GB/s 2024-04-26T23:40:10.393 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping successful for: 2024-04-26T23:40:11.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:10 smithi072 ceph-mon[28184]: purged_snaps scrub starts 2024-04-26T23:40:11.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:10 smithi072 ceph-mon[28184]: purged_snaps scrub ok 2024-04-26T23:40:11.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:10 smithi072 ceph-mon[28184]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:40:11.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:10 smithi072 ceph-mon[28184]: osdmap e52: 8 total, 8 up, 8 in 2024-04-26T23:40:11.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:10 smithi136 ceph-mon[33050]: purged_snaps scrub starts 2024-04-26T23:40:11.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:10 smithi136 ceph-mon[33050]: purged_snaps scrub ok 2024-04-26T23:40:11.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:10 smithi136 ceph-mon[33050]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:40:11.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:10 smithi136 ceph-mon[33050]: osdmap e52: 8 total, 8 up, 8 in 2024-04-26T23:40:11.101 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:10 smithi076 ceph-mon[32774]: purged_snaps scrub starts 2024-04-26T23:40:11.102 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:10 smithi076 ceph-mon[32774]: purged_snaps scrub ok 2024-04-26T23:40:11.102 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:10 smithi076 ceph-mon[32774]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:40:11.102 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:10 smithi076 ceph-mon[32774]: osdmap e52: 8 total, 8 up, 8 in 2024-04-26T23:40:11.108 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi136:vg_nvme/lv_2 2024-04-26T23:40:11.352 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T23:40:12.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:12 smithi072 ceph-mon[28184]: Detected new or changed devices on smithi076 2024-04-26T23:40:12.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:12 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:12.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:12 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:12.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:12 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:40:12.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:12 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:40:12.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:12 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:40:12.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:12 smithi072 ceph-mon[28184]: Adjusting osd_memory_target on smithi076 to 5696M 2024-04-26T23:40:12.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:12 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:12.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:12 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:40:12.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:12 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:40:12.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:12 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:12.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:12 smithi072 ceph-mon[28184]: osdmap e53: 8 total, 8 up, 8 in 2024-04-26T23:40:12.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:12 smithi136 ceph-mon[33050]: Detected new or changed devices on smithi076 2024-04-26T23:40:12.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:12 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:12.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:12 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:12.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:12 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:40:12.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:12 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:40:12.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:12 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:40:12.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:12 smithi136 ceph-mon[33050]: Adjusting osd_memory_target on smithi076 to 5696M 2024-04-26T23:40:12.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:12 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:12.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:12 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:40:12.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:12 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:40:12.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:12 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:12.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:12 smithi136 ceph-mon[33050]: osdmap e53: 8 total, 8 up, 8 in 2024-04-26T23:40:12.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:12 smithi076 ceph-mon[32774]: Detected new or changed devices on smithi076 2024-04-26T23:40:12.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:12 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:12.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:12 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:12.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:12 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:40:12.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:12 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:40:12.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:12 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:40:12.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:12 smithi076 ceph-mon[32774]: Adjusting osd_memory_target on smithi076 to 5696M 2024-04-26T23:40:12.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:12 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:12.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:12 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:40:12.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:12 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:40:12.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:12 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:12.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:12 smithi076 ceph-mon[32774]: osdmap e53: 8 total, 8 up, 8 in 2024-04-26T23:40:13.014 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:13.009+0000 7fa7f8356700 1 -- 172.21.15.136:0/2179246618 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa7f0101740 msgr2=0x7fa7f0101b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:40:13.014 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:13.009+0000 7fa7f8356700 1 --2- 172.21.15.136:0/2179246618 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa7f0101740 0x7fa7f0101b60 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec009990 tx=0x7fa7ec009ca0 comp rx=0 tx=0).stop 2024-04-26T23:40:13.014 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:13.009+0000 7fa7f8356700 1 -- 172.21.15.136:0/2179246618 shutdown_connections 2024-04-26T23:40:13.014 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:13.009+0000 7fa7f8356700 1 --2- 172.21.15.136:0/2179246618 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa7f01050c0 0x7fa7f010b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:40:13.014 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:13.009+0000 7fa7f8356700 1 --2- 172.21.15.136:0/2179246618 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa7f0101740 0x7fa7f0101b60 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:40:13.014 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:13.009+0000 7fa7f8356700 1 --2- 172.21.15.136:0/2179246618 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa7f0104720 0x7fa7f0104b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:40:13.014 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:13.009+0000 7fa7f8356700 1 -- 172.21.15.136:0/2179246618 >> 172.21.15.136:0/2179246618 conn(0x7fa7f00fd080 msgr2=0x7fa7f00ff4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:40:13.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:13.009+0000 7fa7f8356700 1 -- 172.21.15.136:0/2179246618 shutdown_connections 2024-04-26T23:40:13.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:13.009+0000 7fa7f8356700 1 -- 172.21.15.136:0/2179246618 wait complete. 2024-04-26T23:40:13.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:13.010+0000 7fa7f8356700 1 Processor -- start 2024-04-26T23:40:13.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:13.010+0000 7fa7f8356700 1 -- start start 2024-04-26T23:40:13.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:13.010+0000 7fa7f8356700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa7f0101740 0x7fa7f0104030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:40:13.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:13.010+0000 7fa7f8356700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa7f0104720 0x7fa7f01026d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:40:13.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:13.010+0000 7fa7f8356700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa7f01050c0 0x7fa7f0102c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:40:13.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:13.010+0000 7fa7f8356700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fa7f010ef70 con 0x7fa7f0104720 2024-04-26T23:40:13.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:13.010+0000 7fa7f8356700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa7f010edf0 con 0x7fa7f01050c0 2024-04-26T23:40:13.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:13.010+0000 7fa7f8356700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fa7f010f0f0 con 0x7fa7f0101740 2024-04-26T23:40:13.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:13.010+0000 7fa7f60f2700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa7f0101740 0x7fa7f0104030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:40:13.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:13.010+0000 7fa7f60f2700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa7f0101740 0x7fa7f0104030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:55928/0 (socket says 172.21.15.136:55928) 2024-04-26T23:40:13.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:13.010+0000 7fa7f60f2700 1 -- 172.21.15.136:0/3060412991 learned_addr learned my addr 172.21.15.136:0/3060412991 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-26T23:40:13.015 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:13.010+0000 7fa7f60f2700 1 -- 172.21.15.136:0/3060412991 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa7f01050c0 msgr2=0x7fa7f0102c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:40:13.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:13.010+0000 7fa7f60f2700 1 --2- 172.21.15.136:0/3060412991 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa7f01050c0 0x7fa7f0102c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:40:13.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:13.010+0000 7fa7f60f2700 1 -- 172.21.15.136:0/3060412991 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa7f0104720 msgr2=0x7fa7f01026d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:40:13.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:13.010+0000 7fa7f60f2700 1 --2- 172.21.15.136:0/3060412991 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa7f0104720 0x7fa7f01026d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:40:13.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:13.010+0000 7fa7f60f2700 1 -- 172.21.15.136:0/3060412991 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7f01034e0 con 0x7fa7f0101740 2024-04-26T23:40:13.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:13.010+0000 7fa7f60f2700 1 --2- 172.21.15.136:0/3060412991 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa7f0101740 0x7fa7f0104030 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec0036f0 tx=0x7fa7ec005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:40:13.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:13.011+0000 7fa7e77fe700 1 -- 172.21.15.136:0/3060412991 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7ec027070 con 0x7fa7f0101740 2024-04-26T23:40:13.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:13.011+0000 7fa7f8356700 1 -- 172.21.15.136:0/3060412991 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7f01a3770 con 0x7fa7f0101740 2024-04-26T23:40:13.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:13.011+0000 7fa7e77fe700 1 -- 172.21.15.136:0/3060412991 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7ec0041c0 con 0x7fa7f0101740 2024-04-26T23:40:13.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:13.011+0000 7fa7e77fe700 1 -- 172.21.15.136:0/3060412991 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7ec022d90 con 0x7fa7f0101740 2024-04-26T23:40:13.016 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:13.011+0000 7fa7f8356700 1 -- 172.21.15.136:0/3060412991 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7f01a3ca0 con 0x7fa7f0101740 2024-04-26T23:40:13.017 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:13.012+0000 7fa7f8356700 1 -- 172.21.15.136:0/3060412991 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7f006a110 con 0x7fa7f0101740 2024-04-26T23:40:13.017 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:13.013+0000 7fa7e77fe700 1 -- 172.21.15.136:0/3060412991 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa7ec02a070 con 0x7fa7f0101740 2024-04-26T23:40:13.017 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:13.014+0000 7fa7e77fe700 1 --2- 172.21.15.136:0/3060412991 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fa7dc0a3c40 0x7fa7dc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:40:13.017 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:13.014+0000 7fa7e77fe700 1 -- 172.21.15.136:0/3060412991 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5397+0+0 (secure 0 0 0) 0x7fa7ec0d0ff0 con 0x7fa7f0101740 2024-04-26T23:40:13.018 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:13.015+0000 7fa7f58f1700 1 --2- 172.21.15.136:0/3060412991 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fa7dc0a3c40 0x7fa7dc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:40:13.019 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:13.016+0000 7fa7f58f1700 1 --2- 172.21.15.136:0/3060412991 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fa7dc0a3c40 0x7fa7dc0a6100 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa7e0005950 tx=0x7fa7e00058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:40:13.021 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:13.018+0000 7fa7e77fe700 1 -- 172.21.15.136:0/3060412991 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fa7ec031020 con 0x7fa7f0101740 2024-04-26T23:40:13.229 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:13.226+0000 7fa7f8356700 1 -- 172.21.15.136:0/3060412991 --> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fa7f0063920 con 0x7fa7dc0a3c40 2024-04-26T23:40:13.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:13 smithi072 ceph-mon[28184]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:40:13.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:13 smithi072 ceph-mon[28184]: osdmap e54: 8 total, 8 up, 8 in 2024-04-26T23:40:13.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:13 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:40:13.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:13 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:40:13.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:13 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:40:13.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:13 smithi136 ceph-mon[33050]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:40:13.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:13 smithi136 ceph-mon[33050]: osdmap e54: 8 total, 8 up, 8 in 2024-04-26T23:40:13.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:13 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:40:13.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:13 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:40:13.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:13 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:40:13.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:13 smithi076 ceph-mon[32774]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:40:13.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:13 smithi076 ceph-mon[32774]: osdmap e54: 8 total, 8 up, 8 in 2024-04-26T23:40:13.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:13 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:40:13.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:13 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:40:13.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:13 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:40:14.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:14 smithi072 ceph-mon[28184]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:40:14.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:14 smithi072 ceph-mon[28184]: osdmap e55: 8 total, 8 up, 8 in 2024-04-26T23:40:14.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:14 smithi136 ceph-mon[33050]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:40:14.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:14 smithi136 ceph-mon[33050]: osdmap e55: 8 total, 8 up, 8 in 2024-04-26T23:40:14.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:14 smithi076 ceph-mon[32774]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:40:14.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:14 smithi076 ceph-mon[32774]: osdmap e55: 8 total, 8 up, 8 in 2024-04-26T23:40:15.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:15 smithi072 ceph-mon[28184]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:40:15.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:15 smithi136 ceph-mon[33050]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:40:15.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:15 smithi076 ceph-mon[32774]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:40:16.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:16 smithi072 ceph-mon[28184]: from='client.? 172.21.15.136:0/1206959301' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f404062-97e7-4ccb-a582-76dc1ffae548"}]: dispatch 2024-04-26T23:40:16.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:16 smithi072 ceph-mon[28184]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f404062-97e7-4ccb-a582-76dc1ffae548"}]: dispatch 2024-04-26T23:40:16.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:16 smithi072 ceph-mon[28184]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f404062-97e7-4ccb-a582-76dc1ffae548"}]': finished 2024-04-26T23:40:16.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:16 smithi072 ceph-mon[28184]: osdmap e56: 9 total, 8 up, 9 in 2024-04-26T23:40:16.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:16 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T23:40:16.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:16 smithi136 ceph-mon[33050]: from='client.? 172.21.15.136:0/1206959301' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f404062-97e7-4ccb-a582-76dc1ffae548"}]: dispatch 2024-04-26T23:40:16.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:16 smithi136 ceph-mon[33050]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f404062-97e7-4ccb-a582-76dc1ffae548"}]: dispatch 2024-04-26T23:40:16.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:16 smithi136 ceph-mon[33050]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f404062-97e7-4ccb-a582-76dc1ffae548"}]': finished 2024-04-26T23:40:16.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:16 smithi136 ceph-mon[33050]: osdmap e56: 9 total, 8 up, 9 in 2024-04-26T23:40:16.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:16 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T23:40:16.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:16 smithi076 ceph-mon[32774]: from='client.? 172.21.15.136:0/1206959301' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f404062-97e7-4ccb-a582-76dc1ffae548"}]: dispatch 2024-04-26T23:40:16.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:16 smithi076 ceph-mon[32774]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f404062-97e7-4ccb-a582-76dc1ffae548"}]: dispatch 2024-04-26T23:40:16.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:16 smithi076 ceph-mon[32774]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f404062-97e7-4ccb-a582-76dc1ffae548"}]': finished 2024-04-26T23:40:16.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:16 smithi076 ceph-mon[32774]: osdmap e56: 9 total, 8 up, 9 in 2024-04-26T23:40:16.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:16 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T23:40:17.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:17 smithi072 ceph-mon[28184]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:40:17.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:17 smithi072 ceph-mon[28184]: from='client.? 172.21.15.136:0/200985490' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:40:17.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:17 smithi136 ceph-mon[33050]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:40:17.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:17 smithi136 ceph-mon[33050]: from='client.? 172.21.15.136:0/200985490' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:40:17.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:17 smithi076 ceph-mon[32774]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:40:17.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:17 smithi076 ceph-mon[32774]: from='client.? 172.21.15.136:0/200985490' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:40:19.773 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:19 smithi136 ceph-mon[33050]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:40:19.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:19 smithi072 ceph-mon[28184]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:40:19.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:19 smithi076 ceph-mon[32774]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:40:21.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:21 smithi072 ceph-mon[28184]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:40:21.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:21 smithi136 ceph-mon[33050]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:40:21.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:21 smithi076 ceph-mon[32774]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:40:23.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:23 smithi072 ceph-mon[28184]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:40:23.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:23 smithi136 ceph-mon[33050]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:40:23.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:23 smithi076 ceph-mon[32774]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:40:24.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:24 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-26T23:40:24.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:24 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:40:24.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:24 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-26T23:40:24.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:24 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:40:24.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:24 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-26T23:40:24.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:24 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:40:25.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:25 smithi072 ceph-mon[28184]: Deploying daemon osd.8 on smithi136 2024-04-26T23:40:25.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:25 smithi072 ceph-mon[28184]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:40:25.795 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:25 smithi136 ceph-mon[33050]: Deploying daemon osd.8 on smithi136 2024-04-26T23:40:25.796 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:25 smithi136 ceph-mon[33050]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:40:25.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:25 smithi076 ceph-mon[32774]: Deploying daemon osd.8 on smithi136 2024-04-26T23:40:25.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:25 smithi076 ceph-mon[32774]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:40:27.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:27 smithi072 ceph-mon[28184]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:40:27.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:27 smithi136 ceph-mon[33050]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:40:27.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:27 smithi076 ceph-mon[32774]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:40:29.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:29 smithi072 ceph-mon[28184]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:40:29.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:29 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:40:29.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:29 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:29.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:29 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:29.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:29 smithi136 ceph-mon[33050]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:40:29.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:29 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:40:29.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:29 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:29.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:29 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:29.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:29 smithi076 ceph-mon[32774]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:40:29.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:29 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:40:29.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:29 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:29.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:29 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:31.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:31 smithi076 ceph-mon[32774]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:40:31.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:31 smithi136 ceph-mon[33050]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:40:32.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:31 smithi072 ceph-mon[28184]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:40:32.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:32.860+0000 7fa7e77fe700 1 -- 172.21.15.136:0/3060412991 <== mgr.14152 v2:172.21.15.72:6800/4281177709 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa7f0063920 con 0x7fa7dc0a3c40 2024-04-26T23:40:32.865 INFO:teuthology.orchestra.run.smithi136.stdout:Created osd(s) 8 on host 'smithi136' 2024-04-26T23:40:32.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:32.863+0000 7fa7f8356700 1 -- 172.21.15.136:0/3060412991 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fa7dc0a3c40 msgr2=0x7fa7dc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:40:32.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:32.863+0000 7fa7f8356700 1 --2- 172.21.15.136:0/3060412991 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fa7dc0a3c40 0x7fa7dc0a6100 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa7e0005950 tx=0x7fa7e00058e0 comp rx=0 tx=0).stop 2024-04-26T23:40:32.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:32.863+0000 7fa7f8356700 1 -- 172.21.15.136:0/3060412991 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa7f0101740 msgr2=0x7fa7f0104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:40:32.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:32.863+0000 7fa7f8356700 1 --2- 172.21.15.136:0/3060412991 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa7f0101740 0x7fa7f0104030 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec0036f0 tx=0x7fa7ec005e80 comp rx=0 tx=0).stop 2024-04-26T23:40:32.867 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:32.863+0000 7fa7f8356700 1 -- 172.21.15.136:0/3060412991 shutdown_connections 2024-04-26T23:40:32.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:32.863+0000 7fa7f8356700 1 --2- 172.21.15.136:0/3060412991 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fa7dc0a3c40 0x7fa7dc0a6100 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:40:32.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:32.863+0000 7fa7f8356700 1 --2- 172.21.15.136:0/3060412991 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa7f01050c0 0x7fa7f0102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:40:32.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:32.863+0000 7fa7f8356700 1 --2- 172.21.15.136:0/3060412991 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa7f0101740 0x7fa7f0104030 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:40:32.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:32.863+0000 7fa7f8356700 1 --2- 172.21.15.136:0/3060412991 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa7f0104720 0x7fa7f01026d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:40:32.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:32.863+0000 7fa7f8356700 1 -- 172.21.15.136:0/3060412991 >> 172.21.15.136:0/3060412991 conn(0x7fa7f00fd080 msgr2=0x7fa7f0107b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:40:32.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:32.864+0000 7fa7f8356700 1 -- 172.21.15.136:0/3060412991 shutdown_connections 2024-04-26T23:40:32.868 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:40:32.864+0000 7fa7f8356700 1 -- 172.21.15.136:0/3060412991 wait complete. 2024-04-26T23:40:33.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:33 smithi136 ceph-mon[33050]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:40:33.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:33 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:33.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:33 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:33.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:33 smithi136 ceph-mon[33050]: from='osd.8 [v2:172.21.15.136:6816/3446774052,v1:172.21.15.136:6817/3446774052]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T23:40:33.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:33 smithi136 ceph-mon[33050]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T23:40:33.835 DEBUG:teuthology.orchestra.run.smithi136:osd.8> sudo journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@osd.8.service 2024-04-26T23:40:33.837 INFO:tasks.cephadm:Deploying osd.9 on smithi072 with /dev/vg_nvme/lv_1... 2024-04-26T23:40:33.837 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-26T23:40:33.916 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:33 smithi072 ceph-mon[28184]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:40:33.916 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:33 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:33.916 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:33 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:33.916 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:33 smithi072 ceph-mon[28184]: from='osd.8 [v2:172.21.15.136:6816/3446774052,v1:172.21.15.136:6817/3446774052]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T23:40:33.916 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:33 smithi072 ceph-mon[28184]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T23:40:33.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:33 smithi076 ceph-mon[32774]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:40:33.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:33 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:33.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:33 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:33.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:33 smithi076 ceph-mon[32774]: from='osd.8 [v2:172.21.15.136:6816/3446774052,v1:172.21.15.136:6817/3446774052]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T23:40:33.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:33 smithi076 ceph-mon[32774]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T23:40:34.282 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:40:34.937 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:34 smithi136 ceph-mon[33050]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-26T23:40:34.937 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:34 smithi136 ceph-mon[33050]: from='osd.8 [v2:172.21.15.136:6816/3446774052,v1:172.21.15.136:6817/3446774052]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-04-26T23:40:34.937 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:34 smithi136 ceph-mon[33050]: osdmap e57: 9 total, 8 up, 9 in 2024-04-26T23:40:34.937 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:34 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T23:40:34.937 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:34 smithi136 ceph-mon[33050]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-04-26T23:40:34.937 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:34 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:34.937 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:34 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:34.937 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:34 smithi136 ceph-mon[33050]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:40:34.937 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 26 23:40:34 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-26T23:40:34.831+0000 7f81072b9700 -1 osd.8 0 waiting for initial osdmap 2024-04-26T23:40:34.938 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 26 23:40:34 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-26T23:40:34.845+0000 7f81030ac700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T23:40:35.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:34 smithi072 ceph-mon[28184]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-26T23:40:35.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:34 smithi072 ceph-mon[28184]: from='osd.8 [v2:172.21.15.136:6816/3446774052,v1:172.21.15.136:6817/3446774052]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-04-26T23:40:35.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:34 smithi072 ceph-mon[28184]: osdmap e57: 9 total, 8 up, 9 in 2024-04-26T23:40:35.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:34 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T23:40:35.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:34 smithi072 ceph-mon[28184]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-04-26T23:40:35.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:34 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:35.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:34 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:35.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:34 smithi072 ceph-mon[28184]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:40:35.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:34 smithi076 ceph-mon[32774]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-26T23:40:35.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:34 smithi076 ceph-mon[32774]: from='osd.8 [v2:172.21.15.136:6816/3446774052,v1:172.21.15.136:6817/3446774052]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-04-26T23:40:35.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:34 smithi076 ceph-mon[32774]: osdmap e57: 9 total, 8 up, 9 in 2024-04-26T23:40:35.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:34 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T23:40:35.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:34 smithi076 ceph-mon[32774]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-04-26T23:40:35.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:34 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:35.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:34 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:35.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:34 smithi076 ceph-mon[32774]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T23:40:36.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:35 smithi072 ceph-mon[28184]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-04-26T23:40:36.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:35 smithi072 ceph-mon[28184]: osdmap e58: 9 total, 8 up, 9 in 2024-04-26T23:40:36.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:35 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T23:40:36.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:35 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T23:40:36.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:35 smithi072 ceph-mon[28184]: osd.8 [v2:172.21.15.136:6816/3446774052,v1:172.21.15.136:6817/3446774052] boot 2024-04-26T23:40:36.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:35 smithi072 ceph-mon[28184]: osdmap e59: 9 total, 9 up, 9 in 2024-04-26T23:40:36.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:35 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T23:40:36.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:35 smithi076 ceph-mon[32774]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-04-26T23:40:36.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:35 smithi076 ceph-mon[32774]: osdmap e58: 9 total, 8 up, 9 in 2024-04-26T23:40:36.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:35 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T23:40:36.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:35 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T23:40:36.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:35 smithi076 ceph-mon[32774]: osd.8 [v2:172.21.15.136:6816/3446774052,v1:172.21.15.136:6817/3446774052] boot 2024-04-26T23:40:36.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:35 smithi076 ceph-mon[32774]: osdmap e59: 9 total, 9 up, 9 in 2024-04-26T23:40:36.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:35 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T23:40:36.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:35 smithi136 ceph-mon[33050]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-04-26T23:40:36.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:35 smithi136 ceph-mon[33050]: osdmap e58: 9 total, 8 up, 9 in 2024-04-26T23:40:36.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:35 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T23:40:36.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:35 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T23:40:36.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:35 smithi136 ceph-mon[33050]: osd.8 [v2:172.21.15.136:6816/3446774052,v1:172.21.15.136:6817/3446774052] boot 2024-04-26T23:40:36.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:35 smithi136 ceph-mon[33050]: osdmap e59: 9 total, 9 up, 9 in 2024-04-26T23:40:36.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:35 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T23:40:36.982 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:36 smithi136 ceph-mon[33050]: purged_snaps scrub starts 2024-04-26T23:40:36.982 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:36 smithi136 ceph-mon[33050]: purged_snaps scrub ok 2024-04-26T23:40:36.982 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:36 smithi136 ceph-mon[33050]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:40:36.982 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:36 smithi136 ceph-mon[33050]: osdmap e60: 9 total, 9 up, 9 in 2024-04-26T23:40:37.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:36 smithi076 ceph-mon[32774]: purged_snaps scrub starts 2024-04-26T23:40:37.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:36 smithi076 ceph-mon[32774]: purged_snaps scrub ok 2024-04-26T23:40:37.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:36 smithi076 ceph-mon[32774]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:40:37.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:36 smithi076 ceph-mon[32774]: osdmap e60: 9 total, 9 up, 9 in 2024-04-26T23:40:37.267 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:36 smithi072 ceph-mon[28184]: purged_snaps scrub starts 2024-04-26T23:40:37.267 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:36 smithi072 ceph-mon[28184]: purged_snaps scrub ok 2024-04-26T23:40:37.267 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:36 smithi072 ceph-mon[28184]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:40:37.267 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:36 smithi072 ceph-mon[28184]: osdmap e60: 9 total, 9 up, 9 in 2024-04-26T23:40:37.268 INFO:teuthology.orchestra.run.smithi072.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-26T23:40:37.268 INFO:teuthology.orchestra.run.smithi072.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-26T23:40:37.268 INFO:teuthology.orchestra.run.smithi072.stderr: stderr: 10+0 records in 2024-04-26T23:40:37.268 INFO:teuthology.orchestra.run.smithi072.stderr:10+0 records out 2024-04-26T23:40:37.268 INFO:teuthology.orchestra.run.smithi072.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0322265 s, 325 MB/s 2024-04-26T23:40:37.268 INFO:teuthology.orchestra.run.smithi072.stderr:--> Zapping successful for: 2024-04-26T23:40:38.265 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi072:vg_nvme/lv_1 2024-04-26T23:40:38.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:38 smithi072 ceph-mon[28184]: Detected new or changed devices on smithi136 2024-04-26T23:40:38.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:38 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:38.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:38 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:38.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:38 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:40:38.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:38 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:40:38.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:38 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:40:38.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:38 smithi072 ceph-mon[28184]: Adjusting osd_memory_target on smithi136 to 5696M 2024-04-26T23:40:38.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:38 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:38.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:38 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:40:38.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:38 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:40:38.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:38 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:38.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:38 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:40:38.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:38 smithi072 ceph-mon[28184]: osdmap e61: 9 total, 9 up, 9 in 2024-04-26T23:40:38.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:38 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:40:38.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:38 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:40:38.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:38 smithi076 ceph-mon[32774]: Detected new or changed devices on smithi136 2024-04-26T23:40:38.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:38 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:38.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:38 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:38.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:38 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:40:38.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:38 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:40:38.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:38 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:40:38.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:38 smithi076 ceph-mon[32774]: Adjusting osd_memory_target on smithi136 to 5696M 2024-04-26T23:40:38.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:38 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:38.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:38 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:40:38.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:38 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:40:38.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:38 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:38.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:38 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:40:38.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:38 smithi076 ceph-mon[32774]: osdmap e61: 9 total, 9 up, 9 in 2024-04-26T23:40:38.680 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:38 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:40:38.680 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:38 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:40:38.703 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:40:38.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:38 smithi136 ceph-mon[33050]: Detected new or changed devices on smithi136 2024-04-26T23:40:38.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:38 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:38.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:38 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:38.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:38 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:40:38.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:38 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:40:38.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:38 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:40:38.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:38 smithi136 ceph-mon[33050]: Adjusting osd_memory_target on smithi136 to 5696M 2024-04-26T23:40:38.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:38 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:38.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:38 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:40:38.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:38 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:40:38.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:38 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:38.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:38 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:40:38.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:38 smithi136 ceph-mon[33050]: osdmap e61: 9 total, 9 up, 9 in 2024-04-26T23:40:38.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:38 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:40:38.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:38 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:40:39.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:39 smithi072 ceph-mon[28184]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:40:39.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:39 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:39.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:39 smithi072 ceph-mon[28184]: osdmap e62: 9 total, 9 up, 9 in 2024-04-26T23:40:39.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:39 smithi076 ceph-mon[32774]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:40:39.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:39 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:39.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:39 smithi076 ceph-mon[32774]: osdmap e62: 9 total, 9 up, 9 in 2024-04-26T23:40:39.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:39 smithi136 ceph-mon[33050]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:40:39.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:39 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:40:39.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:39 smithi136 ceph-mon[33050]: osdmap e62: 9 total, 9 up, 9 in 2024-04-26T23:40:41.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:40:41.060+0000 7f5421200700 1 -- 172.21.15.72:0/2844022940 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f541c10f980 msgr2=0x7f541c111d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:40:41.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:40:41.060+0000 7f5421200700 1 --2- 172.21.15.72:0/2844022940 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f541c10f980 0x7f541c111d70 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f541400b0a0 tx=0x7f541400b3b0 comp rx=0 tx=0).stop 2024-04-26T23:40:41.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:40:41.060+0000 7f5421200700 1 -- 172.21.15.72:0/2844022940 shutdown_connections 2024-04-26T23:40:41.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:40:41.060+0000 7f5421200700 1 --2- 172.21.15.72:0/2844022940 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f541c10f980 0x7f541c111d70 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:40:41.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:40:41.060+0000 7f5421200700 1 --2- 172.21.15.72:0/2844022940 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f541c077450 0x7f541c0758b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:40:41.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:40:41.060+0000 7f5421200700 1 --2- 172.21.15.72:0/2844022940 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f541c075e80 0x7f541c076300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:40:41.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:40:41.060+0000 7f5421200700 1 -- 172.21.15.72:0/2844022940 >> 172.21.15.72:0/2844022940 conn(0x7f541c0fe960 msgr2=0x7f541c100d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:40:41.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:40:41.060+0000 7f5421200700 1 -- 172.21.15.72:0/2844022940 shutdown_connections 2024-04-26T23:40:41.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:40:41.060+0000 7f5421200700 1 -- 172.21.15.72:0/2844022940 wait complete. 2024-04-26T23:40:41.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:40:41.062+0000 7f5421200700 1 Processor -- start 2024-04-26T23:40:41.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:40:41.062+0000 7f5421200700 1 -- start start 2024-04-26T23:40:41.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:40:41.062+0000 7f5421200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f541c075e80 0x7f541c19fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:40:41.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:40:41.062+0000 7f5421200700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f541c077450 0x7f541c19ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:40:41.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:40:41.062+0000 7f5421200700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f541c10f980 0x7f541c1a6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:40:41.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:40:41.062+0000 7f5421200700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f541c115430 con 0x7f541c10f980 2024-04-26T23:40:41.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:40:41.062+0000 7f5421200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f541c1152b0 con 0x7f541c075e80 2024-04-26T23:40:41.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:40:41.062+0000 7f5421200700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f541c1155b0 con 0x7f541c077450 2024-04-26T23:40:41.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:40:41.063+0000 7f541aa00700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f541c077450 0x7f541c19ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:40:41.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:40:41.063+0000 7f541be00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f541c10f980 0x7f541c1a6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:40:41.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:40:41.063+0000 7f541b400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f541c075e80 0x7f541c19fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:40:41.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:40:41.063+0000 7f541aa00700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f541c077450 0x7f541c19ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.72:45410/0 (socket says 172.21.15.72:45410) 2024-04-26T23:40:41.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:40:41.063+0000 7f541aa00700 1 -- 172.21.15.72:0/606307190 learned_addr learned my addr 172.21.15.72:0/606307190 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:40:41.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:40:41.063+0000 7f541aa00700 1 -- 172.21.15.72:0/606307190 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f541c075e80 msgr2=0x7f541c19fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:40:41.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:40:41.063+0000 7f541aa00700 1 --2- 172.21.15.72:0/606307190 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f541c075e80 0x7f541c19fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:40:41.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:40:41.063+0000 7f541aa00700 1 -- 172.21.15.72:0/606307190 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f541c10f980 msgr2=0x7f541c1a6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:40:41.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:40:41.063+0000 7f541aa00700 1 --2- 172.21.15.72:0/606307190 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f541c10f980 0x7f541c1a6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:40:41.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:40:41.063+0000 7f541aa00700 1 -- 172.21.15.72:0/606307190 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f541c1a7690 con 0x7f541c077450 2024-04-26T23:40:41.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:40:41.063+0000 7f541be00700 1 --2- 172.21.15.72:0/606307190 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f541c10f980 0x7f541c1a6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:40:41.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:40:41.064+0000 7f541aa00700 1 --2- 172.21.15.72:0/606307190 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f541c077450 0x7f541c19ff70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f540800e9a0 tx=0x7f540800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:40:41.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:40:41.064+0000 7f5413e00700 1 -- 172.21.15.72:0/606307190 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5408004220 con 0x7f541c077450 2024-04-26T23:40:41.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:40:41.064+0000 7f5413e00700 1 -- 172.21.15.72:0/606307190 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54080043c0 con 0x7f541c077450 2024-04-26T23:40:41.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:40:41.064+0000 7f5421200700 1 -- 172.21.15.72:0/606307190 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f541c1a7980 con 0x7f541c077450 2024-04-26T23:40:41.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:40:41.064+0000 7f5413e00700 1 -- 172.21.15.72:0/606307190 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5408003cc0 con 0x7f541c077450 2024-04-26T23:40:41.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:40:41.064+0000 7f5421200700 1 -- 172.21.15.72:0/606307190 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f541c1a7ec0 con 0x7f541c077450 2024-04-26T23:40:41.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:40:41.065+0000 7f5421200700 1 -- 172.21.15.72:0/606307190 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f541c06a110 con 0x7f541c077450 2024-04-26T23:40:41.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:40:41.069+0000 7f5413e00700 1 -- 172.21.15.72:0/606307190 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f540800c810 con 0x7f541c077450 2024-04-26T23:40:41.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:40:41.069+0000 7f5413e00700 1 --2- 172.21.15.72:0/606307190 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f54000a3c40 0x7f54000a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:40:41.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:40:41.069+0000 7f5413e00700 1 -- 172.21.15.72:0/606307190 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5829+0+0 (secure 0 0 0) 0x7f54080cfcc0 con 0x7f541c077450 2024-04-26T23:40:41.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:40:41.069+0000 7f541b400700 1 --2- 172.21.15.72:0/606307190 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f54000a3c40 0x7f54000a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:40:41.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:40:41.070+0000 7f541b400700 1 --2- 172.21.15.72:0/606307190 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f54000a3c40 0x7f54000a6100 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f541c077830 tx=0x7f540c0073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:40:41.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:40:41.073+0000 7f5413e00700 1 -- 172.21.15.72:0/606307190 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f54080d6080 con 0x7f541c077450 2024-04-26T23:40:41.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:40 smithi076 ceph-mon[32774]: osdmap e63: 9 total, 9 up, 9 in 2024-04-26T23:40:41.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:40 smithi076 ceph-mon[32774]: pgmap v224: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T23:40:41.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:40 smithi072 ceph-mon[28184]: osdmap e63: 9 total, 9 up, 9 in 2024-04-26T23:40:41.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:40 smithi072 ceph-mon[28184]: pgmap v224: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T23:40:41.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:40 smithi136 ceph-mon[33050]: osdmap e63: 9 total, 9 up, 9 in 2024-04-26T23:40:41.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:40 smithi136 ceph-mon[33050]: pgmap v224: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T23:40:41.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:40:41.311+0000 7f5421200700 1 -- 172.21.15.72:0/606307190 --> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi072:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f541c063860 con 0x7f54000a3c40 2024-04-26T23:40:42.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:41 smithi076 ceph-mon[32774]: from='client.24343 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi072:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:40:42.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:41 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:40:42.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:41 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:40:42.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:41 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:40:42.202 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:41 smithi136 ceph-mon[33050]: from='client.24343 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi072:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:40:42.202 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:41 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:40:42.202 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:41 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:40:42.202 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:41 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:40:42.249 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:41 smithi072 ceph-mon[28184]: from='client.24343 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi072:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:40:42.250 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:41 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:40:42.250 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:41 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:40:42.250 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:41 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:40:43.127 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:42 smithi072 ceph-mon[28184]: pgmap v225: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-04-26T23:40:43.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:42 smithi076 ceph-mon[32774]: pgmap v225: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-04-26T23:40:43.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:42 smithi136 ceph-mon[33050]: pgmap v225: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-04-26T23:40:45.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:45 smithi076 ceph-mon[32774]: pgmap v226: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-04-26T23:40:45.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:45 smithi072 ceph-mon[28184]: pgmap v226: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-04-26T23:40:45.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:45 smithi136 ceph-mon[33050]: pgmap v226: 1 pgs: 1 peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-04-26T23:40:46.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:46 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/3632649404' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2989ca68-3164-4667-9bcf-230dcbb7b8f7"}]: dispatch 2024-04-26T23:40:46.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:46 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/3632649404' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2989ca68-3164-4667-9bcf-230dcbb7b8f7"}]': finished 2024-04-26T23:40:46.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:46 smithi076 ceph-mon[32774]: osdmap e64: 10 total, 9 up, 10 in 2024-04-26T23:40:46.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:46 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T23:40:46.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:46 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/3632649404' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2989ca68-3164-4667-9bcf-230dcbb7b8f7"}]: dispatch 2024-04-26T23:40:46.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:46 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/3632649404' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2989ca68-3164-4667-9bcf-230dcbb7b8f7"}]': finished 2024-04-26T23:40:46.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:46 smithi072 ceph-mon[28184]: osdmap e64: 10 total, 9 up, 10 in 2024-04-26T23:40:46.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:46 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T23:40:46.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:46 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/3632649404' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2989ca68-3164-4667-9bcf-230dcbb7b8f7"}]: dispatch 2024-04-26T23:40:46.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:46 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/3632649404' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2989ca68-3164-4667-9bcf-230dcbb7b8f7"}]': finished 2024-04-26T23:40:46.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:46 smithi136 ceph-mon[33050]: osdmap e64: 10 total, 9 up, 10 in 2024-04-26T23:40:46.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:46 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T23:40:47.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:47 smithi076 ceph-mon[32774]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-26T23:40:47.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:47 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/2411226276' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:40:47.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:47 smithi072 ceph-mon[28184]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-26T23:40:47.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:47 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/2411226276' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:40:47.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:47 smithi136 ceph-mon[33050]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-26T23:40:47.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:47 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/2411226276' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:40:49.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:49 smithi076 ceph-mon[32774]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:40:49.729 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:49 smithi072 ceph-mon[28184]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:40:49.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:49 smithi136 ceph-mon[33050]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:40:51.669 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:51 smithi072 ceph-mon[28184]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:40:51.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:51 smithi136 ceph-mon[33050]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:40:51.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:51 smithi076 ceph-mon[32774]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:40:53.448 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:53 smithi072 ceph-mon[28184]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:40:53.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:53 smithi136 ceph-mon[33050]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:40:53.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:53 smithi076 ceph-mon[32774]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:40:55.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:55 smithi072 ceph-mon[28184]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:40:55.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:55 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-26T23:40:55.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:55 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:40:55.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:55 smithi136 ceph-mon[33050]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:40:55.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:55 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-26T23:40:55.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:55 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:40:55.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:55 smithi076 ceph-mon[32774]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:40:55.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:55 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-26T23:40:55.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:55 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:40:56.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:56 smithi072 ceph-mon[28184]: Deploying daemon osd.9 on smithi072 2024-04-26T23:40:56.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:56 smithi136 ceph-mon[33050]: Deploying daemon osd.9 on smithi072 2024-04-26T23:40:56.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:56 smithi076 ceph-mon[32774]: Deploying daemon osd.9 on smithi072 2024-04-26T23:40:57.740 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:57 smithi072 ceph-mon[28184]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:40:57.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:57 smithi136 ceph-mon[33050]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:40:57.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:57 smithi076 ceph-mon[32774]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:40:59.706 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:40:59 smithi072 ceph-mon[28184]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:40:59.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:40:59 smithi136 ceph-mon[33050]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:40:59.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:40:59 smithi076 ceph-mon[32774]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:41:01.769 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:01 smithi072 ceph-mon[28184]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:41:01.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:01 smithi136 ceph-mon[33050]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:41:01.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:01 smithi076 ceph-mon[32774]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:41:03.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:03 smithi072 ceph-mon[28184]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:41:03.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:03 smithi136 ceph-mon[33050]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:41:03.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:03 smithi076 ceph-mon[32774]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:41:04.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:04 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:04.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:04 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:04.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:04 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:41:04.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:04 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:04.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:04 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:04.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:04 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:41:05.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:04 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:05.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:04 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:05.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:04 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:41:05.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:05 smithi072 ceph-mon[28184]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:41:05.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:05 smithi076 ceph-mon[32774]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:41:06.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:05 smithi136 ceph-mon[33050]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:41:07.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:07 smithi072 ceph-mon[28184]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:41:07.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:07 smithi072 ceph-mon[28184]: from='osd.9 [v2:172.21.15.72:6826/3423789152,v1:172.21.15.72:6827/3423789152]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T23:41:07.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:07 smithi072 ceph-mon[28184]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T23:41:07.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:07 smithi076 ceph-mon[32774]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:41:07.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:07 smithi076 ceph-mon[32774]: from='osd.9 [v2:172.21.15.72:6826/3423789152,v1:172.21.15.72:6827/3423789152]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T23:41:07.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:07 smithi076 ceph-mon[32774]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T23:41:08.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:07 smithi136 ceph-mon[33050]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:41:08.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:07 smithi136 ceph-mon[33050]: from='osd.9 [v2:172.21.15.72:6826/3423789152,v1:172.21.15.72:6827/3423789152]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T23:41:08.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:07 smithi136 ceph-mon[33050]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T23:41:08.805 INFO:teuthology.orchestra.run.smithi072.stdout:Created osd(s) 9 on host 'smithi072' 2024-04-26T23:41:08.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:41:08.801+0000 7f5413e00700 1 -- 172.21.15.72:0/606307190 <== mgr.14152 v2:172.21.15.72:6800/4281177709 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f541c063860 con 0x7f54000a3c40 2024-04-26T23:41:08.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:41:08.804+0000 7f5421200700 1 -- 172.21.15.72:0/606307190 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f54000a3c40 msgr2=0x7f54000a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:41:08.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:41:08.804+0000 7f5421200700 1 --2- 172.21.15.72:0/606307190 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f54000a3c40 0x7f54000a6100 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f541c077830 tx=0x7f540c0073d0 comp rx=0 tx=0).stop 2024-04-26T23:41:08.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:41:08.804+0000 7f5421200700 1 -- 172.21.15.72:0/606307190 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f541c077450 msgr2=0x7f541c19ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:41:08.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:41:08.804+0000 7f5421200700 1 --2- 172.21.15.72:0/606307190 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f541c077450 0x7f541c19ff70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f540800e9a0 tx=0x7f540800ecb0 comp rx=0 tx=0).stop 2024-04-26T23:41:08.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:41:08.804+0000 7f5421200700 1 -- 172.21.15.72:0/606307190 shutdown_connections 2024-04-26T23:41:08.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:41:08.804+0000 7f5421200700 1 --2- 172.21.15.72:0/606307190 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f54000a3c40 0x7f54000a6100 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:41:08.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:41:08.804+0000 7f5421200700 1 --2- 172.21.15.72:0/606307190 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f541c10f980 0x7f541c1a6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:41:08.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:41:08.804+0000 7f5421200700 1 --2- 172.21.15.72:0/606307190 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f541c075e80 0x7f541c19fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:41:08.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:41:08.804+0000 7f5421200700 1 --2- 172.21.15.72:0/606307190 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f541c077450 0x7f541c19ff70 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:41:08.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:41:08.804+0000 7f5421200700 1 -- 172.21.15.72:0/606307190 >> 172.21.15.72:0/606307190 conn(0x7f541c0fe960 msgr2=0x7f541c102d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:41:08.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:41:08.805+0000 7f5421200700 1 -- 172.21.15.72:0/606307190 shutdown_connections 2024-04-26T23:41:08.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:41:08.805+0000 7f5421200700 1 -- 172.21.15.72:0/606307190 wait complete. 2024-04-26T23:41:08.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:08 smithi076 ceph-mon[32774]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-26T23:41:08.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:08 smithi076 ceph-mon[32774]: osdmap e65: 10 total, 9 up, 10 in 2024-04-26T23:41:08.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:08 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T23:41:08.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:08 smithi076 ceph-mon[32774]: from='osd.9 [v2:172.21.15.72:6826/3423789152,v1:172.21.15.72:6827/3423789152]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]: dispatch 2024-04-26T23:41:08.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:08 smithi076 ceph-mon[32774]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]: dispatch 2024-04-26T23:41:08.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:08 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:08.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:08 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:09.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:08 smithi072 ceph-mon[28184]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-26T23:41:09.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:08 smithi072 ceph-mon[28184]: osdmap e65: 10 total, 9 up, 10 in 2024-04-26T23:41:09.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:08 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T23:41:09.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:08 smithi072 ceph-mon[28184]: from='osd.9 [v2:172.21.15.72:6826/3423789152,v1:172.21.15.72:6827/3423789152]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]: dispatch 2024-04-26T23:41:09.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:08 smithi072 ceph-mon[28184]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]: dispatch 2024-04-26T23:41:09.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:08 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:09.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:08 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:09.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:08 smithi136 ceph-mon[33050]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-26T23:41:09.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:08 smithi136 ceph-mon[33050]: osdmap e65: 10 total, 9 up, 10 in 2024-04-26T23:41:09.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:08 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T23:41:09.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:08 smithi136 ceph-mon[33050]: from='osd.9 [v2:172.21.15.72:6826/3423789152,v1:172.21.15.72:6827/3423789152]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]: dispatch 2024-04-26T23:41:09.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:08 smithi136 ceph-mon[33050]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]: dispatch 2024-04-26T23:41:09.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:08 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:09.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:08 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:09.767 DEBUG:teuthology.orchestra.run.smithi072:osd.9> sudo journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@osd.9.service 2024-04-26T23:41:09.771 INFO:tasks.cephadm:Deploying osd.10 on smithi076 with /dev/vg_nvme/lv_1... 2024-04-26T23:41:09.771 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-26T23:41:09.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:09 smithi076 ceph-mon[32774]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:41:09.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:09 smithi076 ceph-mon[32774]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]': finished 2024-04-26T23:41:09.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:09 smithi076 ceph-mon[32774]: osdmap e66: 10 total, 9 up, 10 in 2024-04-26T23:41:09.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:09 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T23:41:09.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:09 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T23:41:09.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:09 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:09.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:09 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:10.014 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T23:41:10.027 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:09 smithi072 ceph-mon[28184]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:41:10.028 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:09 smithi072 ceph-mon[28184]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]': finished 2024-04-26T23:41:10.028 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:09 smithi072 ceph-mon[28184]: osdmap e66: 10 total, 9 up, 10 in 2024-04-26T23:41:10.028 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:09 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T23:41:10.028 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:09 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T23:41:10.028 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:09 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:10.028 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:09 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:10.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:09 smithi136 ceph-mon[33050]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:41:10.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:09 smithi136 ceph-mon[33050]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]': finished 2024-04-26T23:41:10.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:09 smithi136 ceph-mon[33050]: osdmap e66: 10 total, 9 up, 10 in 2024-04-26T23:41:10.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:09 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T23:41:10.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:09 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T23:41:10.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:09 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:10.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:09 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:10.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:10 smithi076 ceph-mon[32774]: purged_snaps scrub starts 2024-04-26T23:41:10.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:10 smithi076 ceph-mon[32774]: purged_snaps scrub ok 2024-04-26T23:41:10.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:10 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T23:41:10.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:10 smithi076 ceph-mon[32774]: osd.9 [v2:172.21.15.72:6826/3423789152,v1:172.21.15.72:6827/3423789152] boot 2024-04-26T23:41:10.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:10 smithi076 ceph-mon[32774]: osdmap e67: 10 total, 10 up, 10 in 2024-04-26T23:41:10.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:10 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T23:41:10.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:10 smithi076 ceph-mon[32774]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:41:10.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:10 smithi072 ceph-mon[28184]: purged_snaps scrub starts 2024-04-26T23:41:10.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:10 smithi072 ceph-mon[28184]: purged_snaps scrub ok 2024-04-26T23:41:10.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:10 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T23:41:10.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:10 smithi072 ceph-mon[28184]: osd.9 [v2:172.21.15.72:6826/3423789152,v1:172.21.15.72:6827/3423789152] boot 2024-04-26T23:41:10.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:10 smithi072 ceph-mon[28184]: osdmap e67: 10 total, 10 up, 10 in 2024-04-26T23:41:10.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:10 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T23:41:10.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:10 smithi072 ceph-mon[28184]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:41:11.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:10 smithi136 ceph-mon[33050]: purged_snaps scrub starts 2024-04-26T23:41:11.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:10 smithi136 ceph-mon[33050]: purged_snaps scrub ok 2024-04-26T23:41:11.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:10 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T23:41:11.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:10 smithi136 ceph-mon[33050]: osd.9 [v2:172.21.15.72:6826/3423789152,v1:172.21.15.72:6827/3423789152] boot 2024-04-26T23:41:11.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:10 smithi136 ceph-mon[33050]: osdmap e67: 10 total, 10 up, 10 in 2024-04-26T23:41:11.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:10 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T23:41:11.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:10 smithi136 ceph-mon[33050]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-26T23:41:11.769 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:11 smithi076 ceph-mon[32774]: osdmap e68: 10 total, 10 up, 10 in 2024-04-26T23:41:11.878 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:11 smithi072 ceph-mon[28184]: osdmap e68: 10 total, 10 up, 10 in 2024-04-26T23:41:12.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:11 smithi136 ceph-mon[33050]: osdmap e68: 10 total, 10 up, 10 in 2024-04-26T23:41:12.216 INFO:teuthology.orchestra.run.smithi076.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-26T23:41:12.217 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-26T23:41:12.217 INFO:teuthology.orchestra.run.smithi076.stderr: stderr: 10+0 records in 2024-04-26T23:41:12.217 INFO:teuthology.orchestra.run.smithi076.stderr:10+0 records out 2024-04-26T23:41:12.217 INFO:teuthology.orchestra.run.smithi076.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.013138 s, 798 MB/s 2024-04-26T23:41:12.217 INFO:teuthology.orchestra.run.smithi076.stderr:--> Zapping successful for: 2024-04-26T23:41:12.915 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:12 smithi076 ceph-mon[32774]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:41:12.949 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi076:vg_nvme/lv_1 2024-04-26T23:41:12.951 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:12 smithi136 ceph-mon[33050]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:41:13.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:12 smithi072 ceph-mon[28184]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:41:13.198 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T23:41:15.022 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:15.018+0000 7fee6a042700 1 -- 172.21.15.76:0/555750706 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fee64105020 msgr2=0x7fee6410b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:41:15.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:15.018+0000 7fee6a042700 1 --2- 172.21.15.76:0/555750706 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fee64105020 0x7fee6410b8d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fee5c00b0d0 tx=0x7fee5c00b3e0 comp rx=0 tx=0).stop 2024-04-26T23:41:15.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:15.018+0000 7fee6a042700 1 -- 172.21.15.76:0/555750706 shutdown_connections 2024-04-26T23:41:15.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:15.018+0000 7fee6a042700 1 --2- 172.21.15.76:0/555750706 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fee64105020 0x7fee6410b8d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:41:15.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:15.018+0000 7fee6a042700 1 --2- 172.21.15.76:0/555750706 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fee64103480 0x7fee641038a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:41:15.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:15.018+0000 7fee6a042700 1 --2- 172.21.15.76:0/555750706 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fee64104680 0x7fee64104ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:41:15.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:15.018+0000 7fee6a042700 1 -- 172.21.15.76:0/555750706 >> 172.21.15.76:0/555750706 conn(0x7fee640fea20 msgr2=0x7fee64100e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:41:15.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:15.018+0000 7fee6a042700 1 -- 172.21.15.76:0/555750706 shutdown_connections 2024-04-26T23:41:15.023 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:15.018+0000 7fee6a042700 1 -- 172.21.15.76:0/555750706 wait complete. 2024-04-26T23:41:15.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:15.018+0000 7fee6a042700 1 Processor -- start 2024-04-26T23:41:15.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:15.018+0000 7fee6a042700 1 -- start start 2024-04-26T23:41:15.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:15.019+0000 7fee6a042700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fee64103480 0x7fee641996e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:41:15.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:15.019+0000 7fee6a042700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fee64104680 0x7fee64199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:41:15.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:15.019+0000 7fee6a042700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fee64105020 0x7fee641a0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:41:15.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:15.019+0000 7fee6a042700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fee6410f010 con 0x7fee64105020 2024-04-26T23:41:15.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:15.019+0000 7fee6a042700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fee6410ee90 con 0x7fee64104680 2024-04-26T23:41:15.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:15.019+0000 7fee6a042700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fee6410f190 con 0x7fee64103480 2024-04-26T23:41:15.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:15.019+0000 7fee62ffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fee64104680 0x7fee64199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:41:15.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:15.019+0000 7fee62ffd700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fee64104680 0x7fee64199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:42886/0 (socket says 172.21.15.76:42886) 2024-04-26T23:41:15.025 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:15.019+0000 7fee62ffd700 1 -- 172.21.15.76:0/1002443838 learned_addr learned my addr 172.21.15.76:0/1002443838 (peer_addr_for_me v2:172.21.15.76:0/0) 2024-04-26T23:41:15.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:15.019+0000 7fee637fe700 1 --2- 172.21.15.76:0/1002443838 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fee64103480 0x7fee641996e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:41:15.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:15.019+0000 7fee63fff700 1 --2- 172.21.15.76:0/1002443838 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fee64105020 0x7fee641a0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:41:15.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:15.019+0000 7fee62ffd700 1 -- 172.21.15.76:0/1002443838 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fee64103480 msgr2=0x7fee641996e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:41:15.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:15.019+0000 7fee62ffd700 1 --2- 172.21.15.76:0/1002443838 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fee64103480 0x7fee641996e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:41:15.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:15.019+0000 7fee62ffd700 1 -- 172.21.15.76:0/1002443838 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fee64105020 msgr2=0x7fee641a0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:41:15.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:15.019+0000 7fee62ffd700 1 --2- 172.21.15.76:0/1002443838 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fee64105020 0x7fee641a0ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:41:15.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:15.019+0000 7fee62ffd700 1 -- 172.21.15.76:0/1002443838 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee641a1340 con 0x7fee64104680 2024-04-26T23:41:15.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:15.019+0000 7fee637fe700 1 --2- 172.21.15.76:0/1002443838 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fee64103480 0x7fee641996e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:41:15.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:15.019+0000 7fee62ffd700 1 --2- 172.21.15.76:0/1002443838 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fee64104680 0x7fee64199c20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fee5800b530 tx=0x7fee5800b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:41:15.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:15.019+0000 7fee60ff9700 1 -- 172.21.15.76:0/1002443838 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee58015020 con 0x7fee64104680 2024-04-26T23:41:15.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:15.019+0000 7fee6a042700 1 -- 172.21.15.76:0/1002443838 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee641a1630 con 0x7fee64104680 2024-04-26T23:41:15.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:15.019+0000 7fee60ff9700 1 -- 172.21.15.76:0/1002443838 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee58011ce0 con 0x7fee64104680 2024-04-26T23:41:15.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:15.019+0000 7fee60ff9700 1 -- 172.21.15.76:0/1002443838 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee58010d50 con 0x7fee64104680 2024-04-26T23:41:15.026 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:15.020+0000 7fee6a042700 1 -- 172.21.15.76:0/1002443838 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee641a1ba0 con 0x7fee64104680 2024-04-26T23:41:15.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:15.021+0000 7fee6a042700 1 -- 172.21.15.76:0/1002443838 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee6406a110 con 0x7fee64104680 2024-04-26T23:41:15.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:15.022+0000 7fee60ff9700 1 -- 172.21.15.76:0/1002443838 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fee58018020 con 0x7fee64104680 2024-04-26T23:41:15.027 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:15.022+0000 7fee60ff9700 1 --2- 172.21.15.76:0/1002443838 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fee4c0a3c40 0x7fee4c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:41:15.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:15.022+0000 7fee60ff9700 1 -- 172.21.15.76:0/1002443838 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6261+0+0 (secure 0 0 0) 0x7fee580cf410 con 0x7fee64104680 2024-04-26T23:41:15.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:15.023+0000 7fee637fe700 1 --2- 172.21.15.76:0/1002443838 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fee4c0a3c40 0x7fee4c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:41:15.028 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:15.023+0000 7fee637fe700 1 --2- 172.21.15.76:0/1002443838 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fee4c0a3c40 0x7fee4c0a6100 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fee641044e0 tx=0x7fee54008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:41:15.031 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:15.027+0000 7fee60ff9700 1 -- 172.21.15.76:0/1002443838 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fee5801f020 con 0x7fee64104680 2024-04-26T23:41:15.243 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:15.238+0000 7fee6a042700 1 -- 172.21.15.76:0/1002443838 --> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fee64063860 con 0x7fee4c0a3c40 2024-04-26T23:41:15.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:15 smithi076 ceph-mon[32774]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:41:15.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:15 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:15.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:15 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:15.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:15 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:41:15.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:15 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:41:15.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:15 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:41:15.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:15 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:41:15.680 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:15 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:15.680 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:15 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:41:15.680 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:15 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:41:15.680 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:15 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:15.680 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:15 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:41:15.680 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:15 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:41:15.680 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:15 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:41:15.680 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:15 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:15.680 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:15 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:41:15.680 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:15 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:41:15.680 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:15 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:41:15.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:15 smithi072 ceph-mon[28184]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:41:15.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:15 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:15.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:15 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:15.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:15 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:41:15.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:15 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:41:15.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:15 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:41:15.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:15 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:41:15.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:15 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:15.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:15 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:41:15.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:15 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:41:15.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:15 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:15.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:15 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:41:15.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:15 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:41:15.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:15 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:41:15.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:15 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:15.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:15 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:41:15.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:15 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:41:15.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:15 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:41:15.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:15 smithi136 ceph-mon[33050]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:41:15.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:15 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:15.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:15 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:15.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:15 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:41:15.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:15 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:41:15.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:15 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:41:15.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:15 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:41:15.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:15 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:15.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:15 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:41:15.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:15 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:41:15.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:15 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:15.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:15 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:41:15.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:15 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:41:15.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:15 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:41:15.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:15 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:15.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:15 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:41:15.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:15 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:41:15.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:15 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:41:16.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:16 smithi072 ceph-mon[28184]: Detected new or changed devices on smithi072 2024-04-26T23:41:16.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:16 smithi072 ceph-mon[28184]: Adjusting osd_memory_target on smithi072 to 4272M 2024-04-26T23:41:16.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:16 smithi072 ceph-mon[28184]: from='client.24359 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:41:16.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:16 smithi136 ceph-mon[33050]: Detected new or changed devices on smithi072 2024-04-26T23:41:16.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:16 smithi136 ceph-mon[33050]: Adjusting osd_memory_target on smithi072 to 4272M 2024-04-26T23:41:16.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:16 smithi136 ceph-mon[33050]: from='client.24359 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:41:16.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:16 smithi076 ceph-mon[32774]: Detected new or changed devices on smithi072 2024-04-26T23:41:16.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:16 smithi076 ceph-mon[32774]: Adjusting osd_memory_target on smithi072 to 4272M 2024-04-26T23:41:16.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:16 smithi076 ceph-mon[32774]: from='client.24359 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:41:17.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:17 smithi076 ceph-mon[32774]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:41:17.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:17 smithi072 ceph-mon[28184]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:41:17.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:17 smithi136 ceph-mon[33050]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:41:18.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:18 smithi072 ceph-mon[28184]: from='client.? 172.21.15.76:0/1072310536' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9ec6d46a-0913-4208-97b5-167188ee8cb3"}]: dispatch 2024-04-26T23:41:18.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:18 smithi072 ceph-mon[28184]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9ec6d46a-0913-4208-97b5-167188ee8cb3"}]: dispatch 2024-04-26T23:41:18.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:18 smithi072 ceph-mon[28184]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9ec6d46a-0913-4208-97b5-167188ee8cb3"}]': finished 2024-04-26T23:41:18.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:18 smithi072 ceph-mon[28184]: osdmap e69: 11 total, 10 up, 11 in 2024-04-26T23:41:18.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:18 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T23:41:18.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:18 smithi136 ceph-mon[33050]: from='client.? 172.21.15.76:0/1072310536' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9ec6d46a-0913-4208-97b5-167188ee8cb3"}]: dispatch 2024-04-26T23:41:18.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:18 smithi136 ceph-mon[33050]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9ec6d46a-0913-4208-97b5-167188ee8cb3"}]: dispatch 2024-04-26T23:41:18.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:18 smithi136 ceph-mon[33050]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9ec6d46a-0913-4208-97b5-167188ee8cb3"}]': finished 2024-04-26T23:41:18.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:18 smithi136 ceph-mon[33050]: osdmap e69: 11 total, 10 up, 11 in 2024-04-26T23:41:18.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:18 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T23:41:18.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:18 smithi076 ceph-mon[32774]: from='client.? 172.21.15.76:0/1072310536' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9ec6d46a-0913-4208-97b5-167188ee8cb3"}]: dispatch 2024-04-26T23:41:18.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:18 smithi076 ceph-mon[32774]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9ec6d46a-0913-4208-97b5-167188ee8cb3"}]: dispatch 2024-04-26T23:41:18.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:18 smithi076 ceph-mon[32774]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9ec6d46a-0913-4208-97b5-167188ee8cb3"}]': finished 2024-04-26T23:41:18.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:18 smithi076 ceph-mon[32774]: osdmap e69: 11 total, 10 up, 11 in 2024-04-26T23:41:18.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:18 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T23:41:19.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:19 smithi072 ceph-mon[28184]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:41:19.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:19 smithi072 ceph-mon[28184]: from='client.? 172.21.15.76:0/2095372049' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:41:19.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:19 smithi136 ceph-mon[33050]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:41:19.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:19 smithi136 ceph-mon[33050]: from='client.? 172.21.15.76:0/2095372049' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:41:19.833 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:19 smithi076 ceph-mon[32774]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:41:19.833 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:19 smithi076 ceph-mon[32774]: from='client.? 172.21.15.76:0/2095372049' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:41:21.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:21 smithi072 ceph-mon[28184]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:41:21.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:21 smithi136 ceph-mon[33050]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:41:21.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:21 smithi076 ceph-mon[32774]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:41:23.771 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:23 smithi076 ceph-mon[32774]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:41:23.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:23 smithi072 ceph-mon[28184]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:41:23.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:23 smithi136 ceph-mon[33050]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:41:25.615 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:25 smithi076 ceph-mon[32774]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:41:25.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:25 smithi072 ceph-mon[28184]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:41:25.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:25 smithi136 ceph-mon[33050]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:41:26.785 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:26 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-26T23:41:26.785 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:26 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:41:26.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:26 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-26T23:41:26.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:26 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:41:26.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:26 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-26T23:41:26.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:26 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:41:27.665 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:27 smithi076 ceph-mon[32774]: Deploying daemon osd.10 on smithi076 2024-04-26T23:41:27.665 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:27 smithi076 ceph-mon[32774]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:41:27.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:27 smithi072 ceph-mon[28184]: Deploying daemon osd.10 on smithi076 2024-04-26T23:41:27.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:27 smithi072 ceph-mon[28184]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:41:27.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:27 smithi136 ceph-mon[33050]: Deploying daemon osd.10 on smithi076 2024-04-26T23:41:27.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:27 smithi136 ceph-mon[33050]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:41:29.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:29 smithi072 ceph-mon[28184]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:41:29.794 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:29 smithi076 ceph-mon[32774]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:41:29.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:29 smithi136 ceph-mon[33050]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:41:31.633 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:31 smithi076 ceph-mon[32774]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:41:31.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:31 smithi072 ceph-mon[28184]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:41:31.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:31 smithi136 ceph-mon[33050]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:41:32.620 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:32 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:41:32.620 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:32 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:32.620 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:32 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:32.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:32 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:41:32.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:32 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:32.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:32 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:32.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:32 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:41:32.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:32 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:32.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:32 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:33.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:33 smithi072 ceph-mon[28184]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:41:33.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:33 smithi136 ceph-mon[33050]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:41:33.906 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:33 smithi076 ceph-mon[32774]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:41:35.783 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:35 smithi076 ceph-mon[32774]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:41:35.783 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:35 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:35.783 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:35 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:35.783 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:35 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:41:35.783 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:35 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:41:35.783 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:35 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:35.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:35 smithi072 ceph-mon[28184]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:41:35.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:35 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:35.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:35 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:35.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:35 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:41:35.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:35 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:41:35.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:35 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:35.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:35 smithi136 ceph-mon[33050]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:41:35.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:35 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:35.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:35 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:35.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:35 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:41:35.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:35 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:41:35.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:35 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:35.983 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:35.979+0000 7fee60ff9700 1 -- 172.21.15.76:0/1002443838 <== mgr.14152 v2:172.21.15.72:6800/4281177709 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fee64063860 con 0x7fee4c0a3c40 2024-04-26T23:41:35.983 INFO:teuthology.orchestra.run.smithi076.stdout:Created osd(s) 10 on host 'smithi076' 2024-04-26T23:41:35.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:35.981+0000 7fee6a042700 1 -- 172.21.15.76:0/1002443838 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fee4c0a3c40 msgr2=0x7fee4c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:41:35.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:35.981+0000 7fee6a042700 1 --2- 172.21.15.76:0/1002443838 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fee4c0a3c40 0x7fee4c0a6100 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fee641044e0 tx=0x7fee54008040 comp rx=0 tx=0).stop 2024-04-26T23:41:35.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:35.981+0000 7fee6a042700 1 -- 172.21.15.76:0/1002443838 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fee64104680 msgr2=0x7fee64199c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:41:35.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:35.981+0000 7fee6a042700 1 --2- 172.21.15.76:0/1002443838 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fee64104680 0x7fee64199c20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fee5800b530 tx=0x7fee5800b8f0 comp rx=0 tx=0).stop 2024-04-26T23:41:35.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:35.981+0000 7fee6a042700 1 -- 172.21.15.76:0/1002443838 shutdown_connections 2024-04-26T23:41:35.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:35.981+0000 7fee6a042700 1 --2- 172.21.15.76:0/1002443838 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fee4c0a3c40 0x7fee4c0a6100 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:41:35.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:35.981+0000 7fee6a042700 1 --2- 172.21.15.76:0/1002443838 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fee64105020 0x7fee641a0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:41:35.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:35.981+0000 7fee6a042700 1 --2- 172.21.15.76:0/1002443838 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fee64103480 0x7fee641996e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:41:35.985 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:35.981+0000 7fee6a042700 1 --2- 172.21.15.76:0/1002443838 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fee64104680 0x7fee64199c20 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:41:35.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:35.981+0000 7fee6a042700 1 -- 172.21.15.76:0/1002443838 >> 172.21.15.76:0/1002443838 conn(0x7fee640fea20 msgr2=0x7fee64107a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:41:35.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:35.981+0000 7fee6a042700 1 -- 172.21.15.76:0/1002443838 shutdown_connections 2024-04-26T23:41:35.986 INFO:teuthology.orchestra.run.smithi076.stderr:2024-04-26T23:41:35.981+0000 7fee6a042700 1 -- 172.21.15.76:0/1002443838 wait complete. 2024-04-26T23:41:36.648 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:36 smithi076 ceph-mon[32774]: from='osd.10 [v2:172.21.15.76:6824/1120991574,v1:172.21.15.76:6825/1120991574]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T23:41:36.648 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:36 smithi076 ceph-mon[32774]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T23:41:36.648 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:36 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:36.648 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:36 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:36.648 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:36 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:41:36.673 DEBUG:teuthology.orchestra.run.smithi076:osd.10> sudo journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@osd.10.service 2024-04-26T23:41:36.676 INFO:tasks.cephadm:Deploying osd.11 on smithi136 with /dev/vg_nvme/lv_1... 2024-04-26T23:41:36.676 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-26T23:41:36.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:36 smithi072 ceph-mon[28184]: from='osd.10 [v2:172.21.15.76:6824/1120991574,v1:172.21.15.76:6825/1120991574]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T23:41:36.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:36 smithi072 ceph-mon[28184]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T23:41:36.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:36 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:36.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:36 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:36.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:36 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:41:36.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:36 smithi136 ceph-mon[33050]: from='osd.10 [v2:172.21.15.76:6824/1120991574,v1:172.21.15.76:6825/1120991574]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T23:41:36.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:36 smithi136 ceph-mon[33050]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T23:41:36.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:36 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:36.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:36 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:36.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:36 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:41:36.921 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T23:41:37.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:37 smithi136 ceph-mon[33050]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:41:37.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:37 smithi136 ceph-mon[33050]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-26T23:41:37.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:37 smithi136 ceph-mon[33050]: from='osd.10 [v2:172.21.15.76:6824/1120991574,v1:172.21.15.76:6825/1120991574]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-04-26T23:41:37.551 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:37 smithi136 ceph-mon[33050]: osdmap e70: 11 total, 10 up, 11 in 2024-04-26T23:41:37.551 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:37 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T23:41:37.551 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:37 smithi136 ceph-mon[33050]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-04-26T23:41:37.551 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:37 smithi136 ceph-mon[33050]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-04-26T23:41:37.551 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:37 smithi136 ceph-mon[33050]: osdmap e71: 11 total, 10 up, 11 in 2024-04-26T23:41:37.551 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:37 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T23:41:37.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:37 smithi072 ceph-mon[28184]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:41:37.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:37 smithi072 ceph-mon[28184]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-26T23:41:37.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:37 smithi072 ceph-mon[28184]: from='osd.10 [v2:172.21.15.76:6824/1120991574,v1:172.21.15.76:6825/1120991574]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-04-26T23:41:37.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:37 smithi072 ceph-mon[28184]: osdmap e70: 11 total, 10 up, 11 in 2024-04-26T23:41:37.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:37 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T23:41:37.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:37 smithi072 ceph-mon[28184]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-04-26T23:41:37.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:37 smithi072 ceph-mon[28184]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-04-26T23:41:37.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:37 smithi072 ceph-mon[28184]: osdmap e71: 11 total, 10 up, 11 in 2024-04-26T23:41:37.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:37 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T23:41:37.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:37 smithi076 ceph-mon[32774]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:41:37.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:37 smithi076 ceph-mon[32774]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-26T23:41:37.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:37 smithi076 ceph-mon[32774]: from='osd.10 [v2:172.21.15.76:6824/1120991574,v1:172.21.15.76:6825/1120991574]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-04-26T23:41:37.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:37 smithi076 ceph-mon[32774]: osdmap e70: 11 total, 10 up, 11 in 2024-04-26T23:41:37.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:37 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T23:41:37.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:37 smithi076 ceph-mon[32774]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2024-04-26T23:41:37.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:37 smithi076 ceph-mon[32774]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2024-04-26T23:41:37.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:37 smithi076 ceph-mon[32774]: osdmap e71: 11 total, 10 up, 11 in 2024-04-26T23:41:37.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:37 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T23:41:38.201 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 26 23:41:37 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-26T23:41:37.932+0000 7fb63e7ae700 -1 osd.10 0 waiting for initial osdmap 2024-04-26T23:41:38.201 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 26 23:41:37 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-26T23:41:37.949+0000 7fb636599700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T23:41:38.604 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:38 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T23:41:38.605 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:38 smithi136 ceph-mon[33050]: from='osd.10 ' entity='osd.10' 2024-04-26T23:41:38.605 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:38 smithi136 ceph-mon[33050]: osd.10 [v2:172.21.15.76:6824/1120991574,v1:172.21.15.76:6825/1120991574] boot 2024-04-26T23:41:38.605 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:38 smithi136 ceph-mon[33050]: osdmap e72: 11 total, 11 up, 11 in 2024-04-26T23:41:38.605 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:38 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T23:41:38.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:38 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T23:41:38.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:38 smithi072 ceph-mon[28184]: from='osd.10 ' entity='osd.10' 2024-04-26T23:41:38.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:38 smithi072 ceph-mon[28184]: osd.10 [v2:172.21.15.76:6824/1120991574,v1:172.21.15.76:6825/1120991574] boot 2024-04-26T23:41:38.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:38 smithi072 ceph-mon[28184]: osdmap e72: 11 total, 11 up, 11 in 2024-04-26T23:41:38.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:38 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T23:41:38.930 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:38 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T23:41:38.931 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:38 smithi076 ceph-mon[32774]: from='osd.10 ' entity='osd.10' 2024-04-26T23:41:38.931 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:38 smithi076 ceph-mon[32774]: osd.10 [v2:172.21.15.76:6824/1120991574,v1:172.21.15.76:6825/1120991574] boot 2024-04-26T23:41:38.931 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:38 smithi076 ceph-mon[32774]: osdmap e72: 11 total, 11 up, 11 in 2024-04-26T23:41:38.931 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:38 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T23:41:39.062 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-26T23:41:39.063 INFO:teuthology.orchestra.run.smithi136.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-26T23:41:39.063 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10+0 records in 2024-04-26T23:41:39.063 INFO:teuthology.orchestra.run.smithi136.stderr:10+0 records out 2024-04-26T23:41:39.063 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0082536 s, 1.3 GB/s 2024-04-26T23:41:39.063 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping successful for: 2024-04-26T23:41:39.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:39 smithi072 ceph-mon[28184]: purged_snaps scrub starts 2024-04-26T23:41:39.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:39 smithi072 ceph-mon[28184]: purged_snaps scrub ok 2024-04-26T23:41:39.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:39 smithi072 ceph-mon[28184]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:41:39.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:39 smithi072 ceph-mon[28184]: osdmap e73: 11 total, 11 up, 11 in 2024-04-26T23:41:39.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:39 smithi136 ceph-mon[33050]: purged_snaps scrub starts 2024-04-26T23:41:39.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:39 smithi136 ceph-mon[33050]: purged_snaps scrub ok 2024-04-26T23:41:39.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:39 smithi136 ceph-mon[33050]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:41:39.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:39 smithi136 ceph-mon[33050]: osdmap e73: 11 total, 11 up, 11 in 2024-04-26T23:41:39.801 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:39 smithi076 ceph-mon[32774]: purged_snaps scrub starts 2024-04-26T23:41:39.801 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:39 smithi076 ceph-mon[32774]: purged_snaps scrub ok 2024-04-26T23:41:39.801 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:39 smithi076 ceph-mon[32774]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-26T23:41:39.801 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:39 smithi076 ceph-mon[32774]: osdmap e73: 11 total, 11 up, 11 in 2024-04-26T23:41:39.829 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi136:vg_nvme/lv_1 2024-04-26T23:41:40.069 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T23:41:41.395 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:41 smithi136 ceph-mon[33050]: osdmap e74: 11 total, 11 up, 11 in 2024-04-26T23:41:41.395 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:41 smithi136 ceph-mon[33050]: Detected new or changed devices on smithi076 2024-04-26T23:41:41.395 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:41 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:41.395 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:41 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:41.395 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:41 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:41:41.395 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:41 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:41:41.395 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:41 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:41:41.395 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:41 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:41:41.396 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:41 smithi136 ceph-mon[33050]: Adjusting osd_memory_target on smithi076 to 4272M 2024-04-26T23:41:41.396 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:41 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:41.396 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:41 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:41:41.396 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:41 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:41:41.396 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:41 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:41.396 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:41 smithi136 ceph-mon[33050]: pgmap v265: 1 pgs: 1 remapped+peering; 577 KiB data, 352 MiB used, 983 GiB / 983 GiB avail 2024-04-26T23:41:41.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:41 smithi076 ceph-mon[32774]: osdmap e74: 11 total, 11 up, 11 in 2024-04-26T23:41:41.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:41 smithi076 ceph-mon[32774]: Detected new or changed devices on smithi076 2024-04-26T23:41:41.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:41 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:41.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:41 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:41.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:41 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:41:41.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:41 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:41:41.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:41 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:41:41.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:41 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:41:41.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:41 smithi076 ceph-mon[32774]: Adjusting osd_memory_target on smithi076 to 4272M 2024-04-26T23:41:41.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:41 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:41.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:41 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:41:41.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:41 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:41:41.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:41 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:41.430 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:41 smithi076 ceph-mon[32774]: pgmap v265: 1 pgs: 1 remapped+peering; 577 KiB data, 352 MiB used, 983 GiB / 983 GiB avail 2024-04-26T23:41:41.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:41 smithi072 ceph-mon[28184]: osdmap e74: 11 total, 11 up, 11 in 2024-04-26T23:41:41.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:41 smithi072 ceph-mon[28184]: Detected new or changed devices on smithi076 2024-04-26T23:41:41.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:41 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:41.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:41 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:41.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:41 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:41:41.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:41 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:41:41.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:41 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:41:41.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:41 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:41:41.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:41 smithi072 ceph-mon[28184]: Adjusting osd_memory_target on smithi076 to 4272M 2024-04-26T23:41:41.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:41 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:41.545 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:41 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:41:41.545 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:41 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:41:41.545 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:41 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:41.545 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:41 smithi072 ceph-mon[28184]: pgmap v265: 1 pgs: 1 remapped+peering; 577 KiB data, 352 MiB used, 983 GiB / 983 GiB avail 2024-04-26T23:41:41.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:41:41.758+0000 7f73f3d5e700 1 -- 172.21.15.136:0/4167183313 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73ec105020 msgr2=0x7f73ec10b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:41:41.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:41:41.758+0000 7f73f3d5e700 1 --2- 172.21.15.136:0/4167183313 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73ec105020 0x7f73ec10b8d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f73e400b0a0 tx=0x7f73e400b3b0 comp rx=0 tx=0).stop 2024-04-26T23:41:41.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:41:41.758+0000 7f73f3d5e700 1 -- 172.21.15.136:0/4167183313 shutdown_connections 2024-04-26T23:41:41.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:41:41.758+0000 7f73f3d5e700 1 --2- 172.21.15.136:0/4167183313 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73ec105020 0x7f73ec10b8d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:41:41.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:41:41.758+0000 7f73f3d5e700 1 --2- 172.21.15.136:0/4167183313 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f73ec103480 0x7f73ec1038a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:41:41.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:41:41.758+0000 7f73f3d5e700 1 --2- 172.21.15.136:0/4167183313 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73ec104680 0x7f73ec104ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:41:41.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:41:41.758+0000 7f73f3d5e700 1 -- 172.21.15.136:0/4167183313 >> 172.21.15.136:0/4167183313 conn(0x7f73ec0fea20 msgr2=0x7f73ec100e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:41:41.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:41:41.758+0000 7f73f3d5e700 1 -- 172.21.15.136:0/4167183313 shutdown_connections 2024-04-26T23:41:41.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:41:41.758+0000 7f73f3d5e700 1 -- 172.21.15.136:0/4167183313 wait complete. 2024-04-26T23:41:41.764 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:41:41.758+0000 7f73f3d5e700 1 Processor -- start 2024-04-26T23:41:41.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:41:41.758+0000 7f73f3d5e700 1 -- start start 2024-04-26T23:41:41.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:41:41.758+0000 7f73f3d5e700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73ec103480 0x7f73ec199600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:41:41.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:41:41.758+0000 7f73f3d5e700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f73ec104680 0x7f73ec199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:41:41.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:41:41.758+0000 7f73f3d5e700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73ec105020 0x7f73ec1a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:41:41.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:41:41.758+0000 7f73f3d5e700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f73ec10ef90 con 0x7f73ec103480 2024-04-26T23:41:41.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:41:41.759+0000 7f73f3d5e700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f73ec10ee10 con 0x7f73ec104680 2024-04-26T23:41:41.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:41:41.759+0000 7f73f3d5e700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f73ec10f110 con 0x7f73ec105020 2024-04-26T23:41:41.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:41:41.759+0000 7f73f22fb700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73ec105020 0x7f73ec1a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:41:41.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:41:41.759+0000 7f73f22fb700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73ec105020 0x7f73ec1a0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:47632/0 (socket says 172.21.15.136:47632) 2024-04-26T23:41:41.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:41:41.759+0000 7f73f22fb700 1 -- 172.21.15.136:0/1284871748 learned_addr learned my addr 172.21.15.136:0/1284871748 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-26T23:41:41.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:41:41.759+0000 7f73f12f9700 1 --2- 172.21.15.136:0/1284871748 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f73ec104680 0x7f73ec199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:41:41.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:41:41.759+0000 7f73f22fb700 1 -- 172.21.15.136:0/1284871748 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f73ec104680 msgr2=0x7f73ec199b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:41:41.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:41:41.759+0000 7f73f22fb700 1 --2- 172.21.15.136:0/1284871748 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f73ec104680 0x7f73ec199b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:41:41.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:41:41.759+0000 7f73f22fb700 1 -- 172.21.15.136:0/1284871748 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73ec103480 msgr2=0x7f73ec199600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:41:41.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:41:41.759+0000 7f73f22fb700 1 --2- 172.21.15.136:0/1284871748 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73ec103480 0x7f73ec199600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:41:41.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:41:41.759+0000 7f73f22fb700 1 -- 172.21.15.136:0/1284871748 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73ec1a1260 con 0x7f73ec105020 2024-04-26T23:41:41.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:41:41.759+0000 7f73f22fb700 1 --2- 172.21.15.136:0/1284871748 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73ec105020 0x7f73ec1a0bc0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f73e4004e40 tx=0x7f73e4004e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:41:41.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:41:41.759+0000 7f73e2ffd700 1 -- 172.21.15.136:0/1284871748 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e4021070 con 0x7f73ec105020 2024-04-26T23:41:41.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:41:41.759+0000 7f73f3d5e700 1 -- 172.21.15.136:0/1284871748 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73ec1a14f0 con 0x7f73ec105020 2024-04-26T23:41:41.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:41:41.759+0000 7f73e2ffd700 1 -- 172.21.15.136:0/1284871748 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73e4007c80 con 0x7f73ec105020 2024-04-26T23:41:41.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:41:41.759+0000 7f73e2ffd700 1 -- 172.21.15.136:0/1284871748 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e4004850 con 0x7f73ec105020 2024-04-26T23:41:41.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:41:41.759+0000 7f73f3d5e700 1 -- 172.21.15.136:0/1284871748 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73ec1a1a70 con 0x7f73ec105020 2024-04-26T23:41:41.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:41:41.760+0000 7f73f3d5e700 1 -- 172.21.15.136:0/1284871748 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73ec06a110 con 0x7f73ec105020 2024-04-26T23:41:41.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:41:41.761+0000 7f73e2ffd700 1 -- 172.21.15.136:0/1284871748 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f73e4003f40 con 0x7f73ec105020 2024-04-26T23:41:41.767 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:41:41.762+0000 7f73e2ffd700 1 --2- 172.21.15.136:0/1284871748 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f73d80a3c40 0x7f73d80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:41:41.767 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:41:41.762+0000 7f73e2ffd700 1 -- 172.21.15.136:0/1284871748 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 6694+0+0 (secure 0 0 0) 0x7f73e400f020 con 0x7f73ec105020 2024-04-26T23:41:41.767 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:41:41.762+0000 7f73f1afa700 1 --2- 172.21.15.136:0/1284871748 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f73d80a3c40 0x7f73d80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:41:41.767 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:41:41.763+0000 7f73f1afa700 1 --2- 172.21.15.136:0/1284871748 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f73d80a3c40 0x7f73d80a6100 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f73ec1044e0 tx=0x7f73e8009470 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:41:41.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:41:41.765+0000 7f73e2ffd700 1 -- 172.21.15.136:0/1284871748 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f73e402a020 con 0x7f73ec105020 2024-04-26T23:41:41.975 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:41:41.971+0000 7f73f3d5e700 1 -- 172.21.15.136:0/1284871748 --> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f73ec063920 con 0x7f73d80a3c40 2024-04-26T23:41:42.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:42 smithi072 ceph-mon[28184]: osdmap e75: 11 total, 11 up, 11 in 2024-04-26T23:41:42.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:42 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:41:42.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:42 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:41:42.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:42 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:41:42.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:42 smithi136 ceph-mon[33050]: osdmap e75: 11 total, 11 up, 11 in 2024-04-26T23:41:42.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:42 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:41:42.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:42 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:41:42.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:42 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:41:42.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:42 smithi076 ceph-mon[32774]: osdmap e75: 11 total, 11 up, 11 in 2024-04-26T23:41:42.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:42 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T23:41:42.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:42 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T23:41:42.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:42 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:41:43.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:43 smithi072 ceph-mon[28184]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:41:43.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:43 smithi072 ceph-mon[28184]: osdmap e76: 11 total, 11 up, 11 in 2024-04-26T23:41:43.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:43 smithi072 ceph-mon[28184]: pgmap v268: 1 pgs: 1 remapped+peering; 577 KiB data, 352 MiB used, 983 GiB / 983 GiB avail 2024-04-26T23:41:43.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:43 smithi136 ceph-mon[33050]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:41:43.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:43 smithi136 ceph-mon[33050]: osdmap e76: 11 total, 11 up, 11 in 2024-04-26T23:41:43.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:43 smithi136 ceph-mon[33050]: pgmap v268: 1 pgs: 1 remapped+peering; 577 KiB data, 352 MiB used, 983 GiB / 983 GiB avail 2024-04-26T23:41:43.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:43 smithi076 ceph-mon[32774]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:41:43.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:43 smithi076 ceph-mon[32774]: osdmap e76: 11 total, 11 up, 11 in 2024-04-26T23:41:43.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:43 smithi076 ceph-mon[32774]: pgmap v268: 1 pgs: 1 remapped+peering; 577 KiB data, 352 MiB used, 983 GiB / 983 GiB avail 2024-04-26T23:41:45.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:45 smithi072 ceph-mon[28184]: pgmap v269: 1 pgs: 1 remapped+peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T23:41:45.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:45 smithi072 ceph-mon[28184]: from='client.? 172.21.15.136:0/3225592697' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bdd0774f-6e58-4918-b4cc-afd4f1118d3f"}]: dispatch 2024-04-26T23:41:45.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:45 smithi072 ceph-mon[28184]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bdd0774f-6e58-4918-b4cc-afd4f1118d3f"}]: dispatch 2024-04-26T23:41:45.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:45 smithi072 ceph-mon[28184]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bdd0774f-6e58-4918-b4cc-afd4f1118d3f"}]': finished 2024-04-26T23:41:45.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:45 smithi072 ceph-mon[28184]: osdmap e77: 12 total, 11 up, 12 in 2024-04-26T23:41:45.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:45 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T23:41:45.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:45 smithi136 ceph-mon[33050]: pgmap v269: 1 pgs: 1 remapped+peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T23:41:45.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:45 smithi136 ceph-mon[33050]: from='client.? 172.21.15.136:0/3225592697' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bdd0774f-6e58-4918-b4cc-afd4f1118d3f"}]: dispatch 2024-04-26T23:41:45.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:45 smithi136 ceph-mon[33050]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bdd0774f-6e58-4918-b4cc-afd4f1118d3f"}]: dispatch 2024-04-26T23:41:45.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:45 smithi136 ceph-mon[33050]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bdd0774f-6e58-4918-b4cc-afd4f1118d3f"}]': finished 2024-04-26T23:41:45.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:45 smithi136 ceph-mon[33050]: osdmap e77: 12 total, 11 up, 12 in 2024-04-26T23:41:45.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:45 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T23:41:45.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:45 smithi076 ceph-mon[32774]: pgmap v269: 1 pgs: 1 remapped+peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T23:41:45.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:45 smithi076 ceph-mon[32774]: from='client.? 172.21.15.136:0/3225592697' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bdd0774f-6e58-4918-b4cc-afd4f1118d3f"}]: dispatch 2024-04-26T23:41:45.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:45 smithi076 ceph-mon[32774]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bdd0774f-6e58-4918-b4cc-afd4f1118d3f"}]: dispatch 2024-04-26T23:41:45.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:45 smithi076 ceph-mon[32774]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bdd0774f-6e58-4918-b4cc-afd4f1118d3f"}]': finished 2024-04-26T23:41:45.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:45 smithi076 ceph-mon[32774]: osdmap e77: 12 total, 11 up, 12 in 2024-04-26T23:41:45.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:45 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T23:41:46.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:46 smithi072 ceph-mon[28184]: from='client.? 172.21.15.136:0/1610743587' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:41:46.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:46 smithi136 ceph-mon[33050]: from='client.? 172.21.15.136:0/1610743587' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:41:46.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:46 smithi076 ceph-mon[32774]: from='client.? 172.21.15.136:0/1610743587' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T23:41:47.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:47 smithi072 ceph-mon[28184]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T23:41:47.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:47 smithi136 ceph-mon[33050]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T23:41:47.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:47 smithi076 ceph-mon[32774]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T23:41:49.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:49 smithi076 ceph-mon[32774]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-04-26T23:41:49.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:49 smithi072 ceph-mon[28184]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-04-26T23:41:49.549 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:49 smithi136 ceph-mon[33050]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-04-26T23:41:51.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:51 smithi072 ceph-mon[28184]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-04-26T23:41:51.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:51 smithi136 ceph-mon[33050]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-04-26T23:41:51.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:51 smithi076 ceph-mon[32774]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-04-26T23:41:52.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:52 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-26T23:41:52.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:52 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:41:52.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:52 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-26T23:41:52.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:52 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:41:52.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:52 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-26T23:41:52.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:52 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:41:53.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:53 smithi072 ceph-mon[28184]: Deploying daemon osd.11 on smithi136 2024-04-26T23:41:53.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:53 smithi072 ceph-mon[28184]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T23:41:53.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:53 smithi136 ceph-mon[33050]: Deploying daemon osd.11 on smithi136 2024-04-26T23:41:53.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:53 smithi136 ceph-mon[33050]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T23:41:53.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:53 smithi076 ceph-mon[32774]: Deploying daemon osd.11 on smithi136 2024-04-26T23:41:53.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:53 smithi076 ceph-mon[32774]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T23:41:55.525 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:55 smithi136 ceph-mon[33050]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T23:41:55.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:55 smithi072 ceph-mon[28184]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T23:41:55.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:55 smithi076 ceph-mon[32774]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T23:41:57.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:57 smithi136 ceph-mon[33050]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T23:41:57.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:57 smithi072 ceph-mon[28184]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T23:41:57.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:57 smithi076 ceph-mon[32774]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T23:41:59.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:58 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:59.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:58 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:59.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:58 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:41:59.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:41:58 smithi076 ceph-mon[32774]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T23:41:59.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:58 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:59.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:58 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:59.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:58 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:41:59.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:41:58 smithi072 ceph-mon[28184]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T23:41:59.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:58 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:59.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:58 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:41:59.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:58 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:41:59.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:41:58 smithi136 ceph-mon[33050]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T23:42:01.516 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:01.512+0000 7f73e2ffd700 1 -- 172.21.15.136:0/1284871748 <== mgr.14152 v2:172.21.15.72:6800/4281177709 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f73ec063920 con 0x7f73d80a3c40 2024-04-26T23:42:01.517 INFO:teuthology.orchestra.run.smithi136.stdout:Created osd(s) 11 on host 'smithi136' 2024-04-26T23:42:01.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:01.515+0000 7f73f3d5e700 1 -- 172.21.15.136:0/1284871748 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f73d80a3c40 msgr2=0x7f73d80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:01.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:01.515+0000 7f73f3d5e700 1 --2- 172.21.15.136:0/1284871748 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f73d80a3c40 0x7f73d80a6100 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f73ec1044e0 tx=0x7f73e8009470 comp rx=0 tx=0).stop 2024-04-26T23:42:01.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:01.515+0000 7f73f3d5e700 1 -- 172.21.15.136:0/1284871748 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73ec105020 msgr2=0x7f73ec1a0bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:01.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:01.515+0000 7f73f3d5e700 1 --2- 172.21.15.136:0/1284871748 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73ec105020 0x7f73ec1a0bc0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f73e4004e40 tx=0x7f73e4004e70 comp rx=0 tx=0).stop 2024-04-26T23:42:01.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:01.515+0000 7f73f3d5e700 1 -- 172.21.15.136:0/1284871748 shutdown_connections 2024-04-26T23:42:01.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:01.515+0000 7f73f3d5e700 1 --2- 172.21.15.136:0/1284871748 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f73d80a3c40 0x7f73d80a6100 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:01.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:01.515+0000 7f73f3d5e700 1 --2- 172.21.15.136:0/1284871748 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73ec105020 0x7f73ec1a0bc0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:01.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:01.515+0000 7f73f3d5e700 1 --2- 172.21.15.136:0/1284871748 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73ec103480 0x7f73ec199600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:01.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:01.515+0000 7f73f3d5e700 1 --2- 172.21.15.136:0/1284871748 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f73ec104680 0x7f73ec199b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:01.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:01.515+0000 7f73f3d5e700 1 -- 172.21.15.136:0/1284871748 >> 172.21.15.136:0/1284871748 conn(0x7f73ec0fea20 msgr2=0x7f73ec107a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:01.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:01.515+0000 7f73f3d5e700 1 -- 172.21.15.136:0/1284871748 shutdown_connections 2024-04-26T23:42:01.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:01.515+0000 7f73f3d5e700 1 -- 172.21.15.136:0/1284871748 wait complete. 2024-04-26T23:42:01.694 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:01 smithi076 ceph-mon[32774]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T23:42:01.757 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:01 smithi136 ceph-mon[33050]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T23:42:01.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:01 smithi072 ceph-mon[28184]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T23:42:02.240 DEBUG:teuthology.orchestra.run.smithi136:osd.11> sudo journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@osd.11.service 2024-04-26T23:42:02.243 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-26T23:42:02.243 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-26T23:42:02.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:02 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:02.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:02 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:02.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:02 smithi136 ceph-mon[33050]: from='osd.11 [v2:172.21.15.136:6824/1135995193,v1:172.21.15.136:6825/1135995193]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T23:42:02.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:02 smithi136 ceph-mon[33050]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T23:42:02.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:02 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:02.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:02 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:02.690 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:42:02.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:02 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:02.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:02 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:02.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:02 smithi072 ceph-mon[28184]: from='osd.11 [v2:172.21.15.136:6824/1135995193,v1:172.21.15.136:6825/1135995193]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T23:42:02.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:02 smithi072 ceph-mon[28184]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T23:42:02.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:02 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:02.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:02 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:02.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:02 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:02.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:02 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:02.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:02 smithi076 ceph-mon[32774]: from='osd.11 [v2:172.21.15.136:6824/1135995193,v1:172.21.15.136:6825/1135995193]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T23:42:02.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:02 smithi076 ceph-mon[32774]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T23:42:02.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:02 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:02.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:02 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:03.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:03 smithi072 ceph-mon[28184]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T23:42:03.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:03 smithi072 ceph-mon[28184]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-26T23:42:03.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:03 smithi072 ceph-mon[28184]: from='osd.11 [v2:172.21.15.136:6824/1135995193,v1:172.21.15.136:6825/1135995193]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-04-26T23:42:03.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:03 smithi072 ceph-mon[28184]: osdmap e78: 12 total, 11 up, 12 in 2024-04-26T23:42:03.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:03 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T23:42:03.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:03 smithi072 ceph-mon[28184]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-04-26T23:42:03.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:03 smithi136 ceph-mon[33050]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T23:42:03.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:03 smithi136 ceph-mon[33050]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-26T23:42:03.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:03 smithi136 ceph-mon[33050]: from='osd.11 [v2:172.21.15.136:6824/1135995193,v1:172.21.15.136:6825/1135995193]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-04-26T23:42:03.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:03 smithi136 ceph-mon[33050]: osdmap e78: 12 total, 11 up, 12 in 2024-04-26T23:42:03.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:03 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T23:42:03.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:03 smithi136 ceph-mon[33050]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-04-26T23:42:03.801 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 26 23:42:03 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-26T23:42:03.477+0000 7fe36fd8e700 -1 osd.11 0 waiting for initial osdmap 2024-04-26T23:42:03.801 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 26 23:42:03 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-26T23:42:03.491+0000 7fe36bb81700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T23:42:03.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:03 smithi076 ceph-mon[32774]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T23:42:03.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:03 smithi076 ceph-mon[32774]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-26T23:42:03.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:03 smithi076 ceph-mon[32774]: from='osd.11 [v2:172.21.15.136:6824/1135995193,v1:172.21.15.136:6825/1135995193]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-04-26T23:42:03.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:03 smithi076 ceph-mon[32774]: osdmap e78: 12 total, 11 up, 12 in 2024-04-26T23:42:03.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:03 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T23:42:03.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:03 smithi076 ceph-mon[32774]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-04-26T23:42:04.706 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:04 smithi136 ceph-mon[33050]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-04-26T23:42:04.707 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:04 smithi136 ceph-mon[33050]: osdmap e79: 12 total, 11 up, 12 in 2024-04-26T23:42:04.707 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:04 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T23:42:04.707 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:04 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T23:42:04.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:04 smithi072 ceph-mon[28184]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-04-26T23:42:04.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:04 smithi072 ceph-mon[28184]: osdmap e79: 12 total, 11 up, 12 in 2024-04-26T23:42:04.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:04 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T23:42:04.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:04 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T23:42:04.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:04 smithi076 ceph-mon[32774]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-04-26T23:42:04.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:04 smithi076 ceph-mon[32774]: osdmap e79: 12 total, 11 up, 12 in 2024-04-26T23:42:04.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:04 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T23:42:04.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:04 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T23:42:05.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.029+0000 7f8263400700 1 -- 172.21.15.72:0/802330372 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f825c105030 msgr2=0x7f825c10b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:05.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.029+0000 7f8263400700 1 --2- 172.21.15.72:0/802330372 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f825c105030 0x7f825c10b8e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8254009990 tx=0x7f8254009ca0 comp rx=0 tx=0).stop 2024-04-26T23:42:05.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.029+0000 7f8263400700 1 -- 172.21.15.72:0/802330372 shutdown_connections 2024-04-26T23:42:05.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.029+0000 7f8263400700 1 --2- 172.21.15.72:0/802330372 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f825c105030 0x7f825c10b8e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:05.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.029+0000 7f8263400700 1 --2- 172.21.15.72:0/802330372 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f825c103490 0x7f825c1038b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:05.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.029+0000 7f8263400700 1 --2- 172.21.15.72:0/802330372 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f825c104690 0x7f825c104af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:05.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.029+0000 7f8263400700 1 -- 172.21.15.72:0/802330372 >> 172.21.15.72:0/802330372 conn(0x7f825c0fe9f0 msgr2=0x7f825c100e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:05.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.029+0000 7f8263400700 1 -- 172.21.15.72:0/802330372 shutdown_connections 2024-04-26T23:42:05.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.030+0000 7f8263400700 1 -- 172.21.15.72:0/802330372 wait complete. 2024-04-26T23:42:05.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.030+0000 7f8263400700 1 Processor -- start 2024-04-26T23:42:05.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.031+0000 7f8263400700 1 -- start start 2024-04-26T23:42:05.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.031+0000 7f8263400700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f825c103490 0x7f825c199570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:05.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.031+0000 7f8263400700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f825c104690 0x7f825c199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:05.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.031+0000 7f8263400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f825c105030 0x7f825c1a0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:05.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.031+0000 7f8263400700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f825c10ef80 con 0x7f825c104690 2024-04-26T23:42:05.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.031+0000 7f8263400700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f825c10ee00 con 0x7f825c105030 2024-04-26T23:42:05.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.031+0000 7f8263400700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f825c10f100 con 0x7f825c103490 2024-04-26T23:42:05.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.031+0000 7f825be00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f825c104690 0x7f825c199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:05.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.031+0000 7f825be00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f825c104690 0x7f825c199ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:50238/0 (socket says 172.21.15.72:50238) 2024-04-26T23:42:05.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.031+0000 7f825be00700 1 -- 172.21.15.72:0/1257812578 learned_addr learned my addr 172.21.15.72:0/1257812578 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:42:05.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.031+0000 7f8261400700 1 --2- 172.21.15.72:0/1257812578 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f825c105030 0x7f825c1a0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:05.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.032+0000 7f825be00700 1 -- 172.21.15.72:0/1257812578 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f825c103490 msgr2=0x7f825c199570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:05.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.032+0000 7f825be00700 1 --2- 172.21.15.72:0/1257812578 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f825c103490 0x7f825c199570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:05.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.032+0000 7f825be00700 1 -- 172.21.15.72:0/1257812578 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f825c105030 msgr2=0x7f825c1a0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:05.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.032+0000 7f825be00700 1 --2- 172.21.15.72:0/1257812578 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f825c105030 0x7f825c1a0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:05.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.032+0000 7f825be00700 1 -- 172.21.15.72:0/1257812578 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f825c1a11d0 con 0x7f825c104690 2024-04-26T23:42:05.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.032+0000 7f8261400700 1 --2- 172.21.15.72:0/1257812578 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f825c105030 0x7f825c1a0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:42:05.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.032+0000 7f825be00700 1 --2- 172.21.15.72:0/1257812578 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f825c104690 0x7f825c199ab0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f824800e9f0 tx=0x7f824800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:05.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.032+0000 7f8259600700 1 -- 172.21.15.72:0/1257812578 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f824800cbd0 con 0x7f825c104690 2024-04-26T23:42:05.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.032+0000 7f8263400700 1 -- 172.21.15.72:0/1257812578 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f825c1a14c0 con 0x7f825c104690 2024-04-26T23:42:05.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.032+0000 7f8259600700 1 -- 172.21.15.72:0/1257812578 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f824800cd70 con 0x7f825c104690 2024-04-26T23:42:05.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.033+0000 7f8259600700 1 -- 172.21.15.72:0/1257812578 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8248005680 con 0x7f825c104690 2024-04-26T23:42:05.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.033+0000 7f8263400700 1 -- 172.21.15.72:0/1257812578 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f825c1a1a00 con 0x7f825c104690 2024-04-26T23:42:05.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.034+0000 7f8263400700 1 -- 172.21.15.72:0/1257812578 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f825c06a110 con 0x7f825c104690 2024-04-26T23:42:05.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.036+0000 7f8259600700 1 -- 172.21.15.72:0/1257812578 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8248004750 con 0x7f825c104690 2024-04-26T23:42:05.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.037+0000 7f8259600700 1 --2- 172.21.15.72:0/1257812578 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f82440a3c40 0x7f82440a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:05.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.037+0000 7f8259600700 1 -- 172.21.15.72:0/1257812578 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 7127+0+0 (secure 0 0 0) 0x7f82480d02a0 con 0x7f825c104690 2024-04-26T23:42:05.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.037+0000 7f8260a00700 1 --2- 172.21.15.72:0/1257812578 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f82440a3c40 0x7f82440a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:05.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.038+0000 7f8260a00700 1 --2- 172.21.15.72:0/1257812578 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f82440a3c40 0x7f82440a6100 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f825c1044f0 tx=0x7f825000b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:05.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.042+0000 7f8259600700 1 -- 172.21.15.72:0/1257812578 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f824809d080 con 0x7f825c104690 2024-04-26T23:42:05.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.271+0000 7f8263400700 1 -- 172.21.15.72:0/1257812578 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f825c064030 con 0x7f825c104690 2024-04-26T23:42:05.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.272+0000 7f8259600700 1 -- 172.21.15.72:0/1257812578 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v80) ==== 74+0+133 (secure 0 0 0) 0x7f824809c980 con 0x7f825c104690 2024-04-26T23:42:05.276 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:42:05.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.276+0000 7f8263400700 1 -- 172.21.15.72:0/1257812578 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f82440a3c40 msgr2=0x7f82440a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:05.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.276+0000 7f8263400700 1 --2- 172.21.15.72:0/1257812578 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f82440a3c40 0x7f82440a6100 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f825c1044f0 tx=0x7f825000b040 comp rx=0 tx=0).stop 2024-04-26T23:42:05.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.276+0000 7f8263400700 1 -- 172.21.15.72:0/1257812578 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f825c104690 msgr2=0x7f825c199ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:05.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.276+0000 7f8263400700 1 --2- 172.21.15.72:0/1257812578 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f825c104690 0x7f825c199ab0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f824800e9f0 tx=0x7f824800ed00 comp rx=0 tx=0).stop 2024-04-26T23:42:05.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.276+0000 7f8263400700 1 -- 172.21.15.72:0/1257812578 shutdown_connections 2024-04-26T23:42:05.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.276+0000 7f8263400700 1 --2- 172.21.15.72:0/1257812578 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f82440a3c40 0x7f82440a6100 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:05.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.276+0000 7f8263400700 1 --2- 172.21.15.72:0/1257812578 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f825c105030 0x7f825c1a0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:05.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.276+0000 7f8263400700 1 --2- 172.21.15.72:0/1257812578 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f825c103490 0x7f825c199570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:05.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.276+0000 7f8263400700 1 --2- 172.21.15.72:0/1257812578 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f825c104690 0x7f825c199ab0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:05.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.276+0000 7f8263400700 1 -- 172.21.15.72:0/1257812578 >> 172.21.15.72:0/1257812578 conn(0x7f825c0fe9f0 msgr2=0x7f825c107990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:05.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.276+0000 7f8263400700 1 -- 172.21.15.72:0/1257812578 shutdown_connections 2024-04-26T23:42:05.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:05.276+0000 7f8263400700 1 -- 172.21.15.72:0/1257812578 wait complete. 2024-04-26T23:42:05.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:05 smithi072 ceph-mon[28184]: purged_snaps scrub starts 2024-04-26T23:42:05.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:05 smithi072 ceph-mon[28184]: purged_snaps scrub ok 2024-04-26T23:42:05.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:05 smithi072 ceph-mon[28184]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T23:42:05.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:05 smithi072 ceph-mon[28184]: osd.11 [v2:172.21.15.136:6824/1135995193,v1:172.21.15.136:6825/1135995193] boot 2024-04-26T23:42:05.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:05 smithi072 ceph-mon[28184]: osdmap e80: 12 total, 12 up, 12 in 2024-04-26T23:42:05.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:05 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T23:42:05.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:05 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/1257812578' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T23:42:05.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:05 smithi136 ceph-mon[33050]: purged_snaps scrub starts 2024-04-26T23:42:05.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:05 smithi136 ceph-mon[33050]: purged_snaps scrub ok 2024-04-26T23:42:05.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:05 smithi136 ceph-mon[33050]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T23:42:05.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:05 smithi136 ceph-mon[33050]: osd.11 [v2:172.21.15.136:6824/1135995193,v1:172.21.15.136:6825/1135995193] boot 2024-04-26T23:42:05.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:05 smithi136 ceph-mon[33050]: osdmap e80: 12 total, 12 up, 12 in 2024-04-26T23:42:05.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:05 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T23:42:05.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:05 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/1257812578' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T23:42:05.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:05 smithi076 ceph-mon[32774]: purged_snaps scrub starts 2024-04-26T23:42:05.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:05 smithi076 ceph-mon[32774]: purged_snaps scrub ok 2024-04-26T23:42:05.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:05 smithi076 ceph-mon[32774]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-04-26T23:42:05.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:05 smithi076 ceph-mon[32774]: osd.11 [v2:172.21.15.136:6824/1135995193,v1:172.21.15.136:6825/1135995193] boot 2024-04-26T23:42:05.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:05 smithi076 ceph-mon[32774]: osdmap e80: 12 total, 12 up, 12 in 2024-04-26T23:42:05.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:05 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T23:42:05.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:05 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/1257812578' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T23:42:06.117 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":80,"num_osds":12,"num_up_osds":12,"osd_up_since":1714174924,"num_in_osds":12,"osd_in_since":1714174904,"num_remapped_pgs":0} 2024-04-26T23:42:06.118 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T23:42:06.554 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:42:06.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:06 smithi072 ceph-mon[28184]: osdmap e81: 12 total, 12 up, 12 in 2024-04-26T23:42:06.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:06 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:06.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:06 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:06.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:06 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:42:06.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:06 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:42:06.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:06 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:42:06.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:06 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:42:06.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:06 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:06.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:06 smithi136 ceph-mon[33050]: osdmap e81: 12 total, 12 up, 12 in 2024-04-26T23:42:06.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:06 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:06.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:06 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:06.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:06 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:42:06.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:06 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:42:06.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:06 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:42:06.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:06 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:42:06.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:06 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:06.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:06 smithi076 ceph-mon[32774]: osdmap e81: 12 total, 12 up, 12 in 2024-04-26T23:42:06.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:06 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:06.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:06 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:06.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:06 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:42:06.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:06 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:42:06.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:06 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:42:06.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:06 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:42:06.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:06 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:07.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:07 smithi072 ceph-mon[28184]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:07.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:07 smithi072 ceph-mon[28184]: Detected new or changed devices on smithi136 2024-04-26T23:42:07.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:07 smithi072 ceph-mon[28184]: Adjusting osd_memory_target on smithi136 to 4272M 2024-04-26T23:42:07.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:07 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:42:07.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:07 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:42:07.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:07 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:07.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:07 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:42:07.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:07 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:42:07.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:07 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:42:07.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:07 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:07.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:07 smithi136 ceph-mon[33050]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:07.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:07 smithi136 ceph-mon[33050]: Detected new or changed devices on smithi136 2024-04-26T23:42:07.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:07 smithi136 ceph-mon[33050]: Adjusting osd_memory_target on smithi136 to 4272M 2024-04-26T23:42:07.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:07 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:42:07.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:07 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:42:07.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:07 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:07.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:07 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:42:07.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:07 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:42:07.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:07 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:42:07.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:07 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:07.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:07 smithi076 ceph-mon[32774]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:07.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:07 smithi076 ceph-mon[32774]: Detected new or changed devices on smithi136 2024-04-26T23:42:07.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:07 smithi076 ceph-mon[32774]: Adjusting osd_memory_target on smithi136 to 4272M 2024-04-26T23:42:07.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:07 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:42:07.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:07 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:42:07.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:07 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:07.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:07 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:42:07.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:07 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:42:07.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:07 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:42:07.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:07 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:08.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:08.764+0000 7fb70f800700 1 -- 172.21.15.72:0/998115638 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb70806bad0 msgr2=0x7fb7081102e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:08.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:08.764+0000 7fb70f800700 1 --2- 172.21.15.72:0/998115638 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb70806bad0 0x7fb7081102e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb70000b0a0 tx=0x7fb70000b3b0 comp rx=0 tx=0).stop 2024-04-26T23:42:08.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:08.764+0000 7fb70f800700 1 -- 172.21.15.72:0/998115638 shutdown_connections 2024-04-26T23:42:08.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:08.764+0000 7fb70f800700 1 --2- 172.21.15.72:0/998115638 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb70806bad0 0x7fb7081102e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:08.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:08.764+0000 7fb70f800700 1 --2- 172.21.15.72:0/998115638 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb7081094a0 0x7fb708109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:08.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:08.764+0000 7fb70f800700 1 --2- 172.21.15.72:0/998115638 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb70806b130 0x7fb70806b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:08.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:08.764+0000 7fb70f800700 1 -- 172.21.15.72:0/998115638 >> 172.21.15.72:0/998115638 conn(0x7fb7080792d0 msgr2=0x7fb7080796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:08.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:08.764+0000 7fb70f800700 1 -- 172.21.15.72:0/998115638 shutdown_connections 2024-04-26T23:42:08.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:08.764+0000 7fb70f800700 1 -- 172.21.15.72:0/998115638 wait complete. 2024-04-26T23:42:08.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:08.766+0000 7fb70f800700 1 Processor -- start 2024-04-26T23:42:08.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:08.766+0000 7fb70f800700 1 -- start start 2024-04-26T23:42:08.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:08.766+0000 7fb70f800700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb70806b130 0x7fb7081023f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:08.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:08.766+0000 7fb70f800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb70806bad0 0x7fb708102930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:08.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:08.766+0000 7fb70f800700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7081094a0 0x7fb708102e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:08.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:08.766+0000 7fb70f800700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fb7081135d0 con 0x7fb7081094a0 2024-04-26T23:42:08.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:08.766+0000 7fb70f800700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb708113450 con 0x7fb70806bad0 2024-04-26T23:42:08.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:08.766+0000 7fb70f800700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fb708113750 con 0x7fb70806b130 2024-04-26T23:42:08.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:08.766+0000 7fb70ce00700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb70806b130 0x7fb7081023f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:08.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:08.767+0000 7fb70d800700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7081094a0 0x7fb708102e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:08.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:08.767+0000 7fb707e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb70806bad0 0x7fb708102930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:08.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:08.767+0000 7fb70ce00700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb70806b130 0x7fb7081023f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.72:59508/0 (socket says 172.21.15.72:59508) 2024-04-26T23:42:08.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:08.767+0000 7fb70ce00700 1 -- 172.21.15.72:0/1562681993 learned_addr learned my addr 172.21.15.72:0/1562681993 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:42:08.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:08.767+0000 7fb707e00700 1 -- 172.21.15.72:0/1562681993 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb70806b130 msgr2=0x7fb7081023f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:08.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:08.767+0000 7fb707e00700 1 --2- 172.21.15.72:0/1562681993 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb70806b130 0x7fb7081023f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:08.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:08.767+0000 7fb707e00700 1 -- 172.21.15.72:0/1562681993 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7081094a0 msgr2=0x7fb708102e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:08.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:08.767+0000 7fb707e00700 1 --2- 172.21.15.72:0/1562681993 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7081094a0 0x7fb708102e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:08.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:08.767+0000 7fb707e00700 1 -- 172.21.15.72:0/1562681993 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7081a7f80 con 0x7fb70806bad0 2024-04-26T23:42:08.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:08.767+0000 7fb707e00700 1 --2- 172.21.15.72:0/1562681993 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb70806bad0 0x7fb708102930 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb6f400e9f0 tx=0x7fb6f400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:08.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:08.767+0000 7fb705600700 1 -- 172.21.15.72:0/1562681993 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f4004220 con 0x7fb70806bad0 2024-04-26T23:42:08.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:08.767+0000 7fb705600700 1 -- 172.21.15.72:0/1562681993 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6f40043c0 con 0x7fb70806bad0 2024-04-26T23:42:08.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:08.767+0000 7fb70f800700 1 -- 172.21.15.72:0/1562681993 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7081a8270 con 0x7fb70806bad0 2024-04-26T23:42:08.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:08.768+0000 7fb705600700 1 -- 172.21.15.72:0/1562681993 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f4005020 con 0x7fb70806bad0 2024-04-26T23:42:08.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:08.768+0000 7fb70f800700 1 -- 172.21.15.72:0/1562681993 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7081a8760 con 0x7fb70806bad0 2024-04-26T23:42:08.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:08.769+0000 7fb70f800700 1 -- 172.21.15.72:0/1562681993 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb70806a110 con 0x7fb70806bad0 2024-04-26T23:42:08.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:08.771+0000 7fb705600700 1 -- 172.21.15.72:0/1562681993 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb6f4004750 con 0x7fb70806bad0 2024-04-26T23:42:08.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:08.772+0000 7fb705600700 1 --2- 172.21.15.72:0/1562681993 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fb6f00a3c40 0x7fb6f00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:08.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:08.772+0000 7fb705600700 1 -- 172.21.15.72:0/1562681993 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb6f40cff80 con 0x7fb70806bad0 2024-04-26T23:42:08.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:08.772+0000 7fb70ce00700 1 --2- 172.21.15.72:0/1562681993 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fb6f00a3c40 0x7fb6f00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:08.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:08.773+0000 7fb70ce00700 1 --2- 172.21.15.72:0/1562681993 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fb6f00a3c40 0x7fb6f00a6100 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc009ca0 tx=0x7fb6fc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:08.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:08.777+0000 7fb705600700 1 -- 172.21.15.72:0/1562681993 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb6f409cd60 con 0x7fb70806bad0 2024-04-26T23:42:09.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:09.003+0000 7fb70f800700 1 -- 172.21.15.72:0/1562681993 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb70802e490 con 0x7fb70806bad0 2024-04-26T23:42:09.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:09.005+0000 7fb705600700 1 -- 172.21.15.72:0/1562681993 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18409 (secure 0 0 0) 0x7fb6f409c660 con 0x7fb70806bad0 2024-04-26T23:42:09.010 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:42:09.010 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":81,"fsid":"3c8453ae-0425-11ef-bc93-c7b262605968","created":"2024-04-26T23:33:48.088300+0000","modified":"2024-04-26T23:42:05.494742+0000","last_up_change":"2024-04-26T23:42:04.482891+0000","last_in_change":"2024-04-26T23:41:44.784415+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:37:44.428727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e20c9ebf-56f6-47c1-a11b-e1e615af042b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6803","nonce":1639298875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6805","nonce":1639298875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6809","nonce":1639298875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6807","nonce":1639298875}]},"public_addr":"172.21.15.72:6803/1639298875","cluster_addr":"172.21.15.72:6805/1639298875","heartbeat_back_addr":"172.21.15.72:6809/1639298875","heartbeat_front_addr":"172.21.15.72:6807/1639298875","state":["exists","up"]},{"osd":1,"uuid":"e6cac469-780c-4d66-b560-b686072371fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6801","nonce":4185342193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6803","nonce":4185342193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6807","nonce":4185342193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6805","nonce":4185342193}]},"public_addr":"172.21.15.76:6801/4185342193","cluster_addr":"172.21.15.76:6803/4185342193","heartbeat_back_addr":"172.21.15.76:6807/4185342193","heartbeat_front_addr":"172.21.15.76:6805/4185342193","state":["exists","up"]},{"osd":2,"uuid":"838a49fc-18de-446c-8336-67b3e05da231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6801","nonce":3033860799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6803","nonce":3033860799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6807","nonce":3033860799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6805","nonce":3033860799}]},"public_addr":"172.21.15.136:6801/3033860799","cluster_addr":"172.21.15.136:6803/3033860799","heartbeat_back_addr":"172.21.15.136:6807/3033860799","heartbeat_front_addr":"172.21.15.136:6805/3033860799","state":["exists","up"]},{"osd":3,"uuid":"e6da89be-5ba7-48a4-9306-f75498660e0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6811","nonce":2071107815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6813","nonce":2071107815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6817","nonce":2071107815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6815","nonce":2071107815}]},"public_addr":"172.21.15.72:6811/2071107815","cluster_addr":"172.21.15.72:6813/2071107815","heartbeat_back_addr":"172.21.15.72:6817/2071107815","heartbeat_front_addr":"172.21.15.72:6815/2071107815","state":["exists","up"]},{"osd":4,"uuid":"0b17c491-d798-488f-892d-521ef2fbc51e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6809","nonce":147449238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6811","nonce":147449238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6815","nonce":147449238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6813","nonce":147449238}]},"public_addr":"172.21.15.76:6809/147449238","cluster_addr":"172.21.15.76:6811/147449238","heartbeat_back_addr":"172.21.15.76:6815/147449238","heartbeat_front_addr":"172.21.15.76:6813/147449238","state":["exists","up"]},{"osd":5,"uuid":"c7e887b3-ab38-4608-9bfd-1ed49956e8af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6809","nonce":2320149212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6811","nonce":2320149212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6815","nonce":2320149212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6813","nonce":2320149212}]},"public_addr":"172.21.15.136:6809/2320149212","cluster_addr":"172.21.15.136:6811/2320149212","heartbeat_back_addr":"172.21.15.136:6815/2320149212","heartbeat_front_addr":"172.21.15.136:6813/2320149212","state":["exists","up"]},{"osd":6,"uuid":"f0bd604b-0f57-4a38-9295-eb200264f1ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6819","nonce":389499207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6821","nonce":389499207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6825","nonce":389499207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6823","nonce":389499207}]},"public_addr":"172.21.15.72:6819/389499207","cluster_addr":"172.21.15.72:6821/389499207","heartbeat_back_addr":"172.21.15.72:6825/389499207","heartbeat_front_addr":"172.21.15.72:6823/389499207","state":["exists","up"]},{"osd":7,"uuid":"1879d4e4-038e-4cd6-a3ec-3a940bcc3797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6817","nonce":375888575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6819","nonce":375888575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6823","nonce":375888575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6821","nonce":375888575}]},"public_addr":"172.21.15.76:6817/375888575","cluster_addr":"172.21.15.76:6819/375888575","heartbeat_back_addr":"172.21.15.76:6823/375888575","heartbeat_front_addr":"172.21.15.76:6821/375888575","state":["exists","up"]},{"osd":8,"uuid":"6f404062-97e7-4ccb-a582-76dc1ffae548","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6817","nonce":3446774052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6819","nonce":3446774052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6823","nonce":3446774052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6821","nonce":3446774052}]},"public_addr":"172.21.15.136:6817/3446774052","cluster_addr":"172.21.15.136:6819/3446774052","heartbeat_back_addr":"172.21.15.136:6823/3446774052","heartbeat_front_addr":"172.21.15.136:6821/3446774052","state":["exists","up"]},{"osd":9,"uuid":"2989ca68-3164-4667-9bcf-230dcbb7b8f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6827","nonce":3423789152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6829","nonce":3423789152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6832","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6833","nonce":3423789152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6831","nonce":3423789152}]},"public_addr":"172.21.15.72:6827/3423789152","cluster_addr":"172.21.15.72:6829/3423789152","heartbeat_back_addr":"172.21.15.72:6833/3423789152","heartbeat_front_addr":"172.21.15.72:6831/3423789152","state":["exists","up"]},{"osd":10,"uuid":"9ec6d46a-0913-4208-97b5-167188ee8cb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6825","nonce":1120991574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6827","nonce":1120991574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6831","nonce":1120991574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6829","nonce":1120991574}]},"public_addr":"172.21.15.76:6825/1120991574","cluster_addr":"172.21.15.76:6827/1120991574","heartbeat_back_addr":"172.21.15.76:6831/1120991574","heartbeat_front_addr":"172.21.15.76:6829/1120991574","state":["exists","up"]},{"osd":11,"uuid":"bdd0774f-6e58-4918-b4cc-afd4f1118d3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6825","nonce":1135995193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6827","nonce":1135995193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6831","nonce":1135995193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6829","nonce":1135995193}]},"public_addr":"172.21.15.136:6825/1135995193","cluster_addr":"172.21.15.136:6827/1135995193","heartbeat_back_addr":"172.21.15.136:6831/1135995193","heartbeat_front_addr":"172.21.15.136:6829/1135995193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:36:52.128788+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:37:17.523654+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:37:41.634677+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:38:15.060797+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:38:41.234111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:39:06.783560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:39:40.884386+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:40:08.340465+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:40:34.344858+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:41:08.525309+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:41:36.782624+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:42:02.756888+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:0/3625071330":"2024-04-27T23:34:58.266218+0000","172.21.15.72:6801/2903979134":"2024-04-27T23:34:58.266218+0000","172.21.15.72:6800/2903979134":"2024-04-27T23:34:58.266218+0000","172.21.15.72:0/1519142576":"2024-04-27T23:34:58.266218+0000","172.21.15.72:0/3669626460":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/945623297":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/3479150073":"2024-04-27T23:34:24.233319+0000","172.21.15.72:6800/4175896290":"2024-04-27T23:34:24.233319+0000","172.21.15.72:6801/4175896290":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/3783175012":"2024-04-27T23:34:58.266218+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:42:09.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:09.008+0000 7fb70f800700 1 -- 172.21.15.72:0/1562681993 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fb6f00a3c40 msgr2=0x7fb6f00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:09.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:09.008+0000 7fb70f800700 1 --2- 172.21.15.72:0/1562681993 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fb6f00a3c40 0x7fb6f00a6100 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc009ca0 tx=0x7fb6fc009260 comp rx=0 tx=0).stop 2024-04-26T23:42:09.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:09.008+0000 7fb70f800700 1 -- 172.21.15.72:0/1562681993 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb70806bad0 msgr2=0x7fb708102930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:09.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:09.008+0000 7fb70f800700 1 --2- 172.21.15.72:0/1562681993 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb70806bad0 0x7fb708102930 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb6f400e9f0 tx=0x7fb6f400ed00 comp rx=0 tx=0).stop 2024-04-26T23:42:09.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:09.008+0000 7fb70f800700 1 -- 172.21.15.72:0/1562681993 shutdown_connections 2024-04-26T23:42:09.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:09.008+0000 7fb70f800700 1 --2- 172.21.15.72:0/1562681993 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fb6f00a3c40 0x7fb6f00a6100 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:09.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:09.008+0000 7fb70f800700 1 --2- 172.21.15.72:0/1562681993 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7081094a0 0x7fb708102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:09.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:09.008+0000 7fb70f800700 1 --2- 172.21.15.72:0/1562681993 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb70806b130 0x7fb7081023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:09.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:09.008+0000 7fb70f800700 1 --2- 172.21.15.72:0/1562681993 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb70806bad0 0x7fb708102930 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:09.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:09.008+0000 7fb70f800700 1 -- 172.21.15.72:0/1562681993 >> 172.21.15.72:0/1562681993 conn(0x7fb7080792d0 msgr2=0x7fb708100370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:09.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:09.008+0000 7fb70f800700 1 -- 172.21.15.72:0/1562681993 shutdown_connections 2024-04-26T23:42:09.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:09.009+0000 7fb70f800700 1 -- 172.21.15.72:0/1562681993 wait complete. 2024-04-26T23:42:09.735 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:09 smithi072 ceph-mon[28184]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:09.736 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:09 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/1562681993' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:42:09.787 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:37:44.428727+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T23:42:09.787 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-26T23:42:09.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:09 smithi136 ceph-mon[33050]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:09.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:09 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/1562681993' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:42:09.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:09 smithi076 ceph-mon[32774]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:09.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:09 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/1562681993' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:42:10.225 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:42:11.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:11 smithi072 ceph-mon[28184]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:11.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:11 smithi136 ceph-mon[33050]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:11.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:11 smithi076 ceph-mon[32774]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:12.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.540+0000 7f9aa3800700 1 -- 172.21.15.72:0/1284591937 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a9c06b130 msgr2=0x7f9a9c06b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:12.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.540+0000 7f9aa3800700 1 --2- 172.21.15.72:0/1284591937 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a9c06b130 0x7f9a9c06b590 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9a88009990 tx=0x7f9a88009ca0 comp rx=0 tx=0).stop 2024-04-26T23:42:12.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.540+0000 7f9aa3800700 1 -- 172.21.15.72:0/1284591937 shutdown_connections 2024-04-26T23:42:12.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.540+0000 7f9aa3800700 1 --2- 172.21.15.72:0/1284591937 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9a9c06bad0 0x7f9a9c1102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:12.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.540+0000 7f9aa3800700 1 --2- 172.21.15.72:0/1284591937 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a9c1094a0 0x7f9a9c109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:12.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.540+0000 7f9aa3800700 1 --2- 172.21.15.72:0/1284591937 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a9c06b130 0x7f9a9c06b590 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:12.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.540+0000 7f9aa3800700 1 -- 172.21.15.72:0/1284591937 >> 172.21.15.72:0/1284591937 conn(0x7f9a9c0792d0 msgr2=0x7f9a9c0796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:12.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.540+0000 7f9aa3800700 1 -- 172.21.15.72:0/1284591937 shutdown_connections 2024-04-26T23:42:12.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.540+0000 7f9aa3800700 1 -- 172.21.15.72:0/1284591937 wait complete. 2024-04-26T23:42:12.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.541+0000 7f9aa3800700 1 Processor -- start 2024-04-26T23:42:12.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.541+0000 7f9aa3800700 1 -- start start 2024-04-26T23:42:12.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.542+0000 7f9aa3800700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a9c06b130 0x7f9a9c1a1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:12.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.542+0000 7f9aa3800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9a9c06bad0 0x7f9a9c1a1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:12.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.542+0000 7f9aa3800700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a9c1094a0 0x7f9a9c19bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:12.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.542+0000 7f9aa3800700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9c1135b0 con 0x7f9a9c06b130 2024-04-26T23:42:12.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.542+0000 7f9aa3800700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9c113430 con 0x7f9a9c06bad0 2024-04-26T23:42:12.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.542+0000 7f9aa3800700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9c113730 con 0x7f9a9c1094a0 2024-04-26T23:42:12.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.542+0000 7f9aa0e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a9c06b130 0x7f9a9c1a1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:12.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.542+0000 7f9aa0e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a9c06b130 0x7f9a9c1a1a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:53158/0 (socket says 172.21.15.72:53158) 2024-04-26T23:42:12.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.542+0000 7f9aa0e00700 1 -- 172.21.15.72:0/4058687421 learned_addr learned my addr 172.21.15.72:0/4058687421 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:42:12.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.542+0000 7f9a9be00700 1 --2- 172.21.15.72:0/4058687421 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9a9c06bad0 0x7f9a9c1a1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:12.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.542+0000 7f9aa1800700 1 --2- 172.21.15.72:0/4058687421 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a9c1094a0 0x7f9a9c19bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:12.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.543+0000 7f9aa0e00700 1 -- 172.21.15.72:0/4058687421 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a9c1094a0 msgr2=0x7f9a9c19bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:12.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.543+0000 7f9aa0e00700 1 --2- 172.21.15.72:0/4058687421 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a9c1094a0 0x7f9a9c19bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:12.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.543+0000 7f9aa0e00700 1 -- 172.21.15.72:0/4058687421 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9a9c06bad0 msgr2=0x7f9a9c1a1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:12.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.543+0000 7f9aa0e00700 1 --2- 172.21.15.72:0/4058687421 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9a9c06bad0 0x7f9a9c1a1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:12.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.543+0000 7f9aa0e00700 1 -- 172.21.15.72:0/4058687421 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a9c19c410 con 0x7f9a9c06b130 2024-04-26T23:42:12.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.543+0000 7f9aa0e00700 1 --2- 172.21.15.72:0/4058687421 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a9c06b130 0x7f9a9c1a1a60 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9a9000e9c0 tx=0x7f9a9000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:12.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.543+0000 7f9a99600700 1 -- 172.21.15.72:0/4058687421 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a9000cbd0 con 0x7f9a9c06b130 2024-04-26T23:42:12.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.543+0000 7f9a99600700 1 -- 172.21.15.72:0/4058687421 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a9000ee60 con 0x7f9a9c06b130 2024-04-26T23:42:12.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.543+0000 7f9aa3800700 1 -- 172.21.15.72:0/4058687421 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a9c19c700 con 0x7f9a9c06b130 2024-04-26T23:42:12.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.543+0000 7f9a99600700 1 -- 172.21.15.72:0/4058687421 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a900055c0 con 0x7f9a9c06b130 2024-04-26T23:42:12.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.543+0000 7f9aa3800700 1 -- 172.21.15.72:0/4058687421 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a9c1a89e0 con 0x7f9a9c06b130 2024-04-26T23:42:12.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.544+0000 7f9aa3800700 1 -- 172.21.15.72:0/4058687421 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a9c104b50 con 0x7f9a9c06b130 2024-04-26T23:42:12.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.547+0000 7f9a99600700 1 -- 172.21.15.72:0/4058687421 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9a90004750 con 0x7f9a9c06b130 2024-04-26T23:42:12.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.548+0000 7f9a99600700 1 --2- 172.21.15.72:0/4058687421 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f9a840a3c40 0x7f9a840a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:12.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.548+0000 7f9a99600700 1 -- 172.21.15.72:0/4058687421 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f9a900d01e0 con 0x7f9a9c06b130 2024-04-26T23:42:12.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.548+0000 7f9a9be00700 1 --2- 172.21.15.72:0/4058687421 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f9a840a3c40 0x7f9a840a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:12.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.549+0000 7f9a9be00700 1 --2- 172.21.15.72:0/4058687421 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f9a840a3c40 0x7f9a840a6100 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9a880098e0 tx=0x7f9a88005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:12.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.553+0000 7f9a99600700 1 -- 172.21.15.72:0/4058687421 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9a9009cfc0 con 0x7f9a9c06b130 2024-04-26T23:42:12.781 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.776+0000 7f9aa3800700 1 -- 172.21.15.72:0/4058687421 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9a9c19c930 con 0x7f9a9c06b130 2024-04-26T23:42:12.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.777+0000 7f9a99600700 1 -- 172.21.15.72:0/4058687421 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f9a9009c8c0 con 0x7f9a9c06b130 2024-04-26T23:42:12.782 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-04-26T23:42:12.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.780+0000 7f9aa3800700 1 -- 172.21.15.72:0/4058687421 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f9a840a3c40 msgr2=0x7f9a840a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:12.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.780+0000 7f9aa3800700 1 --2- 172.21.15.72:0/4058687421 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f9a840a3c40 0x7f9a840a6100 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9a880098e0 tx=0x7f9a88005960 comp rx=0 tx=0).stop 2024-04-26T23:42:12.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.780+0000 7f9aa3800700 1 -- 172.21.15.72:0/4058687421 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a9c06b130 msgr2=0x7f9a9c1a1a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:12.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.780+0000 7f9aa3800700 1 --2- 172.21.15.72:0/4058687421 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a9c06b130 0x7f9a9c1a1a60 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9a9000e9c0 tx=0x7f9a9000ecd0 comp rx=0 tx=0).stop 2024-04-26T23:42:12.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.780+0000 7f9aa3800700 1 -- 172.21.15.72:0/4058687421 shutdown_connections 2024-04-26T23:42:12.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.780+0000 7f9aa3800700 1 --2- 172.21.15.72:0/4058687421 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f9a840a3c40 0x7f9a840a6100 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:12.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.780+0000 7f9aa3800700 1 --2- 172.21.15.72:0/4058687421 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a9c1094a0 0x7f9a9c19bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:12.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.780+0000 7f9aa3800700 1 --2- 172.21.15.72:0/4058687421 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a9c06b130 0x7f9a9c1a1a60 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:12.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.780+0000 7f9aa3800700 1 --2- 172.21.15.72:0/4058687421 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9a9c06bad0 0x7f9a9c1a1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:12.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.780+0000 7f9aa3800700 1 -- 172.21.15.72:0/4058687421 >> 172.21.15.72:0/4058687421 conn(0x7f9a9c0792d0 msgr2=0x7f9a9c10e8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:12.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.781+0000 7f9aa3800700 1 -- 172.21.15.72:0/4058687421 shutdown_connections 2024-04-26T23:42:12.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:12.781+0000 7f9aa3800700 1 -- 172.21.15.72:0/4058687421 wait complete. 2024-04-26T23:42:13.614 INFO:tasks.cephadm:Adding mds.a on smithi072 2024-04-26T23:42:13.614 INFO:tasks.cephadm:Adding mds.d on smithi072 2024-04-26T23:42:13.614 INFO:tasks.cephadm:Adding mds.g on smithi072 2024-04-26T23:42:13.615 INFO:tasks.cephadm:Adding mds.j on smithi072 2024-04-26T23:42:13.615 INFO:tasks.cephadm:Adding mds.b on smithi076 2024-04-26T23:42:13.615 INFO:tasks.cephadm:Adding mds.e on smithi076 2024-04-26T23:42:13.615 INFO:tasks.cephadm:Adding mds.h on smithi076 2024-04-26T23:42:13.615 INFO:tasks.cephadm:Adding mds.k on smithi076 2024-04-26T23:42:13.615 INFO:tasks.cephadm:Adding mds.c on smithi136 2024-04-26T23:42:13.615 INFO:tasks.cephadm:Adding mds.f on smithi136 2024-04-26T23:42:13.615 INFO:tasks.cephadm:Adding mds.i on smithi136 2024-04-26T23:42:13.615 INFO:tasks.cephadm:Adding mds.l on smithi136 2024-04-26T23:42:13.615 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph orch apply mds all '12;smithi072=a;smithi072=d;smithi072=g;smithi072=j;smithi076=b;smithi076=e;smithi076=h;smithi076=k;smithi136=c;smithi136=f;smithi136=i;smithi136=l' 2024-04-26T23:42:13.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:13 smithi072 ceph-mon[28184]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:13.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:13 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/4058687421' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T23:42:13.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:13 smithi136 ceph-mon[33050]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:13.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:13 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/4058687421' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T23:42:13.870 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T23:42:13.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:13 smithi076 ceph-mon[32774]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:13.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:13 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/4058687421' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T23:42:15.662 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:15 smithi136 ceph-mon[33050]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:15.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.663+0000 7f50eb2d1700 1 -- 172.21.15.136:0/2658164529 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f50e41047e0 msgr2=0x7f50e4104c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:15.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.663+0000 7f50eb2d1700 1 --2- 172.21.15.136:0/2658164529 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f50e41047e0 0x7f50e4104c40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f50d4009990 tx=0x7f50d4009ca0 comp rx=0 tx=0).stop 2024-04-26T23:42:15.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.663+0000 7f50eb2d1700 1 -- 172.21.15.136:0/2658164529 shutdown_connections 2024-04-26T23:42:15.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.663+0000 7f50eb2d1700 1 --2- 172.21.15.136:0/2658164529 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50e4105180 0x7f50e410ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:15.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.663+0000 7f50eb2d1700 1 --2- 172.21.15.136:0/2658164529 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50e4101800 0x7f50e4101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:15.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.663+0000 7f50eb2d1700 1 --2- 172.21.15.136:0/2658164529 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f50e41047e0 0x7f50e4104c40 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:15.667 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.663+0000 7f50eb2d1700 1 -- 172.21.15.136:0/2658164529 >> 172.21.15.136:0/2658164529 conn(0x7f50e40fd160 msgr2=0x7f50e40ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:15.668 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.663+0000 7f50eb2d1700 1 -- 172.21.15.136:0/2658164529 shutdown_connections 2024-04-26T23:42:15.668 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.663+0000 7f50eb2d1700 1 -- 172.21.15.136:0/2658164529 wait complete. 2024-04-26T23:42:15.670 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.664+0000 7f50eb2d1700 1 Processor -- start 2024-04-26T23:42:15.670 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.664+0000 7f50eb2d1700 1 -- start start 2024-04-26T23:42:15.670 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.665+0000 7f50eb2d1700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50e4101800 0x7f50e4199700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:15.670 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.665+0000 7f50eb2d1700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50e41047e0 0x7f50e4199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:15.670 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.665+0000 7f50eb2d1700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f50e4105180 0x7f50e41a0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:15.670 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.665+0000 7f50eb2d1700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f50e410f0c0 con 0x7f50e41047e0 2024-04-26T23:42:15.670 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.665+0000 7f50eb2d1700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f50e410ef40 con 0x7f50e4101800 2024-04-26T23:42:15.670 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.665+0000 7f50eb2d1700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f50e410f240 con 0x7f50e4105180 2024-04-26T23:42:15.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.665+0000 7f50e986e700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f50e4105180 0x7f50e41a0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:15.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.665+0000 7f50e986e700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f50e4105180 0x7f50e41a0cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:54806/0 (socket says 172.21.15.136:54806) 2024-04-26T23:42:15.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.665+0000 7f50e986e700 1 -- 172.21.15.136:0/2299515189 learned_addr learned my addr 172.21.15.136:0/2299515189 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-04-26T23:42:15.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.665+0000 7f50e906d700 1 --2- 172.21.15.136:0/2299515189 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50e4101800 0x7f50e4199700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:15.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.665+0000 7f50e986e700 1 -- 172.21.15.136:0/2299515189 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50e4101800 msgr2=0x7f50e4199700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:15.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.665+0000 7f50e986e700 1 --2- 172.21.15.136:0/2299515189 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50e4101800 0x7f50e4199700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:15.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.665+0000 7f50e986e700 1 -- 172.21.15.136:0/2299515189 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50e41047e0 msgr2=0x7f50e4199c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:15.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.665+0000 7f50e986e700 1 --2- 172.21.15.136:0/2299515189 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50e41047e0 0x7f50e4199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:15.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.665+0000 7f50e986e700 1 -- 172.21.15.136:0/2299515189 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50e41a1360 con 0x7f50e4105180 2024-04-26T23:42:15.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.665+0000 7f50e906d700 1 --2- 172.21.15.136:0/2299515189 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50e4101800 0x7f50e4199700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:42:15.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.665+0000 7f50e986e700 1 --2- 172.21.15.136:0/2299515189 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f50e4105180 0x7f50e41a0cc0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f50dc00b9a0 tx=0x7f50dc00bd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:15.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.665+0000 7f50da7fc700 1 -- 172.21.15.136:0/2299515189 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50dc019070 con 0x7f50e4105180 2024-04-26T23:42:15.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.665+0000 7f50eb2d1700 1 -- 172.21.15.136:0/2299515189 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50e41a1650 con 0x7f50e4105180 2024-04-26T23:42:15.671 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.665+0000 7f50da7fc700 1 -- 172.21.15.136:0/2299515189 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f50dc014440 con 0x7f50e4105180 2024-04-26T23:42:15.672 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.665+0000 7f50da7fc700 1 -- 172.21.15.136:0/2299515189 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50dc00ce60 con 0x7f50e4105180 2024-04-26T23:42:15.672 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.665+0000 7f50eb2d1700 1 -- 172.21.15.136:0/2299515189 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50e41a1c60 con 0x7f50e4105180 2024-04-26T23:42:15.672 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.666+0000 7f50eb2d1700 1 -- 172.21.15.136:0/2299515189 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50e406a110 con 0x7f50e4105180 2024-04-26T23:42:15.672 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.668+0000 7f50da7fc700 1 -- 172.21.15.136:0/2299515189 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f50dc01c070 con 0x7f50e4105180 2024-04-26T23:42:15.672 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.668+0000 7f50da7fc700 1 --2- 172.21.15.136:0/2299515189 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f50d00a3c40 0x7f50d00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:15.673 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.668+0000 7f50da7fc700 1 -- 172.21.15.136:0/2299515189 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f50dc0cf860 con 0x7f50e4105180 2024-04-26T23:42:15.673 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.669+0000 7f50e906d700 1 --2- 172.21.15.136:0/2299515189 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f50d00a3c40 0x7f50d00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:15.673 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.669+0000 7f50e906d700 1 --2- 172.21.15.136:0/2299515189 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f50d00a3c40 0x7f50d00a6100 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f50e406b9b0 tx=0x7f50e0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:15.675 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.672+0000 7f50da7fc700 1 -- 172.21.15.136:0/2299515189 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f50dc09c770 con 0x7f50e4105180 2024-04-26T23:42:15.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:15 smithi072 ceph-mon[28184]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:15.880 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.877+0000 7f50eb2d1700 1 -- 172.21.15.136:0/2299515189 --> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi072=a;smithi072=d;smithi072=g;smithi072=j;smithi076=b;smithi076=e;smithi076=h;smithi076=k;smithi136=c;smithi136=f;smithi136=i;smithi136=l", "target": ["mon-mgr", ""]}) -- 0x7f50e4063a60 con 0x7f50d00a3c40 2024-04-26T23:42:15.897 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.893+0000 7f50da7fc700 1 -- 172.21.15.136:0/2299515189 <== mgr.14152 v2:172.21.15.72:6800/4281177709 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f50e4063a60 con 0x7f50d00a3c40 2024-04-26T23:42:15.897 INFO:teuthology.orchestra.run.smithi136.stdout:Scheduled mds.all update... 2024-04-26T23:42:15.900 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.896+0000 7f50eb2d1700 1 -- 172.21.15.136:0/2299515189 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f50d00a3c40 msgr2=0x7f50d00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:15.900 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.896+0000 7f50eb2d1700 1 --2- 172.21.15.136:0/2299515189 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f50d00a3c40 0x7f50d00a6100 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f50e406b9b0 tx=0x7f50e0008040 comp rx=0 tx=0).stop 2024-04-26T23:42:15.900 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.896+0000 7f50eb2d1700 1 -- 172.21.15.136:0/2299515189 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f50e4105180 msgr2=0x7f50e41a0cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:15.900 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.896+0000 7f50eb2d1700 1 --2- 172.21.15.136:0/2299515189 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f50e4105180 0x7f50e41a0cc0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f50dc00b9a0 tx=0x7f50dc00bd60 comp rx=0 tx=0).stop 2024-04-26T23:42:15.900 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.896+0000 7f50eb2d1700 1 -- 172.21.15.136:0/2299515189 shutdown_connections 2024-04-26T23:42:15.900 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.896+0000 7f50eb2d1700 1 --2- 172.21.15.136:0/2299515189 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f50d00a3c40 0x7f50d00a6100 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:15.900 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.896+0000 7f50eb2d1700 1 --2- 172.21.15.136:0/2299515189 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f50e4105180 0x7f50e41a0cc0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:15.900 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.896+0000 7f50eb2d1700 1 --2- 172.21.15.136:0/2299515189 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50e4101800 0x7f50e4199700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:15.900 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.896+0000 7f50eb2d1700 1 --2- 172.21.15.136:0/2299515189 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50e41047e0 0x7f50e4199c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:15.901 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.896+0000 7f50eb2d1700 1 -- 172.21.15.136:0/2299515189 >> 172.21.15.136:0/2299515189 conn(0x7f50e40fd160 msgr2=0x7f50e4107ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:15.901 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.896+0000 7f50eb2d1700 1 -- 172.21.15.136:0/2299515189 shutdown_connections 2024-04-26T23:42:15.901 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-26T23:42:15.896+0000 7f50eb2d1700 1 -- 172.21.15.136:0/2299515189 wait complete. 2024-04-26T23:42:15.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:15 smithi076 ceph-mon[32774]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:16.560 DEBUG:teuthology.orchestra.run.smithi072:mds.a> sudo journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mds.a.service 2024-04-26T23:42:16.568 DEBUG:teuthology.orchestra.run.smithi072:mds.d> sudo journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mds.d.service 2024-04-26T23:42:16.613 DEBUG:teuthology.orchestra.run.smithi072:mds.g> sudo journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mds.g.service 2024-04-26T23:42:16.660 DEBUG:teuthology.orchestra.run.smithi072:mds.j> sudo journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mds.j.service 2024-04-26T23:42:16.704 DEBUG:teuthology.orchestra.run.smithi076:mds.b> sudo journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mds.b.service 2024-04-26T23:42:16.707 DEBUG:teuthology.orchestra.run.smithi076:mds.e> sudo journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mds.e.service 2024-04-26T23:42:16.749 DEBUG:teuthology.orchestra.run.smithi076:mds.h> sudo journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mds.h.service 2024-04-26T23:42:16.792 DEBUG:teuthology.orchestra.run.smithi076:mds.k> sudo journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mds.k.service 2024-04-26T23:42:16.836 DEBUG:teuthology.orchestra.run.smithi136:mds.c> sudo journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mds.c.service 2024-04-26T23:42:16.838 DEBUG:teuthology.orchestra.run.smithi136:mds.f> sudo journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mds.f.service 2024-04-26T23:42:16.881 DEBUG:teuthology.orchestra.run.smithi136:mds.i> sudo journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mds.i.service 2024-04-26T23:42:16.924 DEBUG:teuthology.orchestra.run.smithi136:mds.l> sudo journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mds.l.service 2024-04-26T23:42:16.967 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-26T23:42:16.968 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:42:17.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.165+0000 7f6aeb800640 1 -- 172.21.15.72:0/3541742389 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6ae4100620 msgr2=0x7f6ae4100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:17.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.165+0000 7f6aeb800640 1 --2- 172.21.15.72:0/3541742389 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6ae4100620 0x7f6ae4100aa0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6ad0009900 tx=0x7f6ad002a410 comp rx=0 tx=0).stop 2024-04-26T23:42:17.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.165+0000 7f6aeb800640 1 -- 172.21.15.72:0/3541742389 shutdown_connections 2024-04-26T23:42:17.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.165+0000 7f6aeb800640 1 --2- 172.21.15.72:0/3541742389 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ae4100fe0 0x7f6ae4102bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:17.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.165+0000 7f6aeb800640 1 --2- 172.21.15.72:0/3541742389 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6ae4100620 0x7f6ae4100aa0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:17.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.165+0000 7f6aeb800640 1 --2- 172.21.15.72:0/3541742389 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6ae4101b50 0x7f6ae4101f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:17.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.165+0000 7f6aeb800640 1 -- 172.21.15.72:0/3541742389 >> 172.21.15.72:0/3541742389 conn(0x7f6ae40fc120 msgr2=0x7f6ae40fe540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:17.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.165+0000 7f6aeb800640 1 -- 172.21.15.72:0/3541742389 shutdown_connections 2024-04-26T23:42:17.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.165+0000 7f6aeb800640 1 -- 172.21.15.72:0/3541742389 wait complete. 2024-04-26T23:42:17.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.166+0000 7f6aeb800640 1 Processor -- start 2024-04-26T23:42:17.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.166+0000 7f6aeb800640 1 -- start start 2024-04-26T23:42:17.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.167+0000 7f6aeb800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6ae4100620 0x7f6ae419ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:17.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.167+0000 7f6aeb800640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ae4100fe0 0x7f6ae419b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:17.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.167+0000 7f6aeb800640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6ae4101b50 0x7f6ae41a2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:17.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.167+0000 7f6aeb800640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae4110700 con 0x7f6ae4100fe0 2024-04-26T23:42:17.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.167+0000 7f6aeb800640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae4110580 con 0x7f6ae4100620 2024-04-26T23:42:17.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.167+0000 7f6aeb800640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae4110880 con 0x7f6ae4101b50 2024-04-26T23:42:17.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.167+0000 7f6ae3e00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ae4100fe0 0x7f6ae419b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:17.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.167+0000 7f6ae8e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6ae4100620 0x7f6ae419ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:17.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.167+0000 7f6ae9800640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6ae4101b50 0x7f6ae41a2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:17.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.167+0000 7f6ae9800640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6ae4101b50 0x7f6ae41a2320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.72:36596/0 (socket says 172.21.15.72:36596) 2024-04-26T23:42:17.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.167+0000 7f6ae8e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6ae4100620 0x7f6ae419ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:59052/0 (socket says 172.21.15.72:59052) 2024-04-26T23:42:17.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.167+0000 7f6ae9800640 1 -- 172.21.15.72:0/4138864848 learned_addr learned my addr 172.21.15.72:0/4138864848 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:42:17.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.167+0000 7f6ae9800640 1 -- 172.21.15.72:0/4138864848 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6ae4100620 msgr2=0x7f6ae419ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:17.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.167+0000 7f6ae9800640 1 --2- 172.21.15.72:0/4138864848 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6ae4100620 0x7f6ae419ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:17.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.167+0000 7f6ae9800640 1 -- 172.21.15.72:0/4138864848 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ae4100fe0 msgr2=0x7f6ae419b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:17.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.167+0000 7f6ae9800640 1 --2- 172.21.15.72:0/4138864848 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ae4100fe0 0x7f6ae419b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:17.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.167+0000 7f6ae9800640 1 -- 172.21.15.72:0/4138864848 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ae41a2990 con 0x7f6ae4101b50 2024-04-26T23:42:17.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.168+0000 7f6ae9800640 1 --2- 172.21.15.72:0/4138864848 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6ae4101b50 0x7f6ae41a2320 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6adc00eb10 tx=0x7f6adc00c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:17.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.168+0000 7f6ae1600640 1 -- 172.21.15.72:0/4138864848 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6adc004b20 con 0x7f6ae4101b50 2024-04-26T23:42:17.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.168+0000 7f6aeb800640 1 -- 172.21.15.72:0/4138864848 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ae41a2c80 con 0x7f6ae4101b50 2024-04-26T23:42:17.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.168+0000 7f6ae1600640 1 -- 172.21.15.72:0/4138864848 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6adc004cc0 con 0x7f6ae4101b50 2024-04-26T23:42:17.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.168+0000 7f6ae1600640 1 -- 172.21.15.72:0/4138864848 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6adc010670 con 0x7f6ae4101b50 2024-04-26T23:42:17.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.168+0000 7f6aeb800640 1 -- 172.21.15.72:0/4138864848 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ae41a31a0 con 0x7f6ae4101b50 2024-04-26T23:42:17.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.169+0000 7f6aeb800640 1 -- 172.21.15.72:0/4138864848 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ae410ce70 con 0x7f6ae4101b50 2024-04-26T23:42:17.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.171+0000 7f6ae1600640 1 -- 172.21.15.72:0/4138864848 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6adc005020 con 0x7f6ae4101b50 2024-04-26T23:42:17.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.173+0000 7f6ae1600640 1 --2- 172.21.15.72:0/4138864848 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6acc0a3bb0 0x7f6acc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:17.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.173+0000 7f6ae1600640 1 -- 172.21.15.72:0/4138864848 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f6adc0d0560 con 0x7f6ae4101b50 2024-04-26T23:42:17.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.173+0000 7f6ae8e00640 1 --2- 172.21.15.72:0/4138864848 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6acc0a3bb0 0x7f6acc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:17.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.174+0000 7f6ae8e00640 1 --2- 172.21.15.72:0/4138864848 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6acc0a3bb0 0x7f6acc0a6070 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8009bd0 tx=0x7f6ad8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:17.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:16 smithi076 ceph-mon[32774]: from='client.24421 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi072=a;smithi072=d;smithi072=g;smithi072=j;smithi076=b;smithi076=e;smithi076=h;smithi076=k;smithi136=c;smithi136=f;smithi136=i;smithi136=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:42:17.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:16 smithi076 ceph-mon[32774]: Saving service mds.all spec with placement smithi072=a;smithi072=d;smithi072=g;smithi072=j;smithi076=b;smithi076=e;smithi076=h;smithi076=k;smithi136=c;smithi136=f;smithi136=i;smithi136=l;count:12 2024-04-26T23:42:17.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:16 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:17.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:16 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:42:17.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:16 smithi076 ceph-mon[32774]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:17.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.176+0000 7f6ae1600640 1 -- 172.21.15.72:0/4138864848 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6adc09d340 con 0x7f6ae4101b50 2024-04-26T23:42:17.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:16 smithi072 ceph-mon[28184]: from='client.24421 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi072=a;smithi072=d;smithi072=g;smithi072=j;smithi076=b;smithi076=e;smithi076=h;smithi076=k;smithi136=c;smithi136=f;smithi136=i;smithi136=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:42:17.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:16 smithi072 ceph-mon[28184]: Saving service mds.all spec with placement smithi072=a;smithi072=d;smithi072=g;smithi072=j;smithi076=b;smithi076=e;smithi076=h;smithi076=k;smithi136=c;smithi136=f;smithi136=i;smithi136=l;count:12 2024-04-26T23:42:17.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:16 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:17.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:16 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:42:17.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:16 smithi072 ceph-mon[28184]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:17.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:16 smithi136 ceph-mon[33050]: from='client.24421 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi072=a;smithi072=d;smithi072=g;smithi072=j;smithi076=b;smithi076=e;smithi076=h;smithi076=k;smithi136=c;smithi136=f;smithi136=i;smithi136=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:42:17.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:16 smithi136 ceph-mon[33050]: Saving service mds.all spec with placement smithi072=a;smithi072=d;smithi072=g;smithi072=j;smithi076=b;smithi076=e;smithi076=h;smithi076=k;smithi136=c;smithi136=f;smithi136=i;smithi136=l;count:12 2024-04-26T23:42:17.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:16 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:17.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:16 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:42:17.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:16 smithi136 ceph-mon[33050]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:17.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.383+0000 7f6aeb800640 1 -- 172.21.15.72:0/4138864848 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6ae4063530 con 0x7f6ae4101b50 2024-04-26T23:42:17.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.384+0000 7f6ae1600640 1 -- 172.21.15.72:0/4138864848 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18409 (secure 0 0 0) 0x7f6adc09cc40 con 0x7f6ae4101b50 2024-04-26T23:42:17.388 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:42:17.388 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":81,"fsid":"3c8453ae-0425-11ef-bc93-c7b262605968","created":"2024-04-26T23:33:48.088300+0000","modified":"2024-04-26T23:42:05.494742+0000","last_up_change":"2024-04-26T23:42:04.482891+0000","last_in_change":"2024-04-26T23:41:44.784415+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:37:44.428727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e20c9ebf-56f6-47c1-a11b-e1e615af042b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6803","nonce":1639298875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6805","nonce":1639298875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6809","nonce":1639298875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6807","nonce":1639298875}]},"public_addr":"172.21.15.72:6803/1639298875","cluster_addr":"172.21.15.72:6805/1639298875","heartbeat_back_addr":"172.21.15.72:6809/1639298875","heartbeat_front_addr":"172.21.15.72:6807/1639298875","state":["exists","up"]},{"osd":1,"uuid":"e6cac469-780c-4d66-b560-b686072371fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6801","nonce":4185342193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6803","nonce":4185342193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6807","nonce":4185342193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6805","nonce":4185342193}]},"public_addr":"172.21.15.76:6801/4185342193","cluster_addr":"172.21.15.76:6803/4185342193","heartbeat_back_addr":"172.21.15.76:6807/4185342193","heartbeat_front_addr":"172.21.15.76:6805/4185342193","state":["exists","up"]},{"osd":2,"uuid":"838a49fc-18de-446c-8336-67b3e05da231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6801","nonce":3033860799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6803","nonce":3033860799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6807","nonce":3033860799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6805","nonce":3033860799}]},"public_addr":"172.21.15.136:6801/3033860799","cluster_addr":"172.21.15.136:6803/3033860799","heartbeat_back_addr":"172.21.15.136:6807/3033860799","heartbeat_front_addr":"172.21.15.136:6805/3033860799","state":["exists","up"]},{"osd":3,"uuid":"e6da89be-5ba7-48a4-9306-f75498660e0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6811","nonce":2071107815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6813","nonce":2071107815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6817","nonce":2071107815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6815","nonce":2071107815}]},"public_addr":"172.21.15.72:6811/2071107815","cluster_addr":"172.21.15.72:6813/2071107815","heartbeat_back_addr":"172.21.15.72:6817/2071107815","heartbeat_front_addr":"172.21.15.72:6815/2071107815","state":["exists","up"]},{"osd":4,"uuid":"0b17c491-d798-488f-892d-521ef2fbc51e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6809","nonce":147449238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6811","nonce":147449238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6815","nonce":147449238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6813","nonce":147449238}]},"public_addr":"172.21.15.76:6809/147449238","cluster_addr":"172.21.15.76:6811/147449238","heartbeat_back_addr":"172.21.15.76:6815/147449238","heartbeat_front_addr":"172.21.15.76:6813/147449238","state":["exists","up"]},{"osd":5,"uuid":"c7e887b3-ab38-4608-9bfd-1ed49956e8af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6809","nonce":2320149212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6811","nonce":2320149212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6815","nonce":2320149212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6813","nonce":2320149212}]},"public_addr":"172.21.15.136:6809/2320149212","cluster_addr":"172.21.15.136:6811/2320149212","heartbeat_back_addr":"172.21.15.136:6815/2320149212","heartbeat_front_addr":"172.21.15.136:6813/2320149212","state":["exists","up"]},{"osd":6,"uuid":"f0bd604b-0f57-4a38-9295-eb200264f1ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6819","nonce":389499207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6821","nonce":389499207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6825","nonce":389499207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6823","nonce":389499207}]},"public_addr":"172.21.15.72:6819/389499207","cluster_addr":"172.21.15.72:6821/389499207","heartbeat_back_addr":"172.21.15.72:6825/389499207","heartbeat_front_addr":"172.21.15.72:6823/389499207","state":["exists","up"]},{"osd":7,"uuid":"1879d4e4-038e-4cd6-a3ec-3a940bcc3797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6817","nonce":375888575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6819","nonce":375888575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6823","nonce":375888575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6821","nonce":375888575}]},"public_addr":"172.21.15.76:6817/375888575","cluster_addr":"172.21.15.76:6819/375888575","heartbeat_back_addr":"172.21.15.76:6823/375888575","heartbeat_front_addr":"172.21.15.76:6821/375888575","state":["exists","up"]},{"osd":8,"uuid":"6f404062-97e7-4ccb-a582-76dc1ffae548","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6817","nonce":3446774052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6819","nonce":3446774052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6823","nonce":3446774052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6821","nonce":3446774052}]},"public_addr":"172.21.15.136:6817/3446774052","cluster_addr":"172.21.15.136:6819/3446774052","heartbeat_back_addr":"172.21.15.136:6823/3446774052","heartbeat_front_addr":"172.21.15.136:6821/3446774052","state":["exists","up"]},{"osd":9,"uuid":"2989ca68-3164-4667-9bcf-230dcbb7b8f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6827","nonce":3423789152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6829","nonce":3423789152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6832","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6833","nonce":3423789152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6831","nonce":3423789152}]},"public_addr":"172.21.15.72:6827/3423789152","cluster_addr":"172.21.15.72:6829/3423789152","heartbeat_back_addr":"172.21.15.72:6833/3423789152","heartbeat_front_addr":"172.21.15.72:6831/3423789152","state":["exists","up"]},{"osd":10,"uuid":"9ec6d46a-0913-4208-97b5-167188ee8cb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6825","nonce":1120991574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6827","nonce":1120991574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6831","nonce":1120991574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6829","nonce":1120991574}]},"public_addr":"172.21.15.76:6825/1120991574","cluster_addr":"172.21.15.76:6827/1120991574","heartbeat_back_addr":"172.21.15.76:6831/1120991574","heartbeat_front_addr":"172.21.15.76:6829/1120991574","state":["exists","up"]},{"osd":11,"uuid":"bdd0774f-6e58-4918-b4cc-afd4f1118d3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6825","nonce":1135995193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6827","nonce":1135995193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6831","nonce":1135995193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6829","nonce":1135995193}]},"public_addr":"172.21.15.136:6825/1135995193","cluster_addr":"172.21.15.136:6827/1135995193","heartbeat_back_addr":"172.21.15.136:6831/1135995193","heartbeat_front_addr":"172.21.15.136:6829/1135995193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:36:52.128788+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:37:17.523654+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:37:41.634677+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:38:15.060797+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:38:41.234111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:39:06.783560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:39:40.884386+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:40:08.340465+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:40:34.344858+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:41:08.525309+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:41:36.782624+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:42:02.756888+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:0/3625071330":"2024-04-27T23:34:58.266218+0000","172.21.15.72:6801/2903979134":"2024-04-27T23:34:58.266218+0000","172.21.15.72:6800/2903979134":"2024-04-27T23:34:58.266218+0000","172.21.15.72:0/1519142576":"2024-04-27T23:34:58.266218+0000","172.21.15.72:0/3669626460":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/945623297":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/3479150073":"2024-04-27T23:34:24.233319+0000","172.21.15.72:6800/4175896290":"2024-04-27T23:34:24.233319+0000","172.21.15.72:6801/4175896290":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/3783175012":"2024-04-27T23:34:58.266218+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:42:17.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.387+0000 7f6aeb800640 1 -- 172.21.15.72:0/4138864848 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6acc0a3bb0 msgr2=0x7f6acc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:17.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.387+0000 7f6aeb800640 1 --2- 172.21.15.72:0/4138864848 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6acc0a3bb0 0x7f6acc0a6070 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8009bd0 tx=0x7f6ad8005f90 comp rx=0 tx=0).stop 2024-04-26T23:42:17.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.387+0000 7f6aeb800640 1 -- 172.21.15.72:0/4138864848 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6ae4101b50 msgr2=0x7f6ae41a2320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:17.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.387+0000 7f6aeb800640 1 --2- 172.21.15.72:0/4138864848 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6ae4101b50 0x7f6ae41a2320 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6adc00eb10 tx=0x7f6adc00c2d0 comp rx=0 tx=0).stop 2024-04-26T23:42:17.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.387+0000 7f6aeb800640 1 -- 172.21.15.72:0/4138864848 shutdown_connections 2024-04-26T23:42:17.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.387+0000 7f6aeb800640 1 --2- 172.21.15.72:0/4138864848 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6acc0a3bb0 0x7f6acc0a6070 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:17.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.387+0000 7f6aeb800640 1 --2- 172.21.15.72:0/4138864848 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6ae4101b50 0x7f6ae41a2320 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:17.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.387+0000 7f6aeb800640 1 --2- 172.21.15.72:0/4138864848 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ae4100fe0 0x7f6ae419b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:17.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.387+0000 7f6aeb800640 1 --2- 172.21.15.72:0/4138864848 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6ae4100620 0x7f6ae419ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:17.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.388+0000 7f6aeb800640 1 -- 172.21.15.72:0/4138864848 >> 172.21.15.72:0/4138864848 conn(0x7f6ae40fc120 msgr2=0x7f6ae40fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:17.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.388+0000 7f6aeb800640 1 -- 172.21.15.72:0/4138864848 shutdown_connections 2024-04-26T23:42:17.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.388+0000 7f6aeb800640 1 -- 172.21.15.72:0/4138864848 wait complete. 2024-04-26T23:42:17.408 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:37:44.428727+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T23:42:17.409 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:42:17.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.610+0000 7f15f0000640 1 -- 172.21.15.72:0/645290615 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f15e8105d80 msgr2=0x7f15e8106200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:17.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.610+0000 7f15f0000640 1 --2- 172.21.15.72:0/645290615 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f15e8105d80 0x7f15e8106200 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f15d4009860 tx=0x7f15d402a340 comp rx=0 tx=0).stop 2024-04-26T23:42:17.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.610+0000 7f15f0000640 1 -- 172.21.15.72:0/645290615 shutdown_connections 2024-04-26T23:42:17.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.610+0000 7f15f0000640 1 --2- 172.21.15.72:0/645290615 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f15e8105d80 0x7f15e8106200 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:17.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.610+0000 7f15f0000640 1 --2- 172.21.15.72:0/645290615 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f15e8106740 0x7f15e810cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:17.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.610+0000 7f15f0000640 1 --2- 172.21.15.72:0/645290615 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15e8104b90 0x7f15e8104fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:17.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.610+0000 7f15f0000640 1 -- 172.21.15.72:0/645290615 >> 172.21.15.72:0/645290615 conn(0x7f15e8100120 msgr2=0x7f15e8102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:17.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.610+0000 7f15f0000640 1 -- 172.21.15.72:0/645290615 shutdown_connections 2024-04-26T23:42:17.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.610+0000 7f15f0000640 1 -- 172.21.15.72:0/645290615 wait complete. 2024-04-26T23:42:17.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.611+0000 7f15f0000640 1 Processor -- start 2024-04-26T23:42:17.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.611+0000 7f15f0000640 1 -- start start 2024-04-26T23:42:17.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.612+0000 7f15f0000640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f15e8104b90 0x7f15e819adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:17.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.612+0000 7f15f0000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f15e8105d80 0x7f15e819b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:17.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.612+0000 7f15f0000640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15e8106740 0x7f15e81a23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:17.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.612+0000 7f15f0000640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f15e81106c0 con 0x7f15e8104b90 2024-04-26T23:42:17.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.612+0000 7f15f0000640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f15e8110540 con 0x7f15e8105d80 2024-04-26T23:42:17.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.612+0000 7f15f0000640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f15e8110840 con 0x7f15e8106740 2024-04-26T23:42:17.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.612+0000 7f15ed600640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f15e8104b90 0x7f15e819adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:17.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.612+0000 7f15ed600640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f15e8104b90 0x7f15e819adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47288/0 (socket says 172.21.15.72:47288) 2024-04-26T23:42:17.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.612+0000 7f15ed600640 1 -- 172.21.15.72:0/2793663250 learned_addr learned my addr 172.21.15.72:0/2793663250 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:42:17.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.612+0000 7f15ee000640 1 --2- 172.21.15.72:0/2793663250 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15e8106740 0x7f15e81a23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:17.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.612+0000 7f15ecc00640 1 --2- 172.21.15.72:0/2793663250 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f15e8105d80 0x7f15e819b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:17.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.612+0000 7f15ed600640 1 -- 172.21.15.72:0/2793663250 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15e8106740 msgr2=0x7f15e81a23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:17.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.612+0000 7f15ed600640 1 --2- 172.21.15.72:0/2793663250 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15e8106740 0x7f15e81a23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:17.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.612+0000 7f15ed600640 1 -- 172.21.15.72:0/2793663250 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f15e8105d80 msgr2=0x7f15e819b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:17.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.612+0000 7f15ed600640 1 --2- 172.21.15.72:0/2793663250 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f15e8105d80 0x7f15e819b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:17.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.612+0000 7f15ed600640 1 -- 172.21.15.72:0/2793663250 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15e81a2a10 con 0x7f15e8104b90 2024-04-26T23:42:17.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.612+0000 7f15ecc00640 1 --2- 172.21.15.72:0/2793663250 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f15e8105d80 0x7f15e819b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:42:17.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.613+0000 7f15ed600640 1 --2- 172.21.15.72:0/2793663250 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f15e8104b90 0x7f15e819adf0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f15dc00c3a0 tx=0x7f15dc00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:17.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.613+0000 7f15e6000640 1 -- 172.21.15.72:0/2793663250 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15dc01b070 con 0x7f15e8104b90 2024-04-26T23:42:17.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.613+0000 7f15e6000640 1 -- 172.21.15.72:0/2793663250 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15dc003e20 con 0x7f15e8104b90 2024-04-26T23:42:17.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.613+0000 7f15e6000640 1 -- 172.21.15.72:0/2793663250 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15dc00b890 con 0x7f15e8104b90 2024-04-26T23:42:17.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.613+0000 7f15f0000640 1 -- 172.21.15.72:0/2793663250 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15e81a2d00 con 0x7f15e8104b90 2024-04-26T23:42:17.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.613+0000 7f15f0000640 1 -- 172.21.15.72:0/2793663250 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15e81a31d0 con 0x7f15e8104b90 2024-04-26T23:42:17.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.615+0000 7f15f0000640 1 -- 172.21.15.72:0/2793663250 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15e810bf20 con 0x7f15e8104b90 2024-04-26T23:42:17.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.616+0000 7f15e6000640 1 -- 172.21.15.72:0/2793663250 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f15dc01e070 con 0x7f15e8104b90 2024-04-26T23:42:17.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.618+0000 7f15e6000640 1 --2- 172.21.15.72:0/2793663250 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f15d00a3bb0 0x7f15d00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:17.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.618+0000 7f15e6000640 1 -- 172.21.15.72:0/2793663250 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f15dc0d0110 con 0x7f15e8104b90 2024-04-26T23:42:17.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.618+0000 7f15ecc00640 1 --2- 172.21.15.72:0/2793663250 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f15d00a3bb0 0x7f15d00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:17.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.619+0000 7f15ecc00640 1 --2- 172.21.15.72:0/2793663250 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f15d00a3bb0 0x7f15d00a6070 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f15d402a930 tx=0x7f15d4030450 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:17.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.623+0000 7f15e6000640 1 -- 172.21.15.72:0/2793663250 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f15dc09cef0 con 0x7f15e8104b90 2024-04-26T23:42:17.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.823+0000 7f15f0000640 1 -- 172.21.15.72:0/2793663250 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f15e8063530 con 0x7f15e8104b90 2024-04-26T23:42:17.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.824+0000 7f15e6000640 1 -- 172.21.15.72:0/2793663250 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f15dc09c7f0 con 0x7f15e8104b90 2024-04-26T23:42:17.827 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-04-26T23:42:17.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.827+0000 7f15f0000640 1 -- 172.21.15.72:0/2793663250 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f15d00a3bb0 msgr2=0x7f15d00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:17.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.827+0000 7f15f0000640 1 --2- 172.21.15.72:0/2793663250 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f15d00a3bb0 0x7f15d00a6070 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f15d402a930 tx=0x7f15d4030450 comp rx=0 tx=0).stop 2024-04-26T23:42:17.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.827+0000 7f15f0000640 1 -- 172.21.15.72:0/2793663250 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f15e8104b90 msgr2=0x7f15e819adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:17.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.827+0000 7f15f0000640 1 --2- 172.21.15.72:0/2793663250 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f15e8104b90 0x7f15e819adf0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f15dc00c3a0 tx=0x7f15dc00c860 comp rx=0 tx=0).stop 2024-04-26T23:42:17.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.827+0000 7f15f0000640 1 -- 172.21.15.72:0/2793663250 shutdown_connections 2024-04-26T23:42:17.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.827+0000 7f15f0000640 1 --2- 172.21.15.72:0/2793663250 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f15d00a3bb0 0x7f15d00a6070 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:17.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.827+0000 7f15f0000640 1 --2- 172.21.15.72:0/2793663250 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f15e8105d80 0x7f15e819b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:17.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.827+0000 7f15f0000640 1 --2- 172.21.15.72:0/2793663250 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15e8106740 0x7f15e81a23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:17.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.827+0000 7f15f0000640 1 --2- 172.21.15.72:0/2793663250 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f15e8104b90 0x7f15e819adf0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:17.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.827+0000 7f15f0000640 1 -- 172.21.15.72:0/2793663250 >> 172.21.15.72:0/2793663250 conn(0x7f15e8100120 msgr2=0x7f15e81090f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:17.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.827+0000 7f15f0000640 1 -- 172.21.15.72:0/2793663250 shutdown_connections 2024-04-26T23:42:17.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:17.827+0000 7f15f0000640 1 -- 172.21.15.72:0/2793663250 wait complete. 2024-04-26T23:42:17.849 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:42:18.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.047+0000 7f3931e00640 1 -- 172.21.15.72:0/4090523421 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f392c105b80 msgr2=0x7f392c106000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:18.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.047+0000 7f3931e00640 1 --2- 172.21.15.72:0/4090523421 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f392c105b80 0x7f392c106000 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f3914009900 tx=0x7f391402a3f0 comp rx=0 tx=0).stop 2024-04-26T23:42:18.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.047+0000 7f3931e00640 1 -- 172.21.15.72:0/4090523421 shutdown_connections 2024-04-26T23:42:18.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.047+0000 7f3931e00640 1 --2- 172.21.15.72:0/4090523421 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f392c106540 0x7f392c10cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:18.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.047+0000 7f3931e00640 1 --2- 172.21.15.72:0/4090523421 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f392c105b80 0x7f392c106000 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:18.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.047+0000 7f3931e00640 1 --2- 172.21.15.72:0/4090523421 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f392c104990 0x7f392c104db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:18.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.047+0000 7f3931e00640 1 -- 172.21.15.72:0/4090523421 >> 172.21.15.72:0/4090523421 conn(0x7f392c0fff00 msgr2=0x7f392c102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:18.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.047+0000 7f3931e00640 1 -- 172.21.15.72:0/4090523421 shutdown_connections 2024-04-26T23:42:18.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.047+0000 7f3931e00640 1 -- 172.21.15.72:0/4090523421 wait complete. 2024-04-26T23:42:18.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.048+0000 7f3931e00640 1 Processor -- start 2024-04-26T23:42:18.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.048+0000 7f3931e00640 1 -- start start 2024-04-26T23:42:18.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.048+0000 7f3931e00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f392c104990 0x7f392c19abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:18.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.048+0000 7f3931e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f392c105b80 0x7f392c19b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:18.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.048+0000 7f3931e00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f392c106540 0x7f392c1a2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:18.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.049+0000 7f3931e00640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f392c1104a0 con 0x7f392c104990 2024-04-26T23:42:18.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.049+0000 7f3931e00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f392c110320 con 0x7f392c105b80 2024-04-26T23:42:18.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.049+0000 7f3931e00640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f392c110620 con 0x7f392c106540 2024-04-26T23:42:18.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.049+0000 7f3930a00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f392c104990 0x7f392c19abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:18.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.049+0000 7f3930a00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f392c104990 0x7f392c19abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47306/0 (socket says 172.21.15.72:47306) 2024-04-26T23:42:18.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.049+0000 7f3930a00640 1 -- 172.21.15.72:0/1255069048 learned_addr learned my addr 172.21.15.72:0/1255069048 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:42:18.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.049+0000 7f3923e00640 1 --2- 172.21.15.72:0/1255069048 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f392c105b80 0x7f392c19b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:18.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.049+0000 7f3931400640 1 --2- 172.21.15.72:0/1255069048 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f392c106540 0x7f392c1a2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:18.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.049+0000 7f3930a00640 1 -- 172.21.15.72:0/1255069048 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f392c106540 msgr2=0x7f392c1a2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:18.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.049+0000 7f3930a00640 1 --2- 172.21.15.72:0/1255069048 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f392c106540 0x7f392c1a2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:18.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.049+0000 7f3930a00640 1 -- 172.21.15.72:0/1255069048 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f392c105b80 msgr2=0x7f392c19b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:18.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.049+0000 7f3930a00640 1 --2- 172.21.15.72:0/1255069048 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f392c105b80 0x7f392c19b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:18.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.049+0000 7f3930a00640 1 -- 172.21.15.72:0/1255069048 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f392c1a27f0 con 0x7f392c104990 2024-04-26T23:42:18.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.050+0000 7f3930a00640 1 --2- 172.21.15.72:0/1255069048 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f392c104990 0x7f392c19abd0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f391c00e9c0 tx=0x7f391c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:18.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.050+0000 7f392a000640 1 -- 172.21.15.72:0/1255069048 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f391c00cde0 con 0x7f392c104990 2024-04-26T23:42:18.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.050+0000 7f392a000640 1 -- 172.21.15.72:0/1255069048 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f391c0108e0 con 0x7f392c104990 2024-04-26T23:42:18.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.050+0000 7f3931e00640 1 -- 172.21.15.72:0/1255069048 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f392c1a2ae0 con 0x7f392c104990 2024-04-26T23:42:18.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.050+0000 7f392a000640 1 -- 172.21.15.72:0/1255069048 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f391c018960 con 0x7f392c104990 2024-04-26T23:42:18.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.050+0000 7f3931e00640 1 -- 172.21.15.72:0/1255069048 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f392c1a3000 con 0x7f392c104990 2024-04-26T23:42:18.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.051+0000 7f3931e00640 1 -- 172.21.15.72:0/1255069048 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f392c10bd20 con 0x7f392c104990 2024-04-26T23:42:18.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.053+0000 7f392a000640 1 -- 172.21.15.72:0/1255069048 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f391c01f020 con 0x7f392c104990 2024-04-26T23:42:18.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.054+0000 7f392a000640 1 --2- 172.21.15.72:0/1255069048 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f39100a3bb0 0x7f39100a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:18.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.054+0000 7f392a000640 1 -- 172.21.15.72:0/1255069048 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f391c0d0270 con 0x7f392c104990 2024-04-26T23:42:18.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.054+0000 7f3923e00640 1 --2- 172.21.15.72:0/1255069048 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f39100a3bb0 0x7f39100a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:18.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.055+0000 7f3923e00640 1 --2- 172.21.15.72:0/1255069048 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f39100a3bb0 0x7f39100a6070 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3914000c00 tx=0x7f39140314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:18.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.058+0000 7f392a000640 1 -- 172.21.15.72:0/1255069048 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f391c054060 con 0x7f392c104990 2024-04-26T23:42:18.112 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:17 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:42:18.113 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:17 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:42:18.113 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:17 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:18.113 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:17 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:42:18.113 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:17 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:42:18.113 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:17 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:42:18.113 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:17 smithi136 ceph-mon[33050]: Deploying daemon mds.l on smithi136 2024-04-26T23:42:18.113 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:17 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/4138864848' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:42:18.113 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:17 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/2793663250' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T23:42:18.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:17 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:42:18.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:17 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:42:18.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:17 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:18.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:17 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:42:18.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:17 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:42:18.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:17 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:42:18.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:17 smithi076 ceph-mon[32774]: Deploying daemon mds.l on smithi136 2024-04-26T23:42:18.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:17 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/4138864848' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:42:18.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:17 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/2793663250' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T23:42:18.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:17 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:42:18.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:17 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:42:18.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:17 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:18.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:17 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:42:18.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:17 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:42:18.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:17 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:42:18.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:17 smithi072 ceph-mon[28184]: Deploying daemon mds.l on smithi136 2024-04-26T23:42:18.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:17 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/4138864848' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:42:18.295 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:17 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/2793663250' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T23:42:18.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.298+0000 7f3931e00640 1 -- 172.21.15.72:0/1255069048 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f392c0635d0 con 0x7f392c104990 2024-04-26T23:42:18.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.299+0000 7f392a000640 1 -- 172.21.15.72:0/1255069048 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f391c09d050 con 0x7f392c104990 2024-04-26T23:42:18.302 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:42:18.302 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-04-26T23:42:18.303 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 1 2024-04-26T23:42:18.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.301+0000 7f3931e00640 1 -- 172.21.15.72:0/1255069048 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f39100a3bb0 msgr2=0x7f39100a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:18.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.301+0000 7f3931e00640 1 --2- 172.21.15.72:0/1255069048 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f39100a3bb0 0x7f39100a6070 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3914000c00 tx=0x7f39140314a0 comp rx=0 tx=0).stop 2024-04-26T23:42:18.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.301+0000 7f3931e00640 1 -- 172.21.15.72:0/1255069048 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f392c104990 msgr2=0x7f392c19abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:18.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.301+0000 7f3931e00640 1 --2- 172.21.15.72:0/1255069048 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f392c104990 0x7f392c19abd0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f391c00e9c0 tx=0x7f391c00ee80 comp rx=0 tx=0).stop 2024-04-26T23:42:18.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.301+0000 7f3931e00640 1 -- 172.21.15.72:0/1255069048 shutdown_connections 2024-04-26T23:42:18.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.301+0000 7f3931e00640 1 --2- 172.21.15.72:0/1255069048 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f39100a3bb0 0x7f39100a6070 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:18.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.301+0000 7f3931e00640 1 --2- 172.21.15.72:0/1255069048 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f392c105b80 0x7f392c19b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:18.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.301+0000 7f3931e00640 1 --2- 172.21.15.72:0/1255069048 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f392c106540 0x7f392c1a2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:18.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.301+0000 7f3931e00640 1 --2- 172.21.15.72:0/1255069048 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f392c104990 0x7f392c19abd0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:18.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.301+0000 7f3931e00640 1 -- 172.21.15.72:0/1255069048 >> 172.21.15.72:0/1255069048 conn(0x7f392c0fff00 msgr2=0x7f392c108ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:18.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.302+0000 7f3931e00640 1 -- 172.21.15.72:0/1255069048 shutdown_connections 2024-04-26T23:42:18.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:18.302+0000 7f3931e00640 1 -- 172.21.15.72:0/1255069048 wait complete. 2024-04-26T23:42:19.175 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:18 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/1255069048' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:19.175 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:18 smithi136 ceph-mon[33050]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:19.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:18 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/1255069048' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:19.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:18 smithi076 ceph-mon[32774]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:19.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:18 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/1255069048' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:19.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:18 smithi072 ceph-mon[28184]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:19.550 INFO:journalctl@ceph.mds.l.smithi136.stdout:Apr 26 23:42:19 smithi136 systemd[1]: Starting Ceph mds.l for 3c8453ae-0425-11ef-bc93-c7b262605968... 2024-04-26T23:42:19.840 INFO:journalctl@ceph.mds.l.smithi136.stdout:Apr 26 23:42:19 smithi136 podman[62212]: 2024-04-26 23:42:19.473906239 +0000 UTC m=+0.017792864 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T23:42:19.840 INFO:journalctl@ceph.mds.l.smithi136.stdout:Apr 26 23:42:19 smithi136 podman[62212]: 2024-04-26 23:42:19.600860717 +0000 UTC m=+0.144747374 container create f0dc138430b0426d211e3f19e8be4683bfda5f568ea1427adfd43d23e568f631 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True) 2024-04-26T23:42:19.840 INFO:journalctl@ceph.mds.l.smithi136.stdout:Apr 26 23:42:19 smithi136 podman[62212]: 2024-04-26 23:42:19.837019572 +0000 UTC m=+0.380906187 container init f0dc138430b0426d211e3f19e8be4683bfda5f568ea1427adfd43d23e568f631 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-l, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-26T23:42:20.300 INFO:journalctl@ceph.mds.l.smithi136.stdout:Apr 26 23:42:19 smithi136 podman[62212]: 2024-04-26 23:42:19.83948489 +0000 UTC m=+0.383371506 container start f0dc138430b0426d211e3f19e8be4683bfda5f568ea1427adfd43d23e568f631 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-l, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T23:42:20.300 INFO:journalctl@ceph.mds.l.smithi136.stdout:Apr 26 23:42:19 smithi136 ceph-mds[62231]: -- 172.21.15.136:0/4206323822 <== mon.1 v2:172.21.15.136:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5647c70b1a40 con 0x5647c7085000 2024-04-26T23:42:20.300 INFO:journalctl@ceph.mds.l.smithi136.stdout:Apr 26 23:42:19 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-l[62227]: starting mds.l at 2024-04-26T23:42:20.300 INFO:journalctl@ceph.mds.l.smithi136.stdout:Apr 26 23:42:19 smithi136 bash[62212]: f0dc138430b0426d211e3f19e8be4683bfda5f568ea1427adfd43d23e568f631 2024-04-26T23:42:20.300 INFO:journalctl@ceph.mds.l.smithi136.stdout:Apr 26 23:42:19 smithi136 systemd[1]: Started Ceph mds.l for 3c8453ae-0425-11ef-bc93-c7b262605968. 2024-04-26T23:42:20.325 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:42:20.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.539+0000 7ff718800640 1 -- 172.21.15.72:0/2174959731 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff710108f90 msgr2=0x7ff7101093b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:20.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.539+0000 7ff718800640 1 --2- 172.21.15.72:0/2174959731 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff710108f90 0x7ff7101093b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff704009900 tx=0x7ff70402a3f0 comp rx=0 tx=0).stop 2024-04-26T23:42:20.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.539+0000 7ff718800640 1 -- 172.21.15.72:0/2174959731 shutdown_connections 2024-04-26T23:42:20.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.539+0000 7ff718800640 1 --2- 172.21.15.72:0/2174959731 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff71010ab40 0x7ff7101113d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:20.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.539+0000 7ff718800640 1 --2- 172.21.15.72:0/2174959731 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff71010a180 0x7ff71010a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:20.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.539+0000 7ff718800640 1 --2- 172.21.15.72:0/2174959731 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff710108f90 0x7ff7101093b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:20.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.539+0000 7ff718800640 1 -- 172.21.15.72:0/2174959731 >> 172.21.15.72:0/2174959731 conn(0x7ff710077f40 msgr2=0x7ff71007a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:20.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.539+0000 7ff718800640 1 -- 172.21.15.72:0/2174959731 shutdown_connections 2024-04-26T23:42:20.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.540+0000 7ff718800640 1 -- 172.21.15.72:0/2174959731 wait complete. 2024-04-26T23:42:20.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.540+0000 7ff718800640 1 Processor -- start 2024-04-26T23:42:20.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.541+0000 7ff718800640 1 -- start start 2024-04-26T23:42:20.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.541+0000 7ff718800640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff710108f90 0x7ff71019f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:20.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.541+0000 7ff718800640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff71010a180 0x7ff71019f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:20.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.541+0000 7ff718800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff71010ab40 0x7ff7101a6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:20.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.541+0000 7ff718800640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7ff710114b20 con 0x7ff710108f90 2024-04-26T23:42:20.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.541+0000 7ff718800640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff7101149a0 con 0x7ff71010ab40 2024-04-26T23:42:20.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.541+0000 7ff718800640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff710114ca0 con 0x7ff71010a180 2024-04-26T23:42:20.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.541+0000 7ff715e00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff710108f90 0x7ff71019f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:20.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.541+0000 7ff715e00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff710108f90 0x7ff71019f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47314/0 (socket says 172.21.15.72:47314) 2024-04-26T23:42:20.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.541+0000 7ff715e00640 1 -- 172.21.15.72:0/1556932712 learned_addr learned my addr 172.21.15.72:0/1556932712 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:42:20.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.541+0000 7ff716800640 1 --2- 172.21.15.72:0/1556932712 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff71010ab40 0x7ff7101a6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:20.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.541+0000 7ff715400640 1 --2- 172.21.15.72:0/1556932712 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff71010a180 0x7ff71019f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:20.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.542+0000 7ff715e00640 1 -- 172.21.15.72:0/1556932712 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff71010a180 msgr2=0x7ff71019f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:20.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.542+0000 7ff715e00640 1 --2- 172.21.15.72:0/1556932712 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff71010a180 0x7ff71019f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:20.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.542+0000 7ff715e00640 1 -- 172.21.15.72:0/1556932712 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff71010ab40 msgr2=0x7ff7101a6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:20.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.542+0000 7ff715e00640 1 --2- 172.21.15.72:0/1556932712 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff71010ab40 0x7ff7101a6730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:20.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.542+0000 7ff715e00640 1 -- 172.21.15.72:0/1556932712 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7101a6da0 con 0x7ff710108f90 2024-04-26T23:42:20.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.542+0000 7ff715e00640 1 --2- 172.21.15.72:0/1556932712 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff710108f90 0x7ff71019f180 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff704031500 tx=0x7ff704003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:20.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.542+0000 7ff70aa00640 1 -- 172.21.15.72:0/1556932712 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff70403f070 con 0x7ff710108f90 2024-04-26T23:42:20.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.542+0000 7ff70aa00640 1 -- 172.21.15.72:0/1556932712 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff704031c50 con 0x7ff710108f90 2024-04-26T23:42:20.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.542+0000 7ff718800640 1 -- 172.21.15.72:0/1556932712 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7101a7030 con 0x7ff710108f90 2024-04-26T23:42:20.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.542+0000 7ff70aa00640 1 -- 172.21.15.72:0/1556932712 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff704035030 con 0x7ff710108f90 2024-04-26T23:42:20.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.542+0000 7ff718800640 1 -- 172.21.15.72:0/1556932712 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7101a7520 con 0x7ff710108f90 2024-04-26T23:42:20.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.544+0000 7ff718800640 1 -- 172.21.15.72:0/1556932712 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7101102b0 con 0x7ff710108f90 2024-04-26T23:42:20.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.546+0000 7ff70aa00640 1 -- 172.21.15.72:0/1556932712 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ff704042070 con 0x7ff710108f90 2024-04-26T23:42:20.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.548+0000 7ff70aa00640 1 --2- 172.21.15.72:0/1556932712 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff6f80a3bb0 0x7ff6f80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:20.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.548+0000 7ff70aa00640 1 -- 172.21.15.72:0/1556932712 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7ff7040f0700 con 0x7ff710108f90 2024-04-26T23:42:20.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.548+0000 7ff715400640 1 --2- 172.21.15.72:0/1556932712 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff6f80a3bb0 0x7ff6f80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:20.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.549+0000 7ff715400640 1 --2- 172.21.15.72:0/1556932712 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff6f80a3bb0 0x7ff6f80a6070 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff7000095f0 tx=0x7ff700005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:20.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.553+0000 7ff70aa00640 1 -- 172.21.15.72:0/1556932712 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff7040bce90 con 0x7ff710108f90 2024-04-26T23:42:20.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.797+0000 7ff718800640 1 -- 172.21.15.72:0/1556932712 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff71002dad0 con 0x7ff710108f90 2024-04-26T23:42:20.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.798+0000 7ff70aa00640 1 -- 172.21.15.72:0/1556932712 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7ff7040c1190 con 0x7ff710108f90 2024-04-26T23:42:20.801 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:42:20.801 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1528342053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1528342053},{"type":"v1","addr":"172.21.15.136:6833","nonce":1528342053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-26T23:42:20.801 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 2 2024-04-26T23:42:20.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.800+0000 7ff718800640 1 -- 172.21.15.72:0/1556932712 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff6f80a3bb0 msgr2=0x7ff6f80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:20.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.800+0000 7ff718800640 1 --2- 172.21.15.72:0/1556932712 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff6f80a3bb0 0x7ff6f80a6070 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff7000095f0 tx=0x7ff700005f90 comp rx=0 tx=0).stop 2024-04-26T23:42:20.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.800+0000 7ff718800640 1 -- 172.21.15.72:0/1556932712 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff710108f90 msgr2=0x7ff71019f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:20.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.800+0000 7ff718800640 1 --2- 172.21.15.72:0/1556932712 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff710108f90 0x7ff71019f180 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff704031500 tx=0x7ff704003c10 comp rx=0 tx=0).stop 2024-04-26T23:42:20.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.800+0000 7ff718800640 1 -- 172.21.15.72:0/1556932712 shutdown_connections 2024-04-26T23:42:20.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.800+0000 7ff718800640 1 --2- 172.21.15.72:0/1556932712 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff6f80a3bb0 0x7ff6f80a6070 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:20.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.800+0000 7ff718800640 1 --2- 172.21.15.72:0/1556932712 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff71010ab40 0x7ff7101a6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:20.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.800+0000 7ff718800640 1 --2- 172.21.15.72:0/1556932712 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff71010a180 0x7ff71019f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:20.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.800+0000 7ff718800640 1 --2- 172.21.15.72:0/1556932712 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff710108f90 0x7ff71019f180 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:20.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.800+0000 7ff718800640 1 -- 172.21.15.72:0/1556932712 >> 172.21.15.72:0/1556932712 conn(0x7ff710077f40 msgr2=0x7ff71010d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:20.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.801+0000 7ff718800640 1 -- 172.21.15.72:0/1556932712 shutdown_connections 2024-04-26T23:42:20.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:20.801+0000 7ff718800640 1 -- 172.21.15.72:0/1556932712 wait complete. 2024-04-26T23:42:20.997 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:20 smithi076 ceph-mon[32774]: mds.? [v2:172.21.15.136:6832/1528342053,v1:172.21.15.136:6833/1528342053] up:boot 2024-04-26T23:42:20.998 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:20 smithi076 ceph-mon[32774]: fsmap 1 up:standby 2024-04-26T23:42:20.998 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:20 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T23:42:20.998 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:20 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:20.998 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:20 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:20.998 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:20 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:20.998 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:20 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:42:20.998 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:20 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:42:20.998 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:20 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:42:20.998 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:20 smithi076 ceph-mon[32774]: Deploying daemon mds.k on smithi076 2024-04-26T23:42:20.998 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:20 smithi076 ceph-mon[32774]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:20.998 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:20 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/1556932712' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:21.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:20 smithi072 ceph-mon[28184]: mds.? [v2:172.21.15.136:6832/1528342053,v1:172.21.15.136:6833/1528342053] up:boot 2024-04-26T23:42:21.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:20 smithi072 ceph-mon[28184]: fsmap 1 up:standby 2024-04-26T23:42:21.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:20 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T23:42:21.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:20 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:21.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:20 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:21.295 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:20 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:21.295 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:20 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:42:21.295 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:20 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:42:21.295 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:20 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:42:21.295 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:20 smithi072 ceph-mon[28184]: Deploying daemon mds.k on smithi076 2024-04-26T23:42:21.295 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:20 smithi072 ceph-mon[28184]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:21.295 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:20 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/1556932712' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:21.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:20 smithi136 ceph-mon[33050]: mds.? [v2:172.21.15.136:6832/1528342053,v1:172.21.15.136:6833/1528342053] up:boot 2024-04-26T23:42:21.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:20 smithi136 ceph-mon[33050]: fsmap 1 up:standby 2024-04-26T23:42:21.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:20 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T23:42:21.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:20 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:21.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:20 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:21.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:20 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:21.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:20 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:42:21.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:20 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:42:21.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:20 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:42:21.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:20 smithi136 ceph-mon[33050]: Deploying daemon mds.k on smithi076 2024-04-26T23:42:21.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:20 smithi136 ceph-mon[33050]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:21.301 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:20 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/1556932712' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:22.582 INFO:journalctl@ceph.mds.k.smithi076.stdout:Apr 26 23:42:22 smithi076 systemd[1]: Starting Ceph mds.k for 3c8453ae-0425-11ef-bc93-c7b262605968... 2024-04-26T23:42:22.822 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:42:22.845 INFO:journalctl@ceph.mds.k.smithi076.stdout:Apr 26 23:42:22 smithi076 podman[61309]: 2024-04-26 23:42:22.48158533 +0000 UTC m=+0.018429688 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T23:42:22.845 INFO:journalctl@ceph.mds.k.smithi076.stdout:Apr 26 23:42:22 smithi076 podman[61309]: 2024-04-26 23:42:22.597013532 +0000 UTC m=+0.133857874 container create 78176133a811626c2aa111917c971ad460045ac4358f6426369b2e49f9de874c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-k, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T23:42:22.845 INFO:journalctl@ceph.mds.k.smithi076.stdout:Apr 26 23:42:22 smithi076 podman[61309]: 2024-04-26 23:42:22.84170598 +0000 UTC m=+0.378550322 container init 78176133a811626c2aa111917c971ad460045ac4358f6426369b2e49f9de874c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-k, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-26T23:42:23.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.006+0000 7f7ba9800640 1 -- 172.21.15.72:0/1506507525 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7ba4106530 msgr2=0x7f7ba410cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:23.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.006+0000 7f7ba9800640 1 --2- 172.21.15.72:0/1506507525 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7ba4106530 0x7f7ba410cdc0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c009ca0 tx=0x7f7b9c02a8c0 comp rx=0 tx=0).stop 2024-04-26T23:42:23.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.006+0000 7f7ba9800640 1 -- 172.21.15.72:0/1506507525 shutdown_connections 2024-04-26T23:42:23.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.006+0000 7f7ba9800640 1 --2- 172.21.15.72:0/1506507525 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7ba4106530 0x7f7ba410cdc0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:23.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.006+0000 7f7ba9800640 1 --2- 172.21.15.72:0/1506507525 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ba4105b70 0x7f7ba4105ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:23.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.006+0000 7f7ba9800640 1 --2- 172.21.15.72:0/1506507525 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7ba4104980 0x7f7ba4104da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:23.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.006+0000 7f7ba9800640 1 -- 172.21.15.72:0/1506507525 >> 172.21.15.72:0/1506507525 conn(0x7f7ba40fff10 msgr2=0x7f7ba4102330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:23.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.006+0000 7f7ba9800640 1 -- 172.21.15.72:0/1506507525 shutdown_connections 2024-04-26T23:42:23.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.006+0000 7f7ba9800640 1 -- 172.21.15.72:0/1506507525 wait complete. 2024-04-26T23:42:23.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.007+0000 7f7ba9800640 1 Processor -- start 2024-04-26T23:42:23.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.007+0000 7f7ba9800640 1 -- start start 2024-04-26T23:42:23.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.008+0000 7f7ba9800640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7ba4104980 0x7f7ba419ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:23.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.008+0000 7f7ba9800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7ba4105b70 0x7f7ba419b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:23.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.008+0000 7f7ba9800640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ba4106530 0x7f7ba41a2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:23.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.008+0000 7f7ba9800640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba41104b0 con 0x7f7ba4106530 2024-04-26T23:42:23.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.008+0000 7f7ba9800640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba4110330 con 0x7f7ba4105b70 2024-04-26T23:42:23.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.008+0000 7f7ba9800640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba4110630 con 0x7f7ba4104980 2024-04-26T23:42:23.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.008+0000 7f7ba8e00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ba4106530 0x7f7ba41a2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:23.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.008+0000 7f7ba3e00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7ba4104980 0x7f7ba419ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:23.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.008+0000 7f7ba3e00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7ba4104980 0x7f7ba419ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.72:36690/0 (socket says 172.21.15.72:36690) 2024-04-26T23:42:23.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.008+0000 7f7ba3e00640 1 -- 172.21.15.72:0/2803875840 learned_addr learned my addr 172.21.15.72:0/2803875840 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:42:23.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.008+0000 7f7ba8e00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ba4106530 0x7f7ba41a2130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47324/0 (socket says 172.21.15.72:47324) 2024-04-26T23:42:23.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.008+0000 7f7ba3400640 1 --2- 172.21.15.72:0/2803875840 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7ba4105b70 0x7f7ba419b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:23.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.009+0000 7f7ba3e00640 1 -- 172.21.15.72:0/2803875840 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7ba4105b70 msgr2=0x7f7ba419b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:23.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.009+0000 7f7ba3e00640 1 --2- 172.21.15.72:0/2803875840 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7ba4105b70 0x7f7ba419b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:23.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.009+0000 7f7ba3e00640 1 -- 172.21.15.72:0/2803875840 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ba4106530 msgr2=0x7f7ba41a2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:23.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.009+0000 7f7ba3e00640 1 --2- 172.21.15.72:0/2803875840 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ba4106530 0x7f7ba41a2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:23.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.009+0000 7f7ba3e00640 1 -- 172.21.15.72:0/2803875840 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ba41a27a0 con 0x7f7ba4104980 2024-04-26T23:42:23.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.009+0000 7f7ba3400640 1 --2- 172.21.15.72:0/2803875840 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7ba4105b70 0x7f7ba419b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:42:23.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.009+0000 7f7ba3e00640 1 --2- 172.21.15.72:0/2803875840 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7ba4104980 0x7f7ba419ab80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7b9400e9c0 tx=0x7f7b9400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:23.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.009+0000 7f7ba0c00640 1 -- 172.21.15.72:0/2803875840 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b9400cd80 con 0x7f7ba4104980 2024-04-26T23:42:23.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.009+0000 7f7ba0c00640 1 -- 172.21.15.72:0/2803875840 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b940035f0 con 0x7f7ba4104980 2024-04-26T23:42:23.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.009+0000 7f7ba9800640 1 -- 172.21.15.72:0/2803875840 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ba41a2a90 con 0x7f7ba4104980 2024-04-26T23:42:23.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.009+0000 7f7ba0c00640 1 -- 172.21.15.72:0/2803875840 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b94010690 con 0x7f7ba4104980 2024-04-26T23:42:23.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.009+0000 7f7ba9800640 1 -- 172.21.15.72:0/2803875840 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ba41a2f90 con 0x7f7ba4104980 2024-04-26T23:42:23.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.011+0000 7f7ba9800640 1 -- 172.21.15.72:0/2803875840 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ba410bd20 con 0x7f7ba4104980 2024-04-26T23:42:23.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.012+0000 7f7ba0c00640 1 -- 172.21.15.72:0/2803875840 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f7b94004000 con 0x7f7ba4104980 2024-04-26T23:42:23.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.013+0000 7f7ba0c00640 1 --2- 172.21.15.72:0/2803875840 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f7b880a3bb0 0x7f7b880a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:23.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.013+0000 7f7ba0c00640 1 -- 172.21.15.72:0/2803875840 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f7b940d04f0 con 0x7f7ba4104980 2024-04-26T23:42:23.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.014+0000 7f7ba3400640 1 --2- 172.21.15.72:0/2803875840 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f7b880a3bb0 0x7f7b880a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:23.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.014+0000 7f7ba3400640 1 --2- 172.21.15.72:0/2803875840 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f7b880a3bb0 0x7f7b880a6070 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7ba419c090 tx=0x7f7b90005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:23.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.018+0000 7f7ba0c00640 1 -- 172.21.15.72:0/2803875840 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7b9409d2d0 con 0x7f7ba4104980 2024-04-26T23:42:23.179 INFO:journalctl@ceph.mds.k.smithi076.stdout:Apr 26 23:42:22 smithi076 podman[61309]: 2024-04-26 23:42:22.844477406 +0000 UTC m=+0.381321759 container start 78176133a811626c2aa111917c971ad460045ac4358f6426369b2e49f9de874c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-k, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-26T23:42:23.179 INFO:journalctl@ceph.mds.k.smithi076.stdout:Apr 26 23:42:22 smithi076 ceph-mds[61328]: -- 172.21.15.76:0/984140700 <== mon.2 v2:172.21.15.76:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564e1479fa40 con 0x564e14772c00 2024-04-26T23:42:23.179 INFO:journalctl@ceph.mds.k.smithi076.stdout:Apr 26 23:42:22 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-k[61324]: starting mds.k at 2024-04-26T23:42:23.179 INFO:journalctl@ceph.mds.k.smithi076.stdout:Apr 26 23:42:22 smithi076 bash[61309]: 78176133a811626c2aa111917c971ad460045ac4358f6426369b2e49f9de874c 2024-04-26T23:42:23.179 INFO:journalctl@ceph.mds.k.smithi076.stdout:Apr 26 23:42:22 smithi076 systemd[1]: Started Ceph mds.k for 3c8453ae-0425-11ef-bc93-c7b262605968. 2024-04-26T23:42:23.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.267+0000 7f7ba9800640 1 -- 172.21.15.72:0/2803875840 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7ba4063530 con 0x7f7ba4104980 2024-04-26T23:42:23.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.268+0000 7f7ba0c00640 1 -- 172.21.15.72:0/2803875840 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f7b9409cbd0 con 0x7f7ba4104980 2024-04-26T23:42:23.271 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:42:23.271 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1528342053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1528342053},{"type":"v1","addr":"172.21.15.136:6833","nonce":1528342053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-26T23:42:23.272 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 2 2024-04-26T23:42:23.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.272+0000 7f7ba9800640 1 -- 172.21.15.72:0/2803875840 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f7b880a3bb0 msgr2=0x7f7b880a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:23.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.272+0000 7f7ba9800640 1 --2- 172.21.15.72:0/2803875840 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f7b880a3bb0 0x7f7b880a6070 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7ba419c090 tx=0x7f7b90005f90 comp rx=0 tx=0).stop 2024-04-26T23:42:23.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.272+0000 7f7ba9800640 1 -- 172.21.15.72:0/2803875840 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7ba4104980 msgr2=0x7f7ba419ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:23.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.272+0000 7f7ba9800640 1 --2- 172.21.15.72:0/2803875840 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7ba4104980 0x7f7ba419ab80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7b9400e9c0 tx=0x7f7b9400ee80 comp rx=0 tx=0).stop 2024-04-26T23:42:23.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.272+0000 7f7ba9800640 1 -- 172.21.15.72:0/2803875840 shutdown_connections 2024-04-26T23:42:23.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.272+0000 7f7ba9800640 1 --2- 172.21.15.72:0/2803875840 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f7b880a3bb0 0x7f7b880a6070 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:23.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.272+0000 7f7ba9800640 1 --2- 172.21.15.72:0/2803875840 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7ba4105b70 0x7f7ba419b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:23.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.272+0000 7f7ba9800640 1 --2- 172.21.15.72:0/2803875840 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ba4106530 0x7f7ba41a2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:23.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.272+0000 7f7ba9800640 1 --2- 172.21.15.72:0/2803875840 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7ba4104980 0x7f7ba419ab80 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:23.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.272+0000 7f7ba9800640 1 -- 172.21.15.72:0/2803875840 >> 172.21.15.72:0/2803875840 conn(0x7f7ba40fff10 msgr2=0x7f7ba4108ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:23.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.272+0000 7f7ba9800640 1 -- 172.21.15.72:0/2803875840 shutdown_connections 2024-04-26T23:42:23.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:23.272+0000 7f7ba9800640 1 -- 172.21.15.72:0/2803875840 wait complete. 2024-04-26T23:42:23.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:23 smithi072 ceph-mon[28184]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:23.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:23 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:23.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:23 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:23.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:23 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:23.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:23 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:42:23.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:23 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:42:23.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:23 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:42:23.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:23 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/2803875840' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:23.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:23 smithi136 ceph-mon[33050]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:23.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:23 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:23.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:23 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:23.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:23 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:23.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:23 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:42:23.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:23 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:42:23.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:23 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:42:23.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:23 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/2803875840' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:23.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:23 smithi076 ceph-mon[32774]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:23.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:23 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:23.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:23 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:23.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:23 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:23.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:23 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:42:23.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:23 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:42:23.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:23 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:42:23.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:23 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/2803875840' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:24.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:24 smithi072 ceph-mon[28184]: Deploying daemon mds.j on smithi072 2024-04-26T23:42:24.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:24 smithi072 ceph-mon[28184]: mds.? [v2:172.21.15.76:6832/2427238559,v1:172.21.15.76:6833/2427238559] up:boot 2024-04-26T23:42:24.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:24 smithi072 ceph-mon[28184]: fsmap 2 up:standby 2024-04-26T23:42:24.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:24 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T23:42:24.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:24 smithi136 ceph-mon[33050]: Deploying daemon mds.j on smithi072 2024-04-26T23:42:24.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:24 smithi136 ceph-mon[33050]: mds.? [v2:172.21.15.76:6832/2427238559,v1:172.21.15.76:6833/2427238559] up:boot 2024-04-26T23:42:24.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:24 smithi136 ceph-mon[33050]: fsmap 2 up:standby 2024-04-26T23:42:24.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:24 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T23:42:24.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:24 smithi076 ceph-mon[32774]: Deploying daemon mds.j on smithi072 2024-04-26T23:42:24.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:24 smithi076 ceph-mon[32774]: mds.? [v2:172.21.15.76:6832/2427238559,v1:172.21.15.76:6833/2427238559] up:boot 2024-04-26T23:42:24.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:24 smithi076 ceph-mon[32774]: fsmap 2 up:standby 2024-04-26T23:42:24.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:24 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T23:42:25.293 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:42:25.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:25 smithi072 ceph-mon[28184]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:25.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:25.769+0000 7ff562e00640 1 -- 172.21.15.72:0/1950776774 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff55c106560 msgr2=0x7ff55c10cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:25.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:25.769+0000 7ff562e00640 1 --2- 172.21.15.72:0/1950776774 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff55c106560 0x7ff55c10cdf0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff554013490 tx=0x7ff55402a2e0 comp rx=0 tx=0).stop 2024-04-26T23:42:25.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:25.770+0000 7ff562e00640 1 -- 172.21.15.72:0/1950776774 shutdown_connections 2024-04-26T23:42:25.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:25.770+0000 7ff562e00640 1 --2- 172.21.15.72:0/1950776774 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff55c106560 0x7ff55c10cdf0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:25.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:25.770+0000 7ff562e00640 1 --2- 172.21.15.72:0/1950776774 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff55c105ba0 0x7ff55c106020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:25.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:25.770+0000 7ff562e00640 1 --2- 172.21.15.72:0/1950776774 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff55c1049b0 0x7ff55c104dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:25.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:25.770+0000 7ff562e00640 1 -- 172.21.15.72:0/1950776774 >> 172.21.15.72:0/1950776774 conn(0x7ff55c0fff40 msgr2=0x7ff55c102360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:25.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:25.770+0000 7ff562e00640 1 -- 172.21.15.72:0/1950776774 shutdown_connections 2024-04-26T23:42:25.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:25.770+0000 7ff562e00640 1 -- 172.21.15.72:0/1950776774 wait complete. 2024-04-26T23:42:25.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:25.771+0000 7ff562e00640 1 Processor -- start 2024-04-26T23:42:25.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:25.772+0000 7ff562e00640 1 -- start start 2024-04-26T23:42:25.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:25.772+0000 7ff562e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff55c1049b0 0x7ff55c1b8730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:25.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:25.772+0000 7ff562e00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff55c105ba0 0x7ff55c1b8c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:25.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:25.772+0000 7ff562e00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff55c106560 0x7ff55c1bfc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:25.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:25.772+0000 7ff562e00640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7ff55c079820 con 0x7ff55c106560 2024-04-26T23:42:25.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:25.772+0000 7ff562e00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff55c0796a0 con 0x7ff55c1049b0 2024-04-26T23:42:25.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:25.772+0000 7ff562e00640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff55c0799a0 con 0x7ff55c105ba0 2024-04-26T23:42:25.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:25.773+0000 7ff561a00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff55c1049b0 0x7ff55c1b8730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:25.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:25.773+0000 7ff561a00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff55c1049b0 0x7ff55c1b8730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:59126/0 (socket says 172.21.15.72:59126) 2024-04-26T23:42:25.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:25.773+0000 7ff561a00640 1 -- 172.21.15.72:0/1153626017 learned_addr learned my addr 172.21.15.72:0/1153626017 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:42:25.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:25.773+0000 7ff562400640 1 --2- 172.21.15.72:0/1153626017 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff55c106560 0x7ff55c1bfc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:25.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:25.773+0000 7ff561000640 1 --2- 172.21.15.72:0/1153626017 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff55c105ba0 0x7ff55c1b8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:25.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:25.773+0000 7ff561000640 1 -- 172.21.15.72:0/1153626017 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff55c1049b0 msgr2=0x7ff55c1b8730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:25.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:25.773+0000 7ff561000640 1 --2- 172.21.15.72:0/1153626017 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff55c1049b0 0x7ff55c1b8730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:25.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:25.773+0000 7ff561000640 1 -- 172.21.15.72:0/1153626017 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff55c106560 msgr2=0x7ff55c1bfc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:25.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:25.773+0000 7ff561000640 1 --2- 172.21.15.72:0/1153626017 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff55c106560 0x7ff55c1bfc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:25.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:25.773+0000 7ff561000640 1 -- 172.21.15.72:0/1153626017 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff55c1c0300 con 0x7ff55c105ba0 2024-04-26T23:42:25.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:25.774+0000 7ff561000640 1 --2- 172.21.15.72:0/1153626017 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff55c105ba0 0x7ff55c1b8c70 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff54800e990 tx=0x7ff54800ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:25.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:25.774+0000 7ff55a000640 1 -- 172.21.15.72:0/1153626017 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff54800cca0 con 0x7ff55c105ba0 2024-04-26T23:42:25.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:25.774+0000 7ff562e00640 1 -- 172.21.15.72:0/1153626017 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff55c1c05f0 con 0x7ff55c105ba0 2024-04-26T23:42:25.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:25.774+0000 7ff562e00640 1 -- 172.21.15.72:0/1153626017 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff55c1c0b90 con 0x7ff55c105ba0 2024-04-26T23:42:25.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:25.775+0000 7ff55a000640 1 -- 172.21.15.72:0/1153626017 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff54800ce40 con 0x7ff55c105ba0 2024-04-26T23:42:25.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:25.775+0000 7ff55a000640 1 -- 172.21.15.72:0/1153626017 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff548011470 con 0x7ff55c105ba0 2024-04-26T23:42:25.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:25.776+0000 7ff562e00640 1 -- 172.21.15.72:0/1153626017 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff53c0052c0 con 0x7ff55c105ba0 2024-04-26T23:42:25.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:25.779+0000 7ff55a000640 1 -- 172.21.15.72:0/1153626017 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ff5480035f0 con 0x7ff55c105ba0 2024-04-26T23:42:25.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:25.783+0000 7ff55a000640 1 --2- 172.21.15.72:0/1153626017 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff5440a3c80 0x7ff5440a6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:25.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:25.784+0000 7ff55a000640 1 -- 172.21.15.72:0/1153626017 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7ff5480cfeb0 con 0x7ff55c105ba0 2024-04-26T23:42:25.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:25.784+0000 7ff561a00640 1 --2- 172.21.15.72:0/1153626017 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff5440a3c80 0x7ff5440a6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:25.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:25.785+0000 7ff561a00640 1 --2- 172.21.15.72:0/1153626017 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff5440a3c80 0x7ff5440a6140 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff5500058c0 tx=0x7ff550005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:25.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:25.790+0000 7ff55a000640 1 -- 172.21.15.72:0/1153626017 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff548016030 con 0x7ff55c105ba0 2024-04-26T23:42:25.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:25 smithi136 ceph-mon[33050]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:25.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:25 smithi076 ceph-mon[32774]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:26.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:26.039+0000 7ff562e00640 1 -- 172.21.15.72:0/1153626017 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff53c005880 con 0x7ff55c105ba0 2024-04-26T23:42:26.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:26.040+0000 7ff55a000640 1 -- 172.21.15.72:0/1153626017 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7ff54809cd30 con 0x7ff55c105ba0 2024-04-26T23:42:26.043 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:42:26.043 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1528342053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1528342053},{"type":"v1","addr":"172.21.15.136:6833","nonce":1528342053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6833/2427238559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":2427238559},{"type":"v1","addr":"172.21.15.76:6833","nonce":2427238559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-26T23:42:26.043 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 3 2024-04-26T23:42:26.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:26.042+0000 7ff562e00640 1 -- 172.21.15.72:0/1153626017 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff5440a3c80 msgr2=0x7ff5440a6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:26.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:26.042+0000 7ff562e00640 1 --2- 172.21.15.72:0/1153626017 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff5440a3c80 0x7ff5440a6140 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff5500058c0 tx=0x7ff550005f90 comp rx=0 tx=0).stop 2024-04-26T23:42:26.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:26.042+0000 7ff562e00640 1 -- 172.21.15.72:0/1153626017 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff55c105ba0 msgr2=0x7ff55c1b8c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:26.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:26.042+0000 7ff562e00640 1 --2- 172.21.15.72:0/1153626017 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff55c105ba0 0x7ff55c1b8c70 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff54800e990 tx=0x7ff54800ee50 comp rx=0 tx=0).stop 2024-04-26T23:42:26.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:26.043+0000 7ff562e00640 1 -- 172.21.15.72:0/1153626017 shutdown_connections 2024-04-26T23:42:26.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:26.043+0000 7ff562e00640 1 --2- 172.21.15.72:0/1153626017 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff5440a3c80 0x7ff5440a6140 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:26.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:26.043+0000 7ff562e00640 1 --2- 172.21.15.72:0/1153626017 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff55c106560 0x7ff55c1bfc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:26.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:26.043+0000 7ff562e00640 1 --2- 172.21.15.72:0/1153626017 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff55c105ba0 0x7ff55c1b8c70 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:26.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:26.043+0000 7ff562e00640 1 --2- 172.21.15.72:0/1153626017 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff55c1049b0 0x7ff55c1b8730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:26.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:26.043+0000 7ff562e00640 1 -- 172.21.15.72:0/1153626017 >> 172.21.15.72:0/1153626017 conn(0x7ff55c0fff40 msgr2=0x7ff55c100610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:26.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:26.043+0000 7ff562e00640 1 -- 172.21.15.72:0/1153626017 shutdown_connections 2024-04-26T23:42:26.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:26.043+0000 7ff562e00640 1 -- 172.21.15.72:0/1153626017 wait complete. 2024-04-26T23:42:26.712 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:26 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/1153626017' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:26.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:26 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/1153626017' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:26.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:26 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/1153626017' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:27.043 INFO:journalctl@ceph.mds.j.smithi072.stdout:Apr 26 23:42:26 smithi072 systemd[1]: Started Ceph mds.j for 3c8453ae-0425-11ef-bc93-c7b262605968. 2024-04-26T23:42:27.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:27 smithi072 ceph-mon[28184]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:27.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:27 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:27.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:27 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:27.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:27 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:27.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:27 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:42:27.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:27 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:42:27.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:27 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:42:27.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:27 smithi136 ceph-mon[33050]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:27.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:27 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:27.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:27 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:27.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:27 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:27.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:27 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:42:27.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:27 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:42:27.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:27 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:42:27.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:27 smithi076 ceph-mon[32774]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:27.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:27 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:27.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:27 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:27.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:27 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:27.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:27 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:42:27.930 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:27 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:42:27.930 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:27 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:42:28.073 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:42:28.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.286+0000 7f1988600640 1 -- 172.21.15.72:0/1365634760 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1980104b90 msgr2=0x7f1980104fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:28.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.286+0000 7f1988600640 1 --2- 172.21.15.72:0/1365634760 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1980104b90 0x7f1980104fb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f1974009900 tx=0x7f197402a3f0 comp rx=0 tx=0).stop 2024-04-26T23:42:28.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.286+0000 7f1988600640 1 -- 172.21.15.72:0/1365634760 shutdown_connections 2024-04-26T23:42:28.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.286+0000 7f1988600640 1 --2- 172.21.15.72:0/1365634760 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1980106740 0x7f198010cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:28.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.286+0000 7f1988600640 1 --2- 172.21.15.72:0/1365634760 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1980105d80 0x7f1980106200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:28.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.286+0000 7f1988600640 1 --2- 172.21.15.72:0/1365634760 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1980104b90 0x7f1980104fb0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:28.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.287+0000 7f1988600640 1 -- 172.21.15.72:0/1365634760 >> 172.21.15.72:0/1365634760 conn(0x7f1980100120 msgr2=0x7f1980102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:28.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.287+0000 7f1988600640 1 -- 172.21.15.72:0/1365634760 shutdown_connections 2024-04-26T23:42:28.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.287+0000 7f1988600640 1 -- 172.21.15.72:0/1365634760 wait complete. 2024-04-26T23:42:28.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.288+0000 7f1988600640 1 Processor -- start 2024-04-26T23:42:28.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.288+0000 7f1988600640 1 -- start start 2024-04-26T23:42:28.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.288+0000 7f1988600640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1980104b90 0x7f198019ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:28.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.288+0000 7f1988600640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1980105d80 0x7f198019b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:28.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.288+0000 7f1988600640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1980106740 0x7f19801a2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:28.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.288+0000 7f1988600640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f19801106b0 con 0x7f1980105d80 2024-04-26T23:42:28.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.288+0000 7f1988600640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f1980110530 con 0x7f1980106740 2024-04-26T23:42:28.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.288+0000 7f1988600640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f1980110830 con 0x7f1980104b90 2024-04-26T23:42:28.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.288+0000 7f1985200640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1980105d80 0x7f198019b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:28.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.288+0000 7f1985200640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1980105d80 0x7f198019b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60088/0 (socket says 172.21.15.72:60088) 2024-04-26T23:42:28.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.288+0000 7f1985200640 1 -- 172.21.15.72:0/112764941 learned_addr learned my addr 172.21.15.72:0/112764941 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:42:28.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.288+0000 7f1986600640 1 --2- 172.21.15.72:0/112764941 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1980106740 0x7f19801a2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:28.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.288+0000 7f1985c00640 1 --2- 172.21.15.72:0/112764941 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1980104b90 0x7f198019ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:28.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.289+0000 7f1985200640 1 -- 172.21.15.72:0/112764941 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1980104b90 msgr2=0x7f198019ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:28.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.289+0000 7f1985200640 1 --2- 172.21.15.72:0/112764941 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1980104b90 0x7f198019ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:28.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.289+0000 7f1985200640 1 -- 172.21.15.72:0/112764941 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1980106740 msgr2=0x7f19801a2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:28.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.289+0000 7f1985200640 1 --2- 172.21.15.72:0/112764941 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1980106740 0x7f19801a2390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:28.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.289+0000 7f1985200640 1 -- 172.21.15.72:0/112764941 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19801a2a00 con 0x7f1980105d80 2024-04-26T23:42:28.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.289+0000 7f1985200640 1 --2- 172.21.15.72:0/112764941 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1980105d80 0x7f198019b320 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f196c00e9f0 tx=0x7f196c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:28.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.289+0000 7f197e000640 1 -- 172.21.15.72:0/112764941 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f196c00ccc0 con 0x7f1980105d80 2024-04-26T23:42:28.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.289+0000 7f197e000640 1 -- 172.21.15.72:0/112764941 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f196c00ce60 con 0x7f1980105d80 2024-04-26T23:42:28.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.289+0000 7f1988600640 1 -- 172.21.15.72:0/112764941 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19801a2cf0 con 0x7f1980105d80 2024-04-26T23:42:28.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.289+0000 7f197e000640 1 -- 172.21.15.72:0/112764941 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f196c010430 con 0x7f1980105d80 2024-04-26T23:42:28.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.290+0000 7f1988600640 1 -- 172.21.15.72:0/112764941 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19801a3190 con 0x7f1980105d80 2024-04-26T23:42:28.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.291+0000 7f1988600640 1 -- 172.21.15.72:0/112764941 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f198010bf20 con 0x7f1980105d80 2024-04-26T23:42:28.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.293+0000 7f197e000640 1 -- 172.21.15.72:0/112764941 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f196c0046c0 con 0x7f1980105d80 2024-04-26T23:42:28.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.294+0000 7f197e000640 1 --2- 172.21.15.72:0/112764941 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f19680a3bb0 0x7f19680a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:28.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.294+0000 7f197e000640 1 -- 172.21.15.72:0/112764941 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f196c0d0310 con 0x7f1980105d80 2024-04-26T23:42:28.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.294+0000 7f1985c00640 1 --2- 172.21.15.72:0/112764941 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f19680a3bb0 0x7f19680a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:28.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.295+0000 7f1985c00640 1 --2- 172.21.15.72:0/112764941 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f19680a3bb0 0x7f19680a6070 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1974009230 tx=0x7f19740314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:28.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.299+0000 7f197e000640 1 -- 172.21.15.72:0/112764941 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f196c09d190 con 0x7f1980105d80 2024-04-26T23:42:28.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.549+0000 7f1988600640 1 -- 172.21.15.72:0/112764941 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f198002dad0 con 0x7f1980105d80 2024-04-26T23:42:28.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.549+0000 7f197e000640 1 -- 172.21.15.72:0/112764941 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3046 (secure 0 0 0) 0x7f196c0bd0d0 con 0x7f1980105d80 2024-04-26T23:42:28.553 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:42:28.553 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/2386456166","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2386456166},{"type":"v1","addr":"172.21.15.72:6835","nonce":2386456166}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1528342053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1528342053},{"type":"v1","addr":"172.21.15.136:6833","nonce":1528342053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6833/2427238559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":2427238559},{"type":"v1","addr":"172.21.15.76:6833","nonce":2427238559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-26T23:42:28.553 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 4 2024-04-26T23:42:28.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.552+0000 7f1988600640 1 -- 172.21.15.72:0/112764941 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f19680a3bb0 msgr2=0x7f19680a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:28.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.552+0000 7f1988600640 1 --2- 172.21.15.72:0/112764941 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f19680a3bb0 0x7f19680a6070 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1974009230 tx=0x7f19740314a0 comp rx=0 tx=0).stop 2024-04-26T23:42:28.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.552+0000 7f1988600640 1 -- 172.21.15.72:0/112764941 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1980105d80 msgr2=0x7f198019b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:28.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.552+0000 7f1988600640 1 --2- 172.21.15.72:0/112764941 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1980105d80 0x7f198019b320 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f196c00e9f0 tx=0x7f196c00edf0 comp rx=0 tx=0).stop 2024-04-26T23:42:28.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.552+0000 7f1988600640 1 -- 172.21.15.72:0/112764941 shutdown_connections 2024-04-26T23:42:28.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.552+0000 7f1988600640 1 --2- 172.21.15.72:0/112764941 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f19680a3bb0 0x7f19680a6070 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:28.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.552+0000 7f1988600640 1 --2- 172.21.15.72:0/112764941 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1980106740 0x7f19801a2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:28.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.552+0000 7f1988600640 1 --2- 172.21.15.72:0/112764941 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1980105d80 0x7f198019b320 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:28.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.552+0000 7f1988600640 1 --2- 172.21.15.72:0/112764941 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1980104b90 0x7f198019ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:28.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.552+0000 7f1988600640 1 -- 172.21.15.72:0/112764941 >> 172.21.15.72:0/112764941 conn(0x7f1980100120 msgr2=0x7f19801090f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:28.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.552+0000 7f1988600640 1 -- 172.21.15.72:0/112764941 shutdown_connections 2024-04-26T23:42:28.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:28.552+0000 7f1988600640 1 -- 172.21.15.72:0/112764941 wait complete. 2024-04-26T23:42:28.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:28 smithi072 ceph-mon[28184]: Deploying daemon mds.d on smithi072 2024-04-26T23:42:28.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:28 smithi072 ceph-mon[28184]: mds.? [v2:172.21.15.72:6834/2386456166,v1:172.21.15.72:6835/2386456166] up:boot 2024-04-26T23:42:28.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:28 smithi072 ceph-mon[28184]: fsmap 3 up:standby 2024-04-26T23:42:28.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:28 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T23:42:28.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:28 smithi136 ceph-mon[33050]: Deploying daemon mds.d on smithi072 2024-04-26T23:42:28.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:28 smithi136 ceph-mon[33050]: mds.? [v2:172.21.15.72:6834/2386456166,v1:172.21.15.72:6835/2386456166] up:boot 2024-04-26T23:42:28.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:28 smithi136 ceph-mon[33050]: fsmap 3 up:standby 2024-04-26T23:42:28.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:28 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T23:42:28.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:28 smithi076 ceph-mon[32774]: Deploying daemon mds.d on smithi072 2024-04-26T23:42:28.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:28 smithi076 ceph-mon[32774]: mds.? [v2:172.21.15.72:6834/2386456166,v1:172.21.15.72:6835/2386456166] up:boot 2024-04-26T23:42:28.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:28 smithi076 ceph-mon[32774]: fsmap 3 up:standby 2024-04-26T23:42:28.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:28 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T23:42:29.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:29 smithi072 ceph-mon[28184]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:29.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:29 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/112764941' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:29.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:29 smithi136 ceph-mon[33050]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:29.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:29 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/112764941' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:29.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:29 smithi076 ceph-mon[32774]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:29.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:29 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/112764941' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:30.544 INFO:journalctl@ceph.mds.d.smithi072.stdout:Apr 26 23:42:30 smithi072 systemd[1]: Starting Ceph mds.d for 3c8453ae-0425-11ef-bc93-c7b262605968... 2024-04-26T23:42:30.575 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:42:30.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:30.788+0000 7fd42ce00640 1 -- 172.21.15.72:0/1946236906 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd428104520 msgr2=0x7fd42810adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:30.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:30.788+0000 7fd42ce00640 1 --2- 172.21.15.72:0/1946236906 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd428104520 0x7fd42810adb0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd420009be0 tx=0x7fd42002a6c0 comp rx=0 tx=0).stop 2024-04-26T23:42:30.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:30.789+0000 7fd42ce00640 1 -- 172.21.15.72:0/1946236906 shutdown_connections 2024-04-26T23:42:30.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:30.789+0000 7fd42ce00640 1 --2- 172.21.15.72:0/1946236906 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd428103b60 0x7fd428103fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:30.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:30.789+0000 7fd42ce00640 1 --2- 172.21.15.72:0/1946236906 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd428104520 0x7fd42810adb0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:30.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:30.789+0000 7fd42ce00640 1 --2- 172.21.15.72:0/1946236906 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd428102970 0x7fd428102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:30.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:30.789+0000 7fd42ce00640 1 -- 172.21.15.72:0/1946236906 >> 172.21.15.72:0/1946236906 conn(0x7fd4280fdee0 msgr2=0x7fd428100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:30.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:30.789+0000 7fd42ce00640 1 -- 172.21.15.72:0/1946236906 shutdown_connections 2024-04-26T23:42:30.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:30.789+0000 7fd42ce00640 1 -- 172.21.15.72:0/1946236906 wait complete. 2024-04-26T23:42:30.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:30.790+0000 7fd42ce00640 1 Processor -- start 2024-04-26T23:42:30.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:30.790+0000 7fd42ce00640 1 -- start start 2024-04-26T23:42:30.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:30.790+0000 7fd42ce00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd428102970 0x7fd42819ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:30.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:30.790+0000 7fd42ce00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd428103b60 0x7fd42819b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:30.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:30.790+0000 7fd42ce00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd428104520 0x7fd4281a22c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:30.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:30.790+0000 7fd42ce00640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fd42810e480 con 0x7fd428103b60 2024-04-26T23:42:30.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:30.790+0000 7fd42ce00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fd42810e300 con 0x7fd428104520 2024-04-26T23:42:30.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:30.790+0000 7fd42ce00640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd42810e600 con 0x7fd428102970 2024-04-26T23:42:30.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:30.791+0000 7fd426800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd428104520 0x7fd4281a22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:30.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:30.791+0000 7fd426800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd428104520 0x7fd4281a22c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:54552/0 (socket says 172.21.15.72:54552) 2024-04-26T23:42:30.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:30.791+0000 7fd426800640 1 -- 172.21.15.72:0/4233618350 learned_addr learned my addr 172.21.15.72:0/4233618350 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:42:30.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:30.791+0000 7fd425400640 1 --2- 172.21.15.72:0/4233618350 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd428103b60 0x7fd42819b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:30.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:30.791+0000 7fd425e00640 1 --2- 172.21.15.72:0/4233618350 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd428102970 0x7fd42819ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:30.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:30.791+0000 7fd426800640 1 -- 172.21.15.72:0/4233618350 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd428102970 msgr2=0x7fd42819ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:30.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:30.791+0000 7fd426800640 1 --2- 172.21.15.72:0/4233618350 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd428102970 0x7fd42819ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:30.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:30.791+0000 7fd426800640 1 -- 172.21.15.72:0/4233618350 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd428103b60 msgr2=0x7fd42819b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:30.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:30.791+0000 7fd426800640 1 --2- 172.21.15.72:0/4233618350 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd428103b60 0x7fd42819b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:30.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:30.791+0000 7fd426800640 1 -- 172.21.15.72:0/4233618350 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4281a2960 con 0x7fd428104520 2024-04-26T23:42:30.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:30.791+0000 7fd425400640 1 --2- 172.21.15.72:0/4233618350 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd428103b60 0x7fd42819b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:42:30.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:30.791+0000 7fd425e00640 1 --2- 172.21.15.72:0/4233618350 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd428102970 0x7fd42819ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:42:30.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:30.791+0000 7fd426800640 1 --2- 172.21.15.72:0/4233618350 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd428104520 0x7fd4281a22c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd420004af0 tx=0x7fd420030d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:30.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:30.792+0000 7fd41ea00640 1 -- 172.21.15.72:0/4233618350 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd420041020 con 0x7fd428104520 2024-04-26T23:42:30.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:30.792+0000 7fd41ea00640 1 -- 172.21.15.72:0/4233618350 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd4200318f0 con 0x7fd428104520 2024-04-26T23:42:30.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:30.792+0000 7fd41ea00640 1 -- 172.21.15.72:0/4233618350 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd420036040 con 0x7fd428104520 2024-04-26T23:42:30.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:30.792+0000 7fd42ce00640 1 -- 172.21.15.72:0/4233618350 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4281a2bf0 con 0x7fd428104520 2024-04-26T23:42:30.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:30.792+0000 7fd42ce00640 1 -- 172.21.15.72:0/4233618350 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4281a3090 con 0x7fd428104520 2024-04-26T23:42:30.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:30.793+0000 7fd42ce00640 1 -- 172.21.15.72:0/4233618350 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4281a3260 con 0x7fd428104520 2024-04-26T23:42:30.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:30.795+0000 7fd41ea00640 1 -- 172.21.15.72:0/4233618350 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd420044020 con 0x7fd428104520 2024-04-26T23:42:30.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:30.796+0000 7fd41ea00640 1 --2- 172.21.15.72:0/4233618350 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fd40c0a3bb0 0x7fd40c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:30.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:30.796+0000 7fd41ea00640 1 -- 172.21.15.72:0/4233618350 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fd4200f0e30 con 0x7fd428104520 2024-04-26T23:42:30.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:30.797+0000 7fd425e00640 1 --2- 172.21.15.72:0/4233618350 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fd40c0a3bb0 0x7fd40c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:30.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:30.797+0000 7fd425e00640 1 --2- 172.21.15.72:0/4233618350 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fd40c0a3bb0 0x7fd40c0a6070 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd42819c220 tx=0x7fd418005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:30.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:30.801+0000 7fd41ea00640 1 -- 172.21.15.72:0/4233618350 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd42004b020 con 0x7fd428104520 2024-04-26T23:42:30.880 INFO:journalctl@ceph.mds.d.smithi072.stdout:Apr 26 23:42:30 smithi072 podman[67067]: 2024-04-26 23:42:30.469249727 +0000 UTC m=+0.036445794 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T23:42:30.880 INFO:journalctl@ceph.mds.d.smithi072.stdout:Apr 26 23:42:30 smithi072 podman[67067]: 2024-04-26 23:42:30.626294228 +0000 UTC m=+0.193490282 container create 7714218b798c9c80bb522d36f40dc48c40d38882aad1778e8918cefac5563bc7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-d, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T23:42:30.880 INFO:journalctl@ceph.mds.d.smithi072.stdout:Apr 26 23:42:30 smithi072 podman[67067]: 2024-04-26 23:42:30.87636855 +0000 UTC m=+0.443564650 container init 7714218b798c9c80bb522d36f40dc48c40d38882aad1778e8918cefac5563bc7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-d, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T23:42:31.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:31.048+0000 7fd42ce00640 1 -- 172.21.15.72:0/4233618350 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd4281a3430 con 0x7fd428104520 2024-04-26T23:42:31.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:31.049+0000 7fd41ea00640 1 -- 172.21.15.72:0/4233618350 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3046 (secure 0 0 0) 0x7fd4200bdc10 con 0x7fd428104520 2024-04-26T23:42:31.053 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:42:31.053 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/2386456166","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2386456166},{"type":"v1","addr":"172.21.15.72:6835","nonce":2386456166}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1528342053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1528342053},{"type":"v1","addr":"172.21.15.136:6833","nonce":1528342053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6833/2427238559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":2427238559},{"type":"v1","addr":"172.21.15.76:6833","nonce":2427238559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-26T23:42:31.053 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 4 2024-04-26T23:42:31.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:31.052+0000 7fd42ce00640 1 -- 172.21.15.72:0/4233618350 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fd40c0a3bb0 msgr2=0x7fd40c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:31.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:31.052+0000 7fd42ce00640 1 --2- 172.21.15.72:0/4233618350 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fd40c0a3bb0 0x7fd40c0a6070 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd42819c220 tx=0x7fd418005f90 comp rx=0 tx=0).stop 2024-04-26T23:42:31.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:31.052+0000 7fd42ce00640 1 -- 172.21.15.72:0/4233618350 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd428104520 msgr2=0x7fd4281a22c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:31.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:31.052+0000 7fd42ce00640 1 --2- 172.21.15.72:0/4233618350 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd428104520 0x7fd4281a22c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd420004af0 tx=0x7fd420030d50 comp rx=0 tx=0).stop 2024-04-26T23:42:31.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:31.052+0000 7fd42ce00640 1 -- 172.21.15.72:0/4233618350 shutdown_connections 2024-04-26T23:42:31.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:31.052+0000 7fd42ce00640 1 --2- 172.21.15.72:0/4233618350 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fd40c0a3bb0 0x7fd40c0a6070 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:31.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:31.052+0000 7fd42ce00640 1 --2- 172.21.15.72:0/4233618350 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd428104520 0x7fd4281a22c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:31.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:31.052+0000 7fd42ce00640 1 --2- 172.21.15.72:0/4233618350 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd428103b60 0x7fd42819b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:31.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:31.052+0000 7fd42ce00640 1 --2- 172.21.15.72:0/4233618350 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd428102970 0x7fd42819ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:31.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:31.052+0000 7fd42ce00640 1 -- 172.21.15.72:0/4233618350 >> 172.21.15.72:0/4233618350 conn(0x7fd4280fdee0 msgr2=0x7fd428074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:31.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:31.052+0000 7fd42ce00640 1 -- 172.21.15.72:0/4233618350 shutdown_connections 2024-04-26T23:42:31.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:31.052+0000 7fd42ce00640 1 -- 172.21.15.72:0/4233618350 wait complete. 2024-04-26T23:42:31.294 INFO:journalctl@ceph.mds.d.smithi072.stdout:Apr 26 23:42:30 smithi072 podman[67067]: 2024-04-26 23:42:30.887415219 +0000 UTC m=+0.454611281 container start 7714218b798c9c80bb522d36f40dc48c40d38882aad1778e8918cefac5563bc7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-d, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T23:42:31.294 INFO:journalctl@ceph.mds.d.smithi072.stdout:Apr 26 23:42:30 smithi072 ceph-mds[67125]: -- 172.21.15.72:0/503924774 <== mon.0 v2:172.21.15.72:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5592deaf7a40 con 0x5592deacb000 2024-04-26T23:42:31.294 INFO:journalctl@ceph.mds.d.smithi072.stdout:Apr 26 23:42:30 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-d[67102]: starting mds.d at 2024-04-26T23:42:31.294 INFO:journalctl@ceph.mds.d.smithi072.stdout:Apr 26 23:42:30 smithi072 bash[67067]: 7714218b798c9c80bb522d36f40dc48c40d38882aad1778e8918cefac5563bc7 2024-04-26T23:42:31.294 INFO:journalctl@ceph.mds.d.smithi072.stdout:Apr 26 23:42:30 smithi072 systemd[1]: Started Ceph mds.d for 3c8453ae-0425-11ef-bc93-c7b262605968. 2024-04-26T23:42:31.788 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:31 smithi076 ceph-mon[32774]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:31.788 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:31 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/4233618350' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:31.788 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:31 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:31.788 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:31 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:31.788 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:31 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:31.788 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:31 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:42:31.789 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:31 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:42:31.789 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:31 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:42:31.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:31 smithi072 ceph-mon[28184]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:31.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:31 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/4233618350' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:31.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:31 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:31.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:31 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:31.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:31 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:31.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:31 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:42:31.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:31 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:42:31.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:31 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:42:31.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:31 smithi136 ceph-mon[33050]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:31.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:31 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/4233618350' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:31.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:31 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:31.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:31 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:31.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:31 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:31.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:31 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:42:31.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:31 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:42:31.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:31 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:42:32.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:32 smithi072 ceph-mon[28184]: Deploying daemon mds.e on smithi076 2024-04-26T23:42:32.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:32 smithi072 ceph-mon[28184]: mds.? [v2:172.21.15.72:6836/4050689894,v1:172.21.15.72:6837/4050689894] up:boot 2024-04-26T23:42:32.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:32 smithi072 ceph-mon[28184]: fsmap 4 up:standby 2024-04-26T23:42:32.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:32 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T23:42:32.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:32 smithi136 ceph-mon[33050]: Deploying daemon mds.e on smithi076 2024-04-26T23:42:32.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:32 smithi136 ceph-mon[33050]: mds.? [v2:172.21.15.72:6836/4050689894,v1:172.21.15.72:6837/4050689894] up:boot 2024-04-26T23:42:32.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:32 smithi136 ceph-mon[33050]: fsmap 4 up:standby 2024-04-26T23:42:32.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:32 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T23:42:32.816 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:32 smithi076 ceph-mon[32774]: Deploying daemon mds.e on smithi076 2024-04-26T23:42:32.816 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:32 smithi076 ceph-mon[32774]: mds.? [v2:172.21.15.72:6836/4050689894,v1:172.21.15.72:6837/4050689894] up:boot 2024-04-26T23:42:32.816 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:32 smithi076 ceph-mon[32774]: fsmap 4 up:standby 2024-04-26T23:42:32.816 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:32 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T23:42:33.076 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:42:33.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.280+0000 7f6a01000640 1 -- 172.21.15.72:0/3636706477 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f69fc105d80 msgr2=0x7f69fc106200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:33.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.280+0000 7f6a01000640 1 --2- 172.21.15.72:0/3636706477 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f69fc105d80 0x7f69fc106200 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f69e4009900 tx=0x7f69e402a3f0 comp rx=0 tx=0).stop 2024-04-26T23:42:33.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.280+0000 7f6a01000640 1 -- 172.21.15.72:0/3636706477 shutdown_connections 2024-04-26T23:42:33.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.280+0000 7f6a01000640 1 --2- 172.21.15.72:0/3636706477 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f69fc105d80 0x7f69fc106200 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:33.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.280+0000 7f6a01000640 1 --2- 172.21.15.72:0/3636706477 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f69fc106740 0x7f69fc10cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:33.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.280+0000 7f6a01000640 1 --2- 172.21.15.72:0/3636706477 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69fc104b90 0x7f69fc104fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:33.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.280+0000 7f6a01000640 1 -- 172.21.15.72:0/3636706477 >> 172.21.15.72:0/3636706477 conn(0x7f69fc100120 msgr2=0x7f69fc102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:33.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.281+0000 7f6a01000640 1 -- 172.21.15.72:0/3636706477 shutdown_connections 2024-04-26T23:42:33.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.281+0000 7f6a01000640 1 -- 172.21.15.72:0/3636706477 wait complete. 2024-04-26T23:42:33.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.281+0000 7f6a01000640 1 Processor -- start 2024-04-26T23:42:33.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.282+0000 7f6a01000640 1 -- start start 2024-04-26T23:42:33.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.282+0000 7f6a01000640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f69fc104b90 0x7f69fc19adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:33.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.282+0000 7f6a01000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f69fc105d80 0x7f69fc19b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:33.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.282+0000 7f6a01000640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69fc106740 0x7f69fc1a2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:33.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.282+0000 7f6a01000640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f69fc1106f0 con 0x7f69fc106740 2024-04-26T23:42:33.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.282+0000 7f6a01000640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f69fc110570 con 0x7f69fc105d80 2024-04-26T23:42:33.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.282+0000 7f6a01000640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f69fc110870 con 0x7f69fc104b90 2024-04-26T23:42:33.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.282+0000 7f69fa800640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69fc106740 0x7f69fc1a2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:33.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.282+0000 7f69f9400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f69fc105d80 0x7f69fc19b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:33.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.282+0000 7f69fa800640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69fc106740 0x7f69fc1a2320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60138/0 (socket says 172.21.15.72:60138) 2024-04-26T23:42:33.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.282+0000 7f69fa800640 1 -- 172.21.15.72:0/413742875 learned_addr learned my addr 172.21.15.72:0/413742875 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:42:33.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.282+0000 7f69f9e00640 1 --2- 172.21.15.72:0/413742875 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f69fc104b90 0x7f69fc19adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:33.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.283+0000 7f69f9400640 1 -- 172.21.15.72:0/413742875 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f69fc104b90 msgr2=0x7f69fc19adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:33.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.283+0000 7f69f9400640 1 --2- 172.21.15.72:0/413742875 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f69fc104b90 0x7f69fc19adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:33.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.283+0000 7f69f9400640 1 -- 172.21.15.72:0/413742875 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69fc106740 msgr2=0x7f69fc1a2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:33.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.283+0000 7f69f9400640 1 --2- 172.21.15.72:0/413742875 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69fc106740 0x7f69fc1a2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:33.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.283+0000 7f69f9400640 1 -- 172.21.15.72:0/413742875 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69fc1a29c0 con 0x7f69fc105d80 2024-04-26T23:42:33.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.283+0000 7f69f9400640 1 --2- 172.21.15.72:0/413742875 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f69fc105d80 0x7f69fc19b2f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f69e402a9e0 tx=0x7f69e4003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:33.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.283+0000 7f69f2a00640 1 -- 172.21.15.72:0/413742875 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69e403f070 con 0x7f69fc105d80 2024-04-26T23:42:33.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.283+0000 7f69f2a00640 1 -- 172.21.15.72:0/413742875 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f69e4031c50 con 0x7f69fc105d80 2024-04-26T23:42:33.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.283+0000 7f6a01000640 1 -- 172.21.15.72:0/413742875 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69fc1a2c50 con 0x7f69fc105d80 2024-04-26T23:42:33.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.283+0000 7f69f2a00640 1 -- 172.21.15.72:0/413742875 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69e4035030 con 0x7f69fc105d80 2024-04-26T23:42:33.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.283+0000 7f6a01000640 1 -- 172.21.15.72:0/413742875 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69fc1a3160 con 0x7f69fc105d80 2024-04-26T23:42:33.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.284+0000 7f6a01000640 1 -- 172.21.15.72:0/413742875 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69fc10beb0 con 0x7f69fc105d80 2024-04-26T23:42:33.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.286+0000 7f69f2a00640 1 -- 172.21.15.72:0/413742875 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f69e4042070 con 0x7f69fc105d80 2024-04-26T23:42:33.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.287+0000 7f69f2a00640 1 --2- 172.21.15.72:0/413742875 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f69e00a3c30 0x7f69e00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:33.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.288+0000 7f69f2a00640 1 -- 172.21.15.72:0/413742875 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f69e40f0650 con 0x7f69fc105d80 2024-04-26T23:42:33.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.288+0000 7f69f9e00640 1 --2- 172.21.15.72:0/413742875 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f69e00a3c30 0x7f69e00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:33.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.288+0000 7f69f9e00640 1 --2- 172.21.15.72:0/413742875 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f69e00a3c30 0x7f69e00a60f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f69fc19c280 tx=0x7f69ec005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:33.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.292+0000 7f69f2a00640 1 -- 172.21.15.72:0/413742875 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f69e40bcde0 con 0x7f69fc105d80 2024-04-26T23:42:33.429 INFO:journalctl@ceph.mds.e.smithi076.stdout:Apr 26 23:42:33 smithi076 systemd[1]: Starting Ceph mds.e for 3c8453ae-0425-11ef-bc93-c7b262605968... 2024-04-26T23:42:33.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.539+0000 7f6a01000640 1 -- 172.21.15.72:0/413742875 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f69fc063560 con 0x7f69fc105d80 2024-04-26T23:42:33.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.540+0000 7f69f2a00640 1 -- 172.21.15.72:0/413742875 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3871 (secure 0 0 0) 0x7f69e40c10e0 con 0x7f69fc105d80 2024-04-26T23:42:33.545 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:42:33.545 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/2386456166","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2386456166},{"type":"v1","addr":"172.21.15.72:6835","nonce":2386456166}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1528342053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1528342053},{"type":"v1","addr":"172.21.15.136:6833","nonce":1528342053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6833/2427238559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":2427238559},{"type":"v1","addr":"172.21.15.76:6833","nonce":2427238559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24491,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/4050689894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":4050689894},{"type":"v1","addr":"172.21.15.72:6837","nonce":4050689894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[]} 2024-04-26T23:42:33.545 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 5 2024-04-26T23:42:33.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.543+0000 7f6a01000640 1 -- 172.21.15.72:0/413742875 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f69e00a3c30 msgr2=0x7f69e00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:33.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.543+0000 7f6a01000640 1 --2- 172.21.15.72:0/413742875 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f69e00a3c30 0x7f69e00a60f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f69fc19c280 tx=0x7f69ec005f90 comp rx=0 tx=0).stop 2024-04-26T23:42:33.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.543+0000 7f6a01000640 1 -- 172.21.15.72:0/413742875 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f69fc105d80 msgr2=0x7f69fc19b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:33.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.543+0000 7f6a01000640 1 --2- 172.21.15.72:0/413742875 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f69fc105d80 0x7f69fc19b2f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f69e402a9e0 tx=0x7f69e4003c10 comp rx=0 tx=0).stop 2024-04-26T23:42:33.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.543+0000 7f6a01000640 1 -- 172.21.15.72:0/413742875 shutdown_connections 2024-04-26T23:42:33.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.543+0000 7f6a01000640 1 --2- 172.21.15.72:0/413742875 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f69e00a3c30 0x7f69e00a60f0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:33.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.543+0000 7f6a01000640 1 --2- 172.21.15.72:0/413742875 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f69fc105d80 0x7f69fc19b2f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:33.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.543+0000 7f6a01000640 1 --2- 172.21.15.72:0/413742875 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69fc106740 0x7f69fc1a2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:33.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.543+0000 7f6a01000640 1 --2- 172.21.15.72:0/413742875 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f69fc104b90 0x7f69fc19adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:33.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.543+0000 7f6a01000640 1 -- 172.21.15.72:0/413742875 >> 172.21.15.72:0/413742875 conn(0x7f69fc100120 msgr2=0x7f69fc109080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:33.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.543+0000 7f6a01000640 1 -- 172.21.15.72:0/413742875 shutdown_connections 2024-04-26T23:42:33.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:33.543+0000 7f6a01000640 1 -- 172.21.15.72:0/413742875 wait complete. 2024-04-26T23:42:33.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:33 smithi072 ceph-mon[28184]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:33.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:33 smithi136 ceph-mon[33050]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:33.853 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:33 smithi076 ceph-mon[32774]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:33.853 INFO:journalctl@ceph.mds.e.smithi076.stdout:Apr 26 23:42:33 smithi076 podman[61609]: 2024-04-26 23:42:33.475060267 +0000 UTC m=+0.018260739 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T23:42:33.853 INFO:journalctl@ceph.mds.e.smithi076.stdout:Apr 26 23:42:33 smithi076 podman[61609]: 2024-04-26 23:42:33.596815596 +0000 UTC m=+0.140016039 container create 6131b82b68ba4690383dafbaf2fce920ca9aeb94d5e6eef1d215de34e949a7a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-e, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T23:42:34.179 INFO:journalctl@ceph.mds.e.smithi076.stdout:Apr 26 23:42:33 smithi076 podman[61609]: 2024-04-26 23:42:33.849972536 +0000 UTC m=+0.393172982 container init 6131b82b68ba4690383dafbaf2fce920ca9aeb94d5e6eef1d215de34e949a7a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-e, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-26T23:42:34.179 INFO:journalctl@ceph.mds.e.smithi076.stdout:Apr 26 23:42:33 smithi076 podman[61609]: 2024-04-26 23:42:33.852609223 +0000 UTC m=+0.395809670 container start 6131b82b68ba4690383dafbaf2fce920ca9aeb94d5e6eef1d215de34e949a7a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-e, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-26T23:42:34.179 INFO:journalctl@ceph.mds.e.smithi076.stdout:Apr 26 23:42:33 smithi076 ceph-mds[61628]: -- 172.21.15.76:0/1753320495 <== mon.2 v2:172.21.15.76:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563263d21a40 con 0x563263cf4800 2024-04-26T23:42:34.179 INFO:journalctl@ceph.mds.e.smithi076.stdout:Apr 26 23:42:33 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-e[61624]: starting mds.e at 2024-04-26T23:42:34.179 INFO:journalctl@ceph.mds.e.smithi076.stdout:Apr 26 23:42:33 smithi076 bash[61609]: 6131b82b68ba4690383dafbaf2fce920ca9aeb94d5e6eef1d215de34e949a7a6 2024-04-26T23:42:34.179 INFO:journalctl@ceph.mds.e.smithi076.stdout:Apr 26 23:42:33 smithi076 systemd[1]: Started Ceph mds.e for 3c8453ae-0425-11ef-bc93-c7b262605968. 2024-04-26T23:42:34.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:34 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/413742875' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:34.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:34 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:34.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:34 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:34.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:34 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:34.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:34 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:42:34.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:34 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:42:34.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:34 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:42:34.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:34 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/413742875' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:34.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:34 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:34.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:34 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:34.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:34 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:34.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:34 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:42:34.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:34 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:42:34.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:34 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:42:34.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:34 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/413742875' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:34.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:34 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:34.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:34 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:34.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:34 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:34.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:34 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:42:34.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:34 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:42:34.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:34 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:42:35.566 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:42:35.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:35.773+0000 7f99be800640 1 -- 172.21.15.72:0/1023348827 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99b810a940 msgr2=0x7f99b81111d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:35.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:35.773+0000 7f99be800640 1 --2- 172.21.15.72:0/1023348827 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99b810a940 0x7f99b81111d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f99b0009ca0 tx=0x7f99b002a8b0 comp rx=0 tx=0).stop 2024-04-26T23:42:35.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:35.773+0000 7f99be800640 1 -- 172.21.15.72:0/1023348827 shutdown_connections 2024-04-26T23:42:35.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:35.773+0000 7f99be800640 1 --2- 172.21.15.72:0/1023348827 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99b810a940 0x7f99b81111d0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:35.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:35.773+0000 7f99be800640 1 --2- 172.21.15.72:0/1023348827 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99b8109f80 0x7f99b810a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:35.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:35.773+0000 7f99be800640 1 --2- 172.21.15.72:0/1023348827 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f99b8108d90 0x7f99b81091b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:35.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:35.774+0000 7f99be800640 1 -- 172.21.15.72:0/1023348827 >> 172.21.15.72:0/1023348827 conn(0x7f99b8077d30 msgr2=0x7f99b807a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:35.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:35.774+0000 7f99be800640 1 -- 172.21.15.72:0/1023348827 shutdown_connections 2024-04-26T23:42:35.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:35.774+0000 7f99be800640 1 -- 172.21.15.72:0/1023348827 wait complete. 2024-04-26T23:42:35.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:35.775+0000 7f99be800640 1 Processor -- start 2024-04-26T23:42:35.781 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:35.775+0000 7f99be800640 1 -- start start 2024-04-26T23:42:35.781 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:35.775+0000 7f99be800640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99b8108d90 0x7f99b819eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:35.781 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:35.775+0000 7f99be800640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99b8109f80 0x7f99b819f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:35.781 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:35.775+0000 7f99be800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f99b810a940 0x7f99b81a64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:35.781 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:35.775+0000 7f99be800640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f99b8114910 con 0x7f99b8109f80 2024-04-26T23:42:35.781 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:35.775+0000 7f99be800640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f99b8114790 con 0x7f99b810a940 2024-04-26T23:42:35.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:35.775+0000 7f99be800640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f99b8114a90 con 0x7f99b8108d90 2024-04-26T23:42:35.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:35.775+0000 7f99bca00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99b8109f80 0x7f99b819f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:35.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:35.775+0000 7f99bd400640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99b8108d90 0x7f99b819eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:35.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:35.775+0000 7f99bca00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99b8109f80 0x7f99b819f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60162/0 (socket says 172.21.15.72:60162) 2024-04-26T23:42:35.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:35.775+0000 7f99bca00640 1 -- 172.21.15.72:0/3408466931 learned_addr learned my addr 172.21.15.72:0/3408466931 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:42:35.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:35.775+0000 7f99bde00640 1 --2- 172.21.15.72:0/3408466931 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f99b810a940 0x7f99b81a64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:35.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:35.776+0000 7f99bca00640 1 -- 172.21.15.72:0/3408466931 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99b8108d90 msgr2=0x7f99b819eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:35.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:35.776+0000 7f99bca00640 1 --2- 172.21.15.72:0/3408466931 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99b8108d90 0x7f99b819eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:35.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:35.776+0000 7f99bca00640 1 -- 172.21.15.72:0/3408466931 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f99b810a940 msgr2=0x7f99b81a64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:35.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:35.776+0000 7f99bca00640 1 --2- 172.21.15.72:0/3408466931 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f99b810a940 0x7f99b81a64a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:35.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:35.776+0000 7f99bca00640 1 -- 172.21.15.72:0/3408466931 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99b81a6b10 con 0x7f99b8109f80 2024-04-26T23:42:35.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:35.776+0000 7f99bd400640 1 --2- 172.21.15.72:0/3408466931 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99b8108d90 0x7f99b819eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:42:35.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:35.776+0000 7f99bca00640 1 --2- 172.21.15.72:0/3408466931 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99b8109f80 0x7f99b819f430 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f99a800ea60 tx=0x7f99a800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:35.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:35.776+0000 7f99b6000640 1 -- 172.21.15.72:0/3408466931 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99a800cde0 con 0x7f99b8109f80 2024-04-26T23:42:35.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:35.776+0000 7f99b6000640 1 -- 172.21.15.72:0/3408466931 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99a80035f0 con 0x7f99b8109f80 2024-04-26T23:42:35.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:35.776+0000 7f99be800640 1 -- 172.21.15.72:0/3408466931 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99b81a6e00 con 0x7f99b8109f80 2024-04-26T23:42:35.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:35.776+0000 7f99b6000640 1 -- 172.21.15.72:0/3408466931 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99a8010690 con 0x7f99b8109f80 2024-04-26T23:42:35.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:35.776+0000 7f99be800640 1 -- 172.21.15.72:0/3408466931 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99b81a72c0 con 0x7f99b8109f80 2024-04-26T23:42:35.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:35.777+0000 7f99be800640 1 -- 172.21.15.72:0/3408466931 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99b8110130 con 0x7f99b8109f80 2024-04-26T23:42:35.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:35.780+0000 7f99b6000640 1 -- 172.21.15.72:0/3408466931 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f99a80037a0 con 0x7f99b8109f80 2024-04-26T23:42:35.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:35.781+0000 7f99b6000640 1 --2- 172.21.15.72:0/3408466931 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f99a00a3bb0 0x7f99a00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:35.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:35.781+0000 7f99b6000640 1 -- 172.21.15.72:0/3408466931 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f99a80d06a0 con 0x7f99b8109f80 2024-04-26T23:42:35.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:35.781+0000 7f99bd400640 1 --2- 172.21.15.72:0/3408466931 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f99a00a3bb0 0x7f99a00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:35.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:35.782+0000 7f99bd400640 1 --2- 172.21.15.72:0/3408466931 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f99a00a3bb0 0x7f99a00a6070 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f99b8109de0 tx=0x7f99ac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:35.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:35.784+0000 7f99b6000640 1 -- 172.21.15.72:0/3408466931 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f99a809d480 con 0x7f99b8109f80 2024-04-26T23:42:35.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:35 smithi136 ceph-mon[33050]: Deploying daemon mds.f on smithi136 2024-04-26T23:42:35.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:35 smithi136 ceph-mon[33050]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:35.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:35 smithi136 ceph-mon[33050]: mds.? [v2:172.21.15.76:6834/4251109085,v1:172.21.15.76:6835/4251109085] up:boot 2024-04-26T23:42:35.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:35 smithi136 ceph-mon[33050]: fsmap 5 up:standby 2024-04-26T23:42:35.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:35 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T23:42:35.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:35 smithi076 ceph-mon[32774]: Deploying daemon mds.f on smithi136 2024-04-26T23:42:35.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:35 smithi076 ceph-mon[32774]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:35.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:35 smithi076 ceph-mon[32774]: mds.? [v2:172.21.15.76:6834/4251109085,v1:172.21.15.76:6835/4251109085] up:boot 2024-04-26T23:42:35.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:35 smithi076 ceph-mon[32774]: fsmap 5 up:standby 2024-04-26T23:42:35.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:35 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T23:42:36.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:36.027+0000 7f99be800640 1 -- 172.21.15.72:0/3408466931 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f99b802dad0 con 0x7f99b8109f80 2024-04-26T23:42:36.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:36.028+0000 7f99b6000640 1 -- 172.21.15.72:0/3408466931 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4696 (secure 0 0 0) 0x7f99a809cd80 con 0x7f99b8109f80 2024-04-26T23:42:36.032 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:42:36.033 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/2386456166","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2386456166},{"type":"v1","addr":"172.21.15.72:6835","nonce":2386456166}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1528342053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1528342053},{"type":"v1","addr":"172.21.15.136:6833","nonce":1528342053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6833/2427238559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":2427238559},{"type":"v1","addr":"172.21.15.76:6833","nonce":2427238559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24491,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/4050689894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":4050689894},{"type":"v1","addr":"172.21.15.72:6837","nonce":4050689894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4251109085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4251109085},{"type":"v1","addr":"172.21.15.76:6835","nonce":4251109085}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-26T23:42:36.033 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 6 2024-04-26T23:42:36.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:36.031+0000 7f99be800640 1 -- 172.21.15.72:0/3408466931 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f99a00a3bb0 msgr2=0x7f99a00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:36.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:36.031+0000 7f99be800640 1 --2- 172.21.15.72:0/3408466931 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f99a00a3bb0 0x7f99a00a6070 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f99b8109de0 tx=0x7f99ac005f90 comp rx=0 tx=0).stop 2024-04-26T23:42:36.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:36.031+0000 7f99be800640 1 -- 172.21.15.72:0/3408466931 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99b8109f80 msgr2=0x7f99b819f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:36.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:36.031+0000 7f99be800640 1 --2- 172.21.15.72:0/3408466931 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99b8109f80 0x7f99b819f430 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f99a800ea60 tx=0x7f99a800ef20 comp rx=0 tx=0).stop 2024-04-26T23:42:36.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:36.031+0000 7f99be800640 1 -- 172.21.15.72:0/3408466931 shutdown_connections 2024-04-26T23:42:36.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:36.031+0000 7f99be800640 1 --2- 172.21.15.72:0/3408466931 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f99a00a3bb0 0x7f99a00a6070 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:36.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:36.031+0000 7f99be800640 1 --2- 172.21.15.72:0/3408466931 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f99b810a940 0x7f99b81a64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:36.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:36.031+0000 7f99be800640 1 --2- 172.21.15.72:0/3408466931 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99b8109f80 0x7f99b819f430 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:36.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:36.031+0000 7f99be800640 1 --2- 172.21.15.72:0/3408466931 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99b8108d90 0x7f99b819eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:36.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:36.031+0000 7f99be800640 1 -- 172.21.15.72:0/3408466931 >> 172.21.15.72:0/3408466931 conn(0x7f99b8077d30 msgr2=0x7f99b810d300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:36.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:36.031+0000 7f99be800640 1 -- 172.21.15.72:0/3408466931 shutdown_connections 2024-04-26T23:42:36.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:36.031+0000 7f99be800640 1 -- 172.21.15.72:0/3408466931 wait complete. 2024-04-26T23:42:36.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:35 smithi072 ceph-mon[28184]: Deploying daemon mds.f on smithi136 2024-04-26T23:42:36.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:35 smithi072 ceph-mon[28184]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:36.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:35 smithi072 ceph-mon[28184]: mds.? [v2:172.21.15.76:6834/4251109085,v1:172.21.15.76:6835/4251109085] up:boot 2024-04-26T23:42:36.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:35 smithi072 ceph-mon[28184]: fsmap 5 up:standby 2024-04-26T23:42:36.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:35 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T23:42:36.506 INFO:journalctl@ceph.mds.f.smithi136.stdout:Apr 26 23:42:36 smithi136 systemd[1]: Starting Ceph mds.f for 3c8453ae-0425-11ef-bc93-c7b262605968... 2024-04-26T23:42:36.795 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:36 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/3408466931' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:36.795 INFO:journalctl@ceph.mds.f.smithi136.stdout:Apr 26 23:42:36 smithi136 podman[62515]: 2024-04-26 23:42:36.406499056 +0000 UTC m=+0.017955663 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T23:42:36.795 INFO:journalctl@ceph.mds.f.smithi136.stdout:Apr 26 23:42:36 smithi136 podman[62515]: 2024-04-26 23:42:36.539389921 +0000 UTC m=+0.150846512 container create 760eb78752d2666c87570013d9ff5e4bd8dd55b5f56d710468adb892733ab910 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-26T23:42:36.795 INFO:journalctl@ceph.mds.f.smithi136.stdout:Apr 26 23:42:36 smithi136 podman[62515]: 2024-04-26 23:42:36.792455798 +0000 UTC m=+0.403912389 container init 760eb78752d2666c87570013d9ff5e4bd8dd55b5f56d710468adb892733ab910 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-26T23:42:36.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:36 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/3408466931' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:37.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:36 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/3408466931' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:37.050 INFO:journalctl@ceph.mds.f.smithi136.stdout:Apr 26 23:42:36 smithi136 podman[62515]: 2024-04-26 23:42:36.795365118 +0000 UTC m=+0.406821711 container start 760eb78752d2666c87570013d9ff5e4bd8dd55b5f56d710468adb892733ab910 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-f, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True) 2024-04-26T23:42:37.050 INFO:journalctl@ceph.mds.f.smithi136.stdout:Apr 26 23:42:36 smithi136 ceph-mds[62534]: -- 172.21.15.136:0/2409462879 <== mon.1 v2:172.21.15.136:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b6ff9c5a40 con 0x55b6ff998c00 2024-04-26T23:42:37.050 INFO:journalctl@ceph.mds.f.smithi136.stdout:Apr 26 23:42:36 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-f[62530]: starting mds.f at 2024-04-26T23:42:37.050 INFO:journalctl@ceph.mds.f.smithi136.stdout:Apr 26 23:42:36 smithi136 bash[62515]: 760eb78752d2666c87570013d9ff5e4bd8dd55b5f56d710468adb892733ab910 2024-04-26T23:42:37.050 INFO:journalctl@ceph.mds.f.smithi136.stdout:Apr 26 23:42:36 smithi136 systemd[1]: Started Ceph mds.f for 3c8453ae-0425-11ef-bc93-c7b262605968. 2024-04-26T23:42:37.834 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:37 smithi136 ceph-mon[33050]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:37.834 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:37 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:37.834 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:37 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:37.834 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:37 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:37.834 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:37 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:42:37.834 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:37 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:42:37.834 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:37 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:42:37.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:37 smithi076 ceph-mon[32774]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:37.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:37 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:37.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:37 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:37.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:37 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:37.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:37 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:42:37.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:37 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:42:37.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:37 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:42:38.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:37 smithi072 ceph-mon[28184]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:38.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:37 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:38.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:37 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:38.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:37 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:38.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:37 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:42:38.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:37 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:42:38.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:37 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:42:38.055 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:42:38.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.263+0000 7f8232800640 1 -- 172.21.15.72:0/1188074931 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f822c078ed0 msgr2=0x7f822c079770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:38.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.263+0000 7f8232800640 1 --2- 172.21.15.72:0/1188074931 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f822c078ed0 0x7f822c079770 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8224009ca0 tx=0x7f822402a8c0 comp rx=0 tx=0).stop 2024-04-26T23:42:38.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.264+0000 7f8232800640 1 -- 172.21.15.72:0/1188074931 shutdown_connections 2024-04-26T23:42:38.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.264+0000 7f8232800640 1 --2- 172.21.15.72:0/1188074931 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f822c078ed0 0x7f822c079770 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:38.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.264+0000 7f8232800640 1 --2- 172.21.15.72:0/1188074931 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f822c078530 0x7f822c078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:38.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.264+0000 7f8232800640 1 --2- 172.21.15.72:0/1188074931 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f822c0772f0 0x7f822c077710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:38.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.264+0000 7f8232800640 1 -- 172.21.15.72:0/1188074931 >> 172.21.15.72:0/1188074931 conn(0x7f822c100120 msgr2=0x7f822c102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:38.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.264+0000 7f8232800640 1 -- 172.21.15.72:0/1188074931 shutdown_connections 2024-04-26T23:42:38.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.264+0000 7f8232800640 1 -- 172.21.15.72:0/1188074931 wait complete. 2024-04-26T23:42:38.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.265+0000 7f8232800640 1 Processor -- start 2024-04-26T23:42:38.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.265+0000 7f8232800640 1 -- start start 2024-04-26T23:42:38.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.265+0000 7f8232800640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f822c0772f0 0x7f822c19f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:38.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.265+0000 7f8232800640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f822c078530 0x7f822c19f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:38.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.265+0000 7f8232800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f822c078ed0 0x7f822c1a66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:38.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.265+0000 7f8232800640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f822c114ad0 con 0x7f822c078530 2024-04-26T23:42:38.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.265+0000 7f8232800640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f822c114950 con 0x7f822c078ed0 2024-04-26T23:42:38.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.265+0000 7f8232800640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f822c114c50 con 0x7f822c0772f0 2024-04-26T23:42:38.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.265+0000 7f822b400640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f822c0772f0 0x7f822c19f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:38.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.265+0000 7f822aa00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f822c078530 0x7f822c19f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:38.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.265+0000 7f822b400640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f822c0772f0 0x7f822c19f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.72:34748/0 (socket says 172.21.15.72:34748) 2024-04-26T23:42:38.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.265+0000 7f822b400640 1 -- 172.21.15.72:0/969851644 learned_addr learned my addr 172.21.15.72:0/969851644 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:42:38.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.266+0000 7f822be00640 1 --2- 172.21.15.72:0/969851644 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f822c078ed0 0x7f822c1a66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:38.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.266+0000 7f822b400640 1 -- 172.21.15.72:0/969851644 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f822c078ed0 msgr2=0x7f822c1a66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:38.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.266+0000 7f822b400640 1 --2- 172.21.15.72:0/969851644 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f822c078ed0 0x7f822c1a66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:38.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.266+0000 7f822b400640 1 -- 172.21.15.72:0/969851644 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f822c078530 msgr2=0x7f822c19f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:38.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.266+0000 7f822b400640 1 --2- 172.21.15.72:0/969851644 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f822c078530 0x7f822c19f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:38.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.266+0000 7f822b400640 1 -- 172.21.15.72:0/969851644 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f822c1a6d20 con 0x7f822c0772f0 2024-04-26T23:42:38.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.266+0000 7f822b400640 1 --2- 172.21.15.72:0/969851644 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f822c0772f0 0x7f822c19f100 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f821c00d950 tx=0x7f821c00de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:38.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.266+0000 7f8223e00640 1 -- 172.21.15.72:0/969851644 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f821c017030 con 0x7f822c0772f0 2024-04-26T23:42:38.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.266+0000 7f8223e00640 1 -- 172.21.15.72:0/969851644 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f821c004020 con 0x7f822c0772f0 2024-04-26T23:42:38.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.266+0000 7f8223e00640 1 -- 172.21.15.72:0/969851644 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f821c0047a0 con 0x7f822c0772f0 2024-04-26T23:42:38.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.266+0000 7f8232800640 1 -- 172.21.15.72:0/969851644 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f822c1a6fb0 con 0x7f822c0772f0 2024-04-26T23:42:38.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.267+0000 7f8232800640 1 -- 172.21.15.72:0/969851644 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f822c1a74c0 con 0x7f822c0772f0 2024-04-26T23:42:38.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.268+0000 7f8232800640 1 -- 172.21.15.72:0/969851644 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f822c111690 con 0x7f822c0772f0 2024-04-26T23:42:38.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.269+0000 7f8223e00640 1 -- 172.21.15.72:0/969851644 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f821c009a30 con 0x7f822c0772f0 2024-04-26T23:42:38.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.271+0000 7f8223e00640 1 --2- 172.21.15.72:0/969851644 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f82100a3c30 0x7f82100a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:38.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.271+0000 7f8223e00640 1 -- 172.21.15.72:0/969851644 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f821c0cf990 con 0x7f822c0772f0 2024-04-26T23:42:38.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.271+0000 7f822aa00640 1 --2- 172.21.15.72:0/969851644 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f82100a3c30 0x7f82100a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:38.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.272+0000 7f822aa00640 1 --2- 172.21.15.72:0/969851644 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f82100a3c30 0x7f82100a60f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f822c1a0610 tx=0x7f8218005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:38.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.275+0000 7f8223e00640 1 -- 172.21.15.72:0/969851644 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f821c09c770 con 0x7f822c0772f0 2024-04-26T23:42:38.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.518+0000 7f8232800640 1 -- 172.21.15.72:0/969851644 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f822c02dad0 con 0x7f822c0772f0 2024-04-26T23:42:38.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.519+0000 7f8223e00640 1 -- 172.21.15.72:0/969851644 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5524 (secure 0 0 0) 0x7f821c09c070 con 0x7f822c0772f0 2024-04-26T23:42:38.523 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:42:38.523 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/2386456166","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2386456166},{"type":"v1","addr":"172.21.15.72:6835","nonce":2386456166}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1528342053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1528342053},{"type":"v1","addr":"172.21.15.136:6833","nonce":1528342053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6833/2427238559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":2427238559},{"type":"v1","addr":"172.21.15.76:6833","nonce":2427238559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24491,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/4050689894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":4050689894},{"type":"v1","addr":"172.21.15.72:6837","nonce":4050689894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3681925367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3681925367},{"type":"v1","addr":"172.21.15.136:6835","nonce":3681925367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4251109085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4251109085},{"type":"v1","addr":"172.21.15.76:6835","nonce":4251109085}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-26T23:42:38.523 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 7 2024-04-26T23:42:38.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.521+0000 7f8232800640 1 -- 172.21.15.72:0/969851644 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f82100a3c30 msgr2=0x7f82100a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:38.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.521+0000 7f8232800640 1 --2- 172.21.15.72:0/969851644 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f82100a3c30 0x7f82100a60f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f822c1a0610 tx=0x7f8218005f90 comp rx=0 tx=0).stop 2024-04-26T23:42:38.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.521+0000 7f8232800640 1 -- 172.21.15.72:0/969851644 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f822c0772f0 msgr2=0x7f822c19f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:38.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.521+0000 7f8232800640 1 --2- 172.21.15.72:0/969851644 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f822c0772f0 0x7f822c19f100 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f821c00d950 tx=0x7f821c00de10 comp rx=0 tx=0).stop 2024-04-26T23:42:38.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.521+0000 7f8232800640 1 -- 172.21.15.72:0/969851644 shutdown_connections 2024-04-26T23:42:38.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.521+0000 7f8232800640 1 --2- 172.21.15.72:0/969851644 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f82100a3c30 0x7f82100a60f0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:38.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.521+0000 7f8232800640 1 --2- 172.21.15.72:0/969851644 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f822c078ed0 0x7f822c1a66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:38.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.521+0000 7f8232800640 1 --2- 172.21.15.72:0/969851644 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f822c078530 0x7f822c19f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:38.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.521+0000 7f8232800640 1 --2- 172.21.15.72:0/969851644 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f822c0772f0 0x7f822c19f100 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:38.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.521+0000 7f8232800640 1 -- 172.21.15.72:0/969851644 >> 172.21.15.72:0/969851644 conn(0x7f822c100120 msgr2=0x7f822c101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:38.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.521+0000 7f8232800640 1 -- 172.21.15.72:0/969851644 shutdown_connections 2024-04-26T23:42:38.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:38.521+0000 7f8232800640 1 -- 172.21.15.72:0/969851644 wait complete. 2024-04-26T23:42:38.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:38 smithi072 ceph-mon[28184]: Deploying daemon mds.c on smithi136 2024-04-26T23:42:38.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:38 smithi072 ceph-mon[28184]: mds.? [v2:172.21.15.136:6834/3681925367,v1:172.21.15.136:6835/3681925367] up:boot 2024-04-26T23:42:38.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:38 smithi072 ceph-mon[28184]: fsmap 6 up:standby 2024-04-26T23:42:38.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:38 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T23:42:38.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:38 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/969851644' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:38.838 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:38 smithi136 ceph-mon[33050]: Deploying daemon mds.c on smithi136 2024-04-26T23:42:38.838 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:38 smithi136 ceph-mon[33050]: mds.? [v2:172.21.15.136:6834/3681925367,v1:172.21.15.136:6835/3681925367] up:boot 2024-04-26T23:42:38.838 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:38 smithi136 ceph-mon[33050]: fsmap 6 up:standby 2024-04-26T23:42:38.838 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:38 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T23:42:38.838 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:38 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/969851644' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:38.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:38 smithi076 ceph-mon[32774]: Deploying daemon mds.c on smithi136 2024-04-26T23:42:38.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:38 smithi076 ceph-mon[32774]: mds.? [v2:172.21.15.136:6834/3681925367,v1:172.21.15.136:6835/3681925367] up:boot 2024-04-26T23:42:38.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:38 smithi076 ceph-mon[32774]: fsmap 6 up:standby 2024-04-26T23:42:38.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:38 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T23:42:38.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:38 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/969851644' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:39.551 INFO:journalctl@ceph.mds.c.smithi136.stdout:Apr 26 23:42:39 smithi136 podman[62823]: 2024-04-26 23:42:39.487945561 +0000 UTC m=+0.121350774 container create 2a2b7325f4c05298470f7cc9e57e78c01d51fb207e59d1f1a640b82c614bc87b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-c, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-26T23:42:39.845 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:39 smithi136 ceph-mon[33050]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:39.845 INFO:journalctl@ceph.mds.c.smithi136.stdout:Apr 26 23:42:39 smithi136 podman[62823]: 2024-04-26 23:42:39.816077158 +0000 UTC m=+0.449482372 container init 2a2b7325f4c05298470f7cc9e57e78c01d51fb207e59d1f1a640b82c614bc87b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-c, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T23:42:39.845 INFO:journalctl@ceph.mds.c.smithi136.stdout:Apr 26 23:42:39 smithi136 podman[62823]: 2024-04-26 23:42:39.818780913 +0000 UTC m=+0.452186130 container start 2a2b7325f4c05298470f7cc9e57e78c01d51fb207e59d1f1a640b82c614bc87b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2) 2024-04-26T23:42:39.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:39 smithi076 ceph-mon[32774]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:40.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:39 smithi072 ceph-mon[28184]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:40.300 INFO:journalctl@ceph.mds.c.smithi136.stdout:Apr 26 23:42:39 smithi136 ceph-mds[62842]: -- 172.21.15.136:0/2219502568 <== mon.1 v2:172.21.15.136:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5623f9a2ba40 con 0x5623f99fe800 2024-04-26T23:42:40.300 INFO:journalctl@ceph.mds.c.smithi136.stdout:Apr 26 23:42:39 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-c[62838]: starting mds.c at 2024-04-26T23:42:40.300 INFO:journalctl@ceph.mds.c.smithi136.stdout:Apr 26 23:42:39 smithi136 bash[62823]: 2a2b7325f4c05298470f7cc9e57e78c01d51fb207e59d1f1a640b82c614bc87b 2024-04-26T23:42:40.300 INFO:journalctl@ceph.mds.c.smithi136.stdout:Apr 26 23:42:39 smithi136 systemd[1]: Started Ceph mds.c for 3c8453ae-0425-11ef-bc93-c7b262605968. 2024-04-26T23:42:40.547 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:42:40.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:40.753+0000 7f2f9d400640 1 -- 172.21.15.72:0/3163759812 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2f9810ab40 msgr2=0x7f2f981113d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:40.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:40.753+0000 7f2f9d400640 1 --2- 172.21.15.72:0/3163759812 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2f9810ab40 0x7f2f981113d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2f90009be0 tx=0x7f2f9002a6c0 comp rx=0 tx=0).stop 2024-04-26T23:42:40.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:40.753+0000 7f2f9d400640 1 -- 172.21.15.72:0/3163759812 shutdown_connections 2024-04-26T23:42:40.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:40.753+0000 7f2f9d400640 1 --2- 172.21.15.72:0/3163759812 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2f9810ab40 0x7f2f981113d0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:40.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:40.753+0000 7f2f9d400640 1 --2- 172.21.15.72:0/3163759812 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2f9810a180 0x7f2f9810a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:40.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:40.753+0000 7f2f9d400640 1 --2- 172.21.15.72:0/3163759812 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f98108f90 0x7f2f981093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:40.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:40.753+0000 7f2f9d400640 1 -- 172.21.15.72:0/3163759812 >> 172.21.15.72:0/3163759812 conn(0x7f2f98077f40 msgr2=0x7f2f9807a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:40.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:40.754+0000 7f2f9d400640 1 -- 172.21.15.72:0/3163759812 shutdown_connections 2024-04-26T23:42:40.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:40.754+0000 7f2f9d400640 1 -- 172.21.15.72:0/3163759812 wait complete. 2024-04-26T23:42:40.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:40.754+0000 7f2f9d400640 1 Processor -- start 2024-04-26T23:42:40.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:40.755+0000 7f2f9d400640 1 -- start start 2024-04-26T23:42:40.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:40.755+0000 7f2f9d400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2f98108f90 0x7f2f9819f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:40.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:40.755+0000 7f2f9d400640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f9810a180 0x7f2f9819f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:40.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:40.755+0000 7f2f9d400640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2f9810ab40 0x7f2f981a6700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:40.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:40.755+0000 7f2f9d400640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f2f98114ad0 con 0x7f2f9810a180 2024-04-26T23:42:40.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:40.755+0000 7f2f9d400640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2f98114950 con 0x7f2f98108f90 2024-04-26T23:42:40.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:40.755+0000 7f2f9d400640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f2f98114c50 con 0x7f2f9810ab40 2024-04-26T23:42:40.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:40.755+0000 7f2f96800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2f98108f90 0x7f2f9819f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:40.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:40.755+0000 7f2f96800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2f98108f90 0x7f2f9819f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:60190/0 (socket says 172.21.15.72:60190) 2024-04-26T23:42:40.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:40.755+0000 7f2f96800640 1 -- 172.21.15.72:0/2252957688 learned_addr learned my addr 172.21.15.72:0/2252957688 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:42:40.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:40.755+0000 7f2f8fe00640 1 --2- 172.21.15.72:0/2252957688 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f9810a180 0x7f2f9819f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:40.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:40.755+0000 7f2f97200640 1 --2- 172.21.15.72:0/2252957688 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2f9810ab40 0x7f2f981a6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:40.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:40.756+0000 7f2f96800640 1 -- 172.21.15.72:0/2252957688 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2f9810ab40 msgr2=0x7f2f981a6700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:40.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:40.756+0000 7f2f96800640 1 --2- 172.21.15.72:0/2252957688 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2f9810ab40 0x7f2f981a6700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:40.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:40.756+0000 7f2f96800640 1 -- 172.21.15.72:0/2252957688 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f9810a180 msgr2=0x7f2f9819f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:40.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:40.756+0000 7f2f96800640 1 --2- 172.21.15.72:0/2252957688 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f9810a180 0x7f2f9819f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:40.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:40.756+0000 7f2f96800640 1 -- 172.21.15.72:0/2252957688 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f981a6d70 con 0x7f2f98108f90 2024-04-26T23:42:40.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:40.756+0000 7f2f8fe00640 1 --2- 172.21.15.72:0/2252957688 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f9810a180 0x7f2f9819f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:42:40.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:40.756+0000 7f2f96800640 1 --2- 172.21.15.72:0/2252957688 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2f98108f90 0x7f2f9819f150 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2f8800e9f0 tx=0x7f2f8800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:40.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:40.756+0000 7f2f8f400640 1 -- 172.21.15.72:0/2252957688 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f8800ccc0 con 0x7f2f98108f90 2024-04-26T23:42:40.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:40.756+0000 7f2f8f400640 1 -- 172.21.15.72:0/2252957688 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f880035f0 con 0x7f2f98108f90 2024-04-26T23:42:40.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:40.756+0000 7f2f8f400640 1 -- 172.21.15.72:0/2252957688 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f8800ce60 con 0x7f2f98108f90 2024-04-26T23:42:40.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:40.756+0000 7f2f9d400640 1 -- 172.21.15.72:0/2252957688 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f981a7060 con 0x7f2f98108f90 2024-04-26T23:42:40.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:40.756+0000 7f2f9d400640 1 -- 172.21.15.72:0/2252957688 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f981a7600 con 0x7f2f98108f90 2024-04-26T23:42:40.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:40.757+0000 7f2f9d400640 1 -- 172.21.15.72:0/2252957688 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f981102b0 con 0x7f2f98108f90 2024-04-26T23:42:40.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:40.759+0000 7f2f8f400640 1 -- 172.21.15.72:0/2252957688 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2f88004000 con 0x7f2f98108f90 2024-04-26T23:42:40.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:40.760+0000 7f2f8f400640 1 --2- 172.21.15.72:0/2252957688 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f2f7c0a3c30 0x7f2f7c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:40.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:40.761+0000 7f2f8f400640 1 -- 172.21.15.72:0/2252957688 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f2f880d02e0 con 0x7f2f98108f90 2024-04-26T23:42:40.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:40.761+0000 7f2f8fe00640 1 --2- 172.21.15.72:0/2252957688 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f2f7c0a3c30 0x7f2f7c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:40.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:40.761+0000 7f2f8fe00640 1 --2- 172.21.15.72:0/2252957688 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f2f7c0a3c30 0x7f2f7c0a60f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2f981a0660 tx=0x7f2f80005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:40.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:40.765+0000 7f2f8f400640 1 -- 172.21.15.72:0/2252957688 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f2f8809d0c0 con 0x7f2f98108f90 2024-04-26T23:42:41.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:41.005+0000 7f2f9d400640 1 -- 172.21.15.72:0/2252957688 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2f9802dad0 con 0x7f2f98108f90 2024-04-26T23:42:41.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:41.006+0000 7f2f8f400640 1 -- 172.21.15.72:0/2252957688 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6352 (secure 0 0 0) 0x7f2f8809c9c0 con 0x7f2f98108f90 2024-04-26T23:42:41.010 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:42:41.010 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/2386456166","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2386456166},{"type":"v1","addr":"172.21.15.72:6835","nonce":2386456166}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1528342053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1528342053},{"type":"v1","addr":"172.21.15.136:6833","nonce":1528342053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6833/2427238559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":2427238559},{"type":"v1","addr":"172.21.15.76:6833","nonce":2427238559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24491,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/4050689894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":4050689894},{"type":"v1","addr":"172.21.15.72:6837","nonce":4050689894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3681925367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3681925367},{"type":"v1","addr":"172.21.15.136:6835","nonce":3681925367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4251109085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4251109085},{"type":"v1","addr":"172.21.15.76:6835","nonce":4251109085}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24514,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/3639308555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3639308555},{"type":"v1","addr":"172.21.15.136:6837","nonce":3639308555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-26T23:42:41.011 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 8 2024-04-26T23:42:41.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:41.009+0000 7f2f9d400640 1 -- 172.21.15.72:0/2252957688 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f2f7c0a3c30 msgr2=0x7f2f7c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:41.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:41.009+0000 7f2f9d400640 1 --2- 172.21.15.72:0/2252957688 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f2f7c0a3c30 0x7f2f7c0a60f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2f981a0660 tx=0x7f2f80005f90 comp rx=0 tx=0).stop 2024-04-26T23:42:41.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:41.009+0000 7f2f9d400640 1 -- 172.21.15.72:0/2252957688 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2f98108f90 msgr2=0x7f2f9819f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:41.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:41.009+0000 7f2f9d400640 1 --2- 172.21.15.72:0/2252957688 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2f98108f90 0x7f2f9819f150 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2f8800e9f0 tx=0x7f2f8800eeb0 comp rx=0 tx=0).stop 2024-04-26T23:42:41.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:41.009+0000 7f2f9d400640 1 -- 172.21.15.72:0/2252957688 shutdown_connections 2024-04-26T23:42:41.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:41.009+0000 7f2f9d400640 1 --2- 172.21.15.72:0/2252957688 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f2f7c0a3c30 0x7f2f7c0a60f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:41.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:41.009+0000 7f2f9d400640 1 --2- 172.21.15.72:0/2252957688 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2f9810ab40 0x7f2f981a6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:41.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:41.009+0000 7f2f9d400640 1 --2- 172.21.15.72:0/2252957688 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f9810a180 0x7f2f9819f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:41.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:41.009+0000 7f2f9d400640 1 --2- 172.21.15.72:0/2252957688 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2f98108f90 0x7f2f9819f150 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:41.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:41.009+0000 7f2f9d400640 1 -- 172.21.15.72:0/2252957688 >> 172.21.15.72:0/2252957688 conn(0x7f2f98077f40 msgr2=0x7f2f9810d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:41.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:41.010+0000 7f2f9d400640 1 -- 172.21.15.72:0/2252957688 shutdown_connections 2024-04-26T23:42:41.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:41.010+0000 7f2f9d400640 1 -- 172.21.15.72:0/2252957688 wait complete. 2024-04-26T23:42:41.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:40 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:41.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:40 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:41.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:40 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:41.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:40 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:42:41.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:40 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:42:41.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:40 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:42:41.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:40 smithi072 ceph-mon[28184]: Deploying daemon mds.h on smithi076 2024-04-26T23:42:41.295 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:40 smithi072 ceph-mon[28184]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:41.295 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:40 smithi072 ceph-mon[28184]: mds.? [v2:172.21.15.136:6836/3639308555,v1:172.21.15.136:6837/3639308555] up:boot 2024-04-26T23:42:41.295 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:40 smithi072 ceph-mon[28184]: fsmap 7 up:standby 2024-04-26T23:42:41.295 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:40 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T23:42:41.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:40 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:41.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:40 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:41.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:40 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:41.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:40 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:42:41.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:40 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:42:41.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:40 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:42:41.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:40 smithi136 ceph-mon[33050]: Deploying daemon mds.h on smithi076 2024-04-26T23:42:41.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:40 smithi136 ceph-mon[33050]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:41.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:40 smithi136 ceph-mon[33050]: mds.? [v2:172.21.15.136:6836/3639308555,v1:172.21.15.136:6837/3639308555] up:boot 2024-04-26T23:42:41.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:40 smithi136 ceph-mon[33050]: fsmap 7 up:standby 2024-04-26T23:42:41.301 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:40 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T23:42:41.370 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:40 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:41.370 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:40 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:41.370 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:40 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:41.371 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:40 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:42:41.371 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:40 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:42:41.371 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:40 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:42:41.371 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:40 smithi076 ceph-mon[32774]: Deploying daemon mds.h on smithi076 2024-04-26T23:42:41.371 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:40 smithi076 ceph-mon[32774]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:41.371 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:40 smithi076 ceph-mon[32774]: mds.? [v2:172.21.15.136:6836/3639308555,v1:172.21.15.136:6837/3639308555] up:boot 2024-04-26T23:42:41.371 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:40 smithi076 ceph-mon[32774]: fsmap 7 up:standby 2024-04-26T23:42:41.371 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:40 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T23:42:42.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:41 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/2252957688' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:42.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:41 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/2252957688' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:42.338 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:41 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/2252957688' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:42.339 INFO:journalctl@ceph.mds.h.smithi076.stdout:Apr 26 23:42:42 smithi076 systemd[1]: Starting Ceph mds.h for 3c8453ae-0425-11ef-bc93-c7b262605968... 2024-04-26T23:42:42.611 INFO:journalctl@ceph.mds.h.smithi076.stdout:Apr 26 23:42:42 smithi076 podman[61915]: 2024-04-26 23:42:42.238126295 +0000 UTC m=+0.017920176 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T23:42:42.611 INFO:journalctl@ceph.mds.h.smithi076.stdout:Apr 26 23:42:42 smithi076 podman[61915]: 2024-04-26 23:42:42.363630304 +0000 UTC m=+0.143424155 container create 9efa73adf79457aa174c0d2436e7f7b83a7fabfe7a2cb643c3a3a3d9fd84c224 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-h, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True) 2024-04-26T23:42:42.611 INFO:journalctl@ceph.mds.h.smithi076.stdout:Apr 26 23:42:42 smithi076 podman[61915]: 2024-04-26 23:42:42.608534746 +0000 UTC m=+0.388328599 container init 9efa73adf79457aa174c0d2436e7f7b83a7fabfe7a2cb643c3a3a3d9fd84c224 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-h, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-26T23:42:42.929 INFO:journalctl@ceph.mds.h.smithi076.stdout:Apr 26 23:42:42 smithi076 podman[61915]: 2024-04-26 23:42:42.611206505 +0000 UTC m=+0.391000360 container start 9efa73adf79457aa174c0d2436e7f7b83a7fabfe7a2cb643c3a3a3d9fd84c224 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-h, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T23:42:42.929 INFO:journalctl@ceph.mds.h.smithi076.stdout:Apr 26 23:42:42 smithi076 ceph-mds[61934]: -- 172.21.15.76:0/2910616455 <== mon.2 v2:172.21.15.76:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558d231dba40 con 0x558d231ae800 2024-04-26T23:42:42.929 INFO:journalctl@ceph.mds.h.smithi076.stdout:Apr 26 23:42:42 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-h[61930]: starting mds.h at 2024-04-26T23:42:42.929 INFO:journalctl@ceph.mds.h.smithi076.stdout:Apr 26 23:42:42 smithi076 bash[61915]: 9efa73adf79457aa174c0d2436e7f7b83a7fabfe7a2cb643c3a3a3d9fd84c224 2024-04-26T23:42:42.929 INFO:journalctl@ceph.mds.h.smithi076.stdout:Apr 26 23:42:42 smithi076 systemd[1]: Started Ceph mds.h for 3c8453ae-0425-11ef-bc93-c7b262605968. 2024-04-26T23:42:43.033 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:42:43.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.240+0000 7f9f6b800640 1 -- 172.21.15.72:0/706052996 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f64108f90 msgr2=0x7f9f641093b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:43.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.240+0000 7f9f6b800640 1 --2- 172.21.15.72:0/706052996 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f64108f90 0x7f9f641093b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9f58009900 tx=0x7f9f5802a410 comp rx=0 tx=0).stop 2024-04-26T23:42:43.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.240+0000 7f9f6b800640 1 -- 172.21.15.72:0/706052996 shutdown_connections 2024-04-26T23:42:43.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.240+0000 7f9f6b800640 1 --2- 172.21.15.72:0/706052996 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9f6410a180 0x7f9f6410a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:43.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.240+0000 7f9f6b800640 1 --2- 172.21.15.72:0/706052996 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f6410ab40 0x7f9f641113d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:43.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.240+0000 7f9f6b800640 1 --2- 172.21.15.72:0/706052996 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f64108f90 0x7f9f641093b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:43.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.240+0000 7f9f6b800640 1 -- 172.21.15.72:0/706052996 >> 172.21.15.72:0/706052996 conn(0x7f9f64077f40 msgr2=0x7f9f6407a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:43.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.240+0000 7f9f6b800640 1 -- 172.21.15.72:0/706052996 shutdown_connections 2024-04-26T23:42:43.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.240+0000 7f9f6b800640 1 -- 172.21.15.72:0/706052996 wait complete. 2024-04-26T23:42:43.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.241+0000 7f9f6b800640 1 Processor -- start 2024-04-26T23:42:43.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.241+0000 7f9f6b800640 1 -- start start 2024-04-26T23:42:43.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.241+0000 7f9f6b800640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f64108f90 0x7f9f6419f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:43.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.241+0000 7f9f6b800640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f6410a180 0x7f9f6419f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:43.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.242+0000 7f9f6b800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9f6410ab40 0x7f9f641a6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:43.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.242+0000 7f9f6b800640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f9f64114ae0 con 0x7f9f6410a180 2024-04-26T23:42:43.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.242+0000 7f9f6b800640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f9f64114960 con 0x7f9f6410ab40 2024-04-26T23:42:43.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.242+0000 7f9f6b800640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f9f64114c60 con 0x7f9f64108f90 2024-04-26T23:42:43.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.242+0000 7f9f68e00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f64108f90 0x7f9f6419f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:43.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.242+0000 7f9f63e00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f6410a180 0x7f9f6419f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:43.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.242+0000 7f9f68e00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f64108f90 0x7f9f6419f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.72:34790/0 (socket says 172.21.15.72:34790) 2024-04-26T23:42:43.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.242+0000 7f9f68e00640 1 -- 172.21.15.72:0/2090135709 learned_addr learned my addr 172.21.15.72:0/2090135709 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:42:43.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.242+0000 7f9f69800640 1 --2- 172.21.15.72:0/2090135709 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9f6410ab40 0x7f9f641a6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:43.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.242+0000 7f9f68e00640 1 -- 172.21.15.72:0/2090135709 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9f6410ab40 msgr2=0x7f9f641a6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:43.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.242+0000 7f9f68e00640 1 --2- 172.21.15.72:0/2090135709 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9f6410ab40 0x7f9f641a6730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:43.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.242+0000 7f9f68e00640 1 -- 172.21.15.72:0/2090135709 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f6410a180 msgr2=0x7f9f6419f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:43.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.242+0000 7f9f68e00640 1 --2- 172.21.15.72:0/2090135709 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f6410a180 0x7f9f6419f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:43.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.242+0000 7f9f68e00640 1 -- 172.21.15.72:0/2090135709 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f641a6da0 con 0x7f9f64108f90 2024-04-26T23:42:43.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.243+0000 7f9f68e00640 1 --2- 172.21.15.72:0/2090135709 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f64108f90 0x7f9f6419f180 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9f58031500 tx=0x7f9f58003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:43.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.243+0000 7f9f61600640 1 -- 172.21.15.72:0/2090135709 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f5803f070 con 0x7f9f64108f90 2024-04-26T23:42:43.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.243+0000 7f9f61600640 1 -- 172.21.15.72:0/2090135709 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9f58031c50 con 0x7f9f64108f90 2024-04-26T23:42:43.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.243+0000 7f9f61600640 1 -- 172.21.15.72:0/2090135709 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f58035030 con 0x7f9f64108f90 2024-04-26T23:42:43.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.243+0000 7f9f6b800640 1 -- 172.21.15.72:0/2090135709 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f641a7030 con 0x7f9f64108f90 2024-04-26T23:42:43.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.243+0000 7f9f6b800640 1 -- 172.21.15.72:0/2090135709 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f641a74d0 con 0x7f9f64108f90 2024-04-26T23:42:43.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.244+0000 7f9f6b800640 1 -- 172.21.15.72:0/2090135709 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f641102b0 con 0x7f9f64108f90 2024-04-26T23:42:43.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.246+0000 7f9f61600640 1 -- 172.21.15.72:0/2090135709 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9f58042070 con 0x7f9f64108f90 2024-04-26T23:42:43.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.247+0000 7f9f61600640 1 --2- 172.21.15.72:0/2090135709 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f9f4c0a3c30 0x7f9f4c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:43.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.247+0000 7f9f61600640 1 -- 172.21.15.72:0/2090135709 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f9f580f0640 con 0x7f9f64108f90 2024-04-26T23:42:43.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.247+0000 7f9f63e00640 1 --2- 172.21.15.72:0/2090135709 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f9f4c0a3c30 0x7f9f4c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:43.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.248+0000 7f9f63e00640 1 --2- 172.21.15.72:0/2090135709 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f9f4c0a3c30 0x7f9f4c0a60f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9f641a0690 tx=0x7f9f50005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:43.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.251+0000 7f9f61600640 1 -- 172.21.15.72:0/2090135709 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9f580bcdd0 con 0x7f9f64108f90 2024-04-26T23:42:43.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:42 smithi072 ceph-mon[28184]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:43.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:42 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:43.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:42 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:43.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:42 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:43.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:42 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:42:43.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:42 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:42:43.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:42 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:42:43.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:42 smithi136 ceph-mon[33050]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:43.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:42 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:43.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:42 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:43.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:42 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:43.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:42 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:42:43.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:42 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:42:43.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:42 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:42:43.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:42 smithi076 ceph-mon[32774]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:43.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:42 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:43.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:42 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:43.430 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:42 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:43.430 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:42 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:42:43.430 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:42 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:42:43.430 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:42 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:42:43.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.491+0000 7f9f6b800640 1 -- 172.21.15.72:0/2090135709 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9f6402dad0 con 0x7f9f64108f90 2024-04-26T23:42:43.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.492+0000 7f9f61600640 1 -- 172.21.15.72:0/2090135709 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7177 (secure 0 0 0) 0x7f9f580c10d0 con 0x7f9f64108f90 2024-04-26T23:42:43.496 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:42:43.496 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/2386456166","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2386456166},{"type":"v1","addr":"172.21.15.72:6835","nonce":2386456166}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1528342053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1528342053},{"type":"v1","addr":"172.21.15.136:6833","nonce":1528342053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6833/2427238559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":2427238559},{"type":"v1","addr":"172.21.15.76:6833","nonce":2427238559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24491,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/4050689894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":4050689894},{"type":"v1","addr":"172.21.15.72:6837","nonce":4050689894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3681925367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3681925367},{"type":"v1","addr":"172.21.15.136:6835","nonce":3681925367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4251109085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4251109085},{"type":"v1","addr":"172.21.15.76:6835","nonce":4251109085}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24514,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/3639308555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3639308555},{"type":"v1","addr":"172.21.15.136:6837","nonce":3639308555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/2506930945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2506930945},{"type":"v1","addr":"172.21.15.76:6837","nonce":2506930945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-26T23:42:43.497 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 9 2024-04-26T23:42:43.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.494+0000 7f9f6b800640 1 -- 172.21.15.72:0/2090135709 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f9f4c0a3c30 msgr2=0x7f9f4c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:43.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.494+0000 7f9f6b800640 1 --2- 172.21.15.72:0/2090135709 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f9f4c0a3c30 0x7f9f4c0a60f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9f641a0690 tx=0x7f9f50005f90 comp rx=0 tx=0).stop 2024-04-26T23:42:43.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.494+0000 7f9f6b800640 1 -- 172.21.15.72:0/2090135709 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f64108f90 msgr2=0x7f9f6419f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:43.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.494+0000 7f9f6b800640 1 --2- 172.21.15.72:0/2090135709 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f64108f90 0x7f9f6419f180 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9f58031500 tx=0x7f9f58003c10 comp rx=0 tx=0).stop 2024-04-26T23:42:43.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.494+0000 7f9f6b800640 1 -- 172.21.15.72:0/2090135709 shutdown_connections 2024-04-26T23:42:43.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.494+0000 7f9f6b800640 1 --2- 172.21.15.72:0/2090135709 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f9f4c0a3c30 0x7f9f4c0a60f0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:43.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.494+0000 7f9f6b800640 1 --2- 172.21.15.72:0/2090135709 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9f6410ab40 0x7f9f641a6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:43.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.494+0000 7f9f6b800640 1 --2- 172.21.15.72:0/2090135709 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f6410a180 0x7f9f6419f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:43.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.494+0000 7f9f6b800640 1 --2- 172.21.15.72:0/2090135709 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f64108f90 0x7f9f6419f180 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:43.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.494+0000 7f9f6b800640 1 -- 172.21.15.72:0/2090135709 >> 172.21.15.72:0/2090135709 conn(0x7f9f64077f40 msgr2=0x7f9f6410d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:43.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.495+0000 7f9f6b800640 1 -- 172.21.15.72:0/2090135709 shutdown_connections 2024-04-26T23:42:43.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:43.495+0000 7f9f6b800640 1 -- 172.21.15.72:0/2090135709 wait complete. 2024-04-26T23:42:44.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:43 smithi072 ceph-mon[28184]: Deploying daemon mds.i on smithi136 2024-04-26T23:42:44.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:43 smithi072 ceph-mon[28184]: mds.? [v2:172.21.15.76:6836/2506930945,v1:172.21.15.76:6837/2506930945] up:boot 2024-04-26T23:42:44.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:43 smithi072 ceph-mon[28184]: fsmap 8 up:standby 2024-04-26T23:42:44.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:43 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T23:42:44.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:43 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/2090135709' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:44.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:43 smithi136 ceph-mon[33050]: Deploying daemon mds.i on smithi136 2024-04-26T23:42:44.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:43 smithi136 ceph-mon[33050]: mds.? [v2:172.21.15.76:6836/2506930945,v1:172.21.15.76:6837/2506930945] up:boot 2024-04-26T23:42:44.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:43 smithi136 ceph-mon[33050]: fsmap 8 up:standby 2024-04-26T23:42:44.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:43 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T23:42:44.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:43 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/2090135709' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:44.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:43 smithi076 ceph-mon[32774]: Deploying daemon mds.i on smithi136 2024-04-26T23:42:44.430 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:43 smithi076 ceph-mon[32774]: mds.? [v2:172.21.15.76:6836/2506930945,v1:172.21.15.76:6837/2506930945] up:boot 2024-04-26T23:42:44.430 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:43 smithi076 ceph-mon[32774]: fsmap 8 up:standby 2024-04-26T23:42:44.430 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:43 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T23:42:44.430 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:43 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/2090135709' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:45.167 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:44 smithi136 ceph-mon[33050]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:45.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:44 smithi072 ceph-mon[28184]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:45.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:44 smithi076 ceph-mon[32774]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:45.519 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:42:45.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.726+0000 7fea8e800640 1 -- 172.21.15.72:0/2997819363 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea8810a180 msgr2=0x7fea8810a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:45.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.726+0000 7fea8e800640 1 --2- 172.21.15.72:0/2997819363 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea8810a180 0x7fea8810a600 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fea70009860 tx=0x7fea7002a340 comp rx=0 tx=0).stop 2024-04-26T23:42:45.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.727+0000 7fea8e800640 1 -- 172.21.15.72:0/2997819363 shutdown_connections 2024-04-26T23:42:45.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.727+0000 7fea8e800640 1 --2- 172.21.15.72:0/2997819363 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea8810ab40 0x7fea881113d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:45.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.727+0000 7fea8e800640 1 --2- 172.21.15.72:0/2997819363 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea8810a180 0x7fea8810a600 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:45.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.727+0000 7fea8e800640 1 --2- 172.21.15.72:0/2997819363 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fea88108f90 0x7fea881093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:45.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.727+0000 7fea8e800640 1 -- 172.21.15.72:0/2997819363 >> 172.21.15.72:0/2997819363 conn(0x7fea88077f40 msgr2=0x7fea8807a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:45.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.727+0000 7fea8e800640 1 -- 172.21.15.72:0/2997819363 shutdown_connections 2024-04-26T23:42:45.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.727+0000 7fea8e800640 1 -- 172.21.15.72:0/2997819363 wait complete. 2024-04-26T23:42:45.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.728+0000 7fea8e800640 1 Processor -- start 2024-04-26T23:42:45.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.728+0000 7fea8e800640 1 -- start start 2024-04-26T23:42:45.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.728+0000 7fea8e800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fea88108f90 0x7fea8819f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:45.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.728+0000 7fea8e800640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea8810a180 0x7fea8819f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:45.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.728+0000 7fea8e800640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea8810ab40 0x7fea881a66f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:45.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.728+0000 7fea8e800640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fea88114ad0 con 0x7fea8810a180 2024-04-26T23:42:45.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.728+0000 7fea8e800640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fea88114950 con 0x7fea88108f90 2024-04-26T23:42:45.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.728+0000 7fea8e800640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fea88114c50 con 0x7fea8810ab40 2024-04-26T23:42:45.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.728+0000 7fea7fe00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea8810a180 0x7fea8819f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:45.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.728+0000 7fea7fe00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea8810a180 0x7fea8819f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45338/0 (socket says 172.21.15.72:45338) 2024-04-26T23:42:45.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.728+0000 7fea7fe00640 1 -- 172.21.15.72:0/3424936644 learned_addr learned my addr 172.21.15.72:0/3424936644 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:42:45.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.729+0000 7fea87400640 1 --2- 172.21.15.72:0/3424936644 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fea88108f90 0x7fea8819f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:45.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.729+0000 7fea87e00640 1 --2- 172.21.15.72:0/3424936644 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea8810ab40 0x7fea881a66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:45.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.729+0000 7fea7fe00640 1 -- 172.21.15.72:0/3424936644 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea8810ab40 msgr2=0x7fea881a66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:45.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.729+0000 7fea7fe00640 1 --2- 172.21.15.72:0/3424936644 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea8810ab40 0x7fea881a66f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:45.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.729+0000 7fea7fe00640 1 -- 172.21.15.72:0/3424936644 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fea88108f90 msgr2=0x7fea8819f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:45.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.729+0000 7fea7fe00640 1 --2- 172.21.15.72:0/3424936644 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fea88108f90 0x7fea8819f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:45.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.729+0000 7fea7fe00640 1 -- 172.21.15.72:0/3424936644 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea881a6d60 con 0x7fea8810a180 2024-04-26T23:42:45.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.729+0000 7fea7fe00640 1 --2- 172.21.15.72:0/3424936644 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea8810a180 0x7fea8819f680 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fea70005940 tx=0x7fea7002ae20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:45.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.729+0000 7fea84c00640 1 -- 172.21.15.72:0/3424936644 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea70040020 con 0x7fea8810a180 2024-04-26T23:42:45.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.730+0000 7fea84c00640 1 -- 172.21.15.72:0/3424936644 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea70030e50 con 0x7fea8810a180 2024-04-26T23:42:45.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.730+0000 7fea8e800640 1 -- 172.21.15.72:0/3424936644 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea881a6ff0 con 0x7fea8810a180 2024-04-26T23:42:45.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.730+0000 7fea84c00640 1 -- 172.21.15.72:0/3424936644 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea70037040 con 0x7fea8810a180 2024-04-26T23:42:45.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.730+0000 7fea8e800640 1 -- 172.21.15.72:0/3424936644 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea881a7460 con 0x7fea8810a180 2024-04-26T23:42:45.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.731+0000 7fea8e800640 1 -- 172.21.15.72:0/3424936644 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea881102d0 con 0x7fea8810a180 2024-04-26T23:42:45.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.734+0000 7fea84c00640 1 -- 172.21.15.72:0/3424936644 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fea70043020 con 0x7fea8810a180 2024-04-26T23:42:45.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.736+0000 7fea84c00640 1 --2- 172.21.15.72:0/3424936644 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fea6c0a3c30 0x7fea6c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:45.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.736+0000 7fea84c00640 1 -- 172.21.15.72:0/3424936644 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fea700f04e0 con 0x7fea8810a180 2024-04-26T23:42:45.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.736+0000 7fea87400640 1 --2- 172.21.15.72:0/3424936644 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fea6c0a3c30 0x7fea6c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:45.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.736+0000 7fea87400640 1 --2- 172.21.15.72:0/3424936644 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fea6c0a3c30 0x7fea6c0a60f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fea88109fe0 tx=0x7fea78005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:45.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.738+0000 7fea84c00640 1 -- 172.21.15.72:0/3424936644 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fea7004a020 con 0x7fea8810a180 2024-04-26T23:42:45.800 INFO:journalctl@ceph.mds.i.smithi136.stdout:Apr 26 23:42:45 smithi136 systemd[1]: Started Ceph mds.i for 3c8453ae-0425-11ef-bc93-c7b262605968. 2024-04-26T23:42:45.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.987+0000 7fea8e800640 1 -- 172.21.15.72:0/3424936644 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fea8802dad0 con 0x7fea8810a180 2024-04-26T23:42:45.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.988+0000 7fea84c00640 1 -- 172.21.15.72:0/3424936644 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7177 (secure 0 0 0) 0x7fea700bd340 con 0x7fea8810a180 2024-04-26T23:42:45.993 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:42:45.993 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/2386456166","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2386456166},{"type":"v1","addr":"172.21.15.72:6835","nonce":2386456166}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1528342053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1528342053},{"type":"v1","addr":"172.21.15.136:6833","nonce":1528342053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6833/2427238559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":2427238559},{"type":"v1","addr":"172.21.15.76:6833","nonce":2427238559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24491,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/4050689894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":4050689894},{"type":"v1","addr":"172.21.15.72:6837","nonce":4050689894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3681925367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3681925367},{"type":"v1","addr":"172.21.15.136:6835","nonce":3681925367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4251109085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4251109085},{"type":"v1","addr":"172.21.15.76:6835","nonce":4251109085}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24514,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/3639308555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3639308555},{"type":"v1","addr":"172.21.15.136:6837","nonce":3639308555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/2506930945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2506930945},{"type":"v1","addr":"172.21.15.76:6837","nonce":2506930945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-26T23:42:45.993 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 9 2024-04-26T23:42:45.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.990+0000 7fea8e800640 1 -- 172.21.15.72:0/3424936644 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fea6c0a3c30 msgr2=0x7fea6c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:45.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.990+0000 7fea8e800640 1 --2- 172.21.15.72:0/3424936644 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fea6c0a3c30 0x7fea6c0a60f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fea88109fe0 tx=0x7fea78005f90 comp rx=0 tx=0).stop 2024-04-26T23:42:45.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.991+0000 7fea8e800640 1 -- 172.21.15.72:0/3424936644 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea8810a180 msgr2=0x7fea8819f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:45.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.991+0000 7fea8e800640 1 --2- 172.21.15.72:0/3424936644 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea8810a180 0x7fea8819f680 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fea70005940 tx=0x7fea7002ae20 comp rx=0 tx=0).stop 2024-04-26T23:42:45.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.991+0000 7fea8e800640 1 -- 172.21.15.72:0/3424936644 shutdown_connections 2024-04-26T23:42:45.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.991+0000 7fea8e800640 1 --2- 172.21.15.72:0/3424936644 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fea6c0a3c30 0x7fea6c0a60f0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:45.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.991+0000 7fea8e800640 1 --2- 172.21.15.72:0/3424936644 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea8810ab40 0x7fea881a66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:45.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.991+0000 7fea8e800640 1 --2- 172.21.15.72:0/3424936644 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea8810a180 0x7fea8819f680 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:45.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.991+0000 7fea8e800640 1 --2- 172.21.15.72:0/3424936644 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fea88108f90 0x7fea8819f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:45.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.991+0000 7fea8e800640 1 -- 172.21.15.72:0/3424936644 >> 172.21.15.72:0/3424936644 conn(0x7fea88077f40 msgr2=0x7fea8810d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:45.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.991+0000 7fea8e800640 1 -- 172.21.15.72:0/3424936644 shutdown_connections 2024-04-26T23:42:45.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:45.991+0000 7fea8e800640 1 -- 172.21.15.72:0/3424936644 wait complete. 2024-04-26T23:42:47.078 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:46 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:47.079 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:46 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:47.079 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:46 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:47.079 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:46 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:42:47.079 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:46 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:42:47.079 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:46 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:42:47.079 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:46 smithi072 ceph-mon[28184]: Deploying daemon mds.g on smithi072 2024-04-26T23:42:47.079 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:46 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/3424936644' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:47.079 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:46 smithi072 ceph-mon[28184]: mds.? [v2:172.21.15.136:6838/3938677630,v1:172.21.15.136:6839/3938677630] up:boot 2024-04-26T23:42:47.079 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:46 smithi072 ceph-mon[28184]: fsmap 9 up:standby 2024-04-26T23:42:47.079 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:46 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T23:42:47.079 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:46 smithi072 ceph-mon[28184]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:47.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:46 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:47.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:46 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:47.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:46 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:47.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:46 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:42:47.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:46 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:42:47.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:46 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:42:47.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:46 smithi076 ceph-mon[32774]: Deploying daemon mds.g on smithi072 2024-04-26T23:42:47.180 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:46 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/3424936644' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:47.180 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:46 smithi076 ceph-mon[32774]: mds.? [v2:172.21.15.136:6838/3938677630,v1:172.21.15.136:6839/3938677630] up:boot 2024-04-26T23:42:47.180 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:46 smithi076 ceph-mon[32774]: fsmap 9 up:standby 2024-04-26T23:42:47.180 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:46 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T23:42:47.180 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:46 smithi076 ceph-mon[32774]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:47.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:46 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:47.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:46 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:47.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:46 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:47.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:46 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:42:47.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:46 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:42:47.301 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:46 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:42:47.301 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:46 smithi136 ceph-mon[33050]: Deploying daemon mds.g on smithi072 2024-04-26T23:42:47.301 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:46 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/3424936644' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:47.301 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:46 smithi136 ceph-mon[33050]: mds.? [v2:172.21.15.136:6838/3938677630,v1:172.21.15.136:6839/3938677630] up:boot 2024-04-26T23:42:47.301 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:46 smithi136 ceph-mon[33050]: fsmap 9 up:standby 2024-04-26T23:42:47.301 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:46 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T23:42:47.301 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:46 smithi136 ceph-mon[33050]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:48.015 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:42:48.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.228+0000 7f211fe00640 1 -- 172.21.15.72:0/4095721997 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2120104980 msgr2=0x7f2120104da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:48.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.228+0000 7f211fe00640 1 --2- 172.21.15.72:0/4095721997 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2120104980 0x7f2120104da0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2110009900 tx=0x7f211002a410 comp rx=0 tx=0).stop 2024-04-26T23:42:48.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.228+0000 7f211fe00640 1 -- 172.21.15.72:0/4095721997 shutdown_connections 2024-04-26T23:42:48.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.228+0000 7f211fe00640 1 --2- 172.21.15.72:0/4095721997 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2120106530 0x7f212010cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:48.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.228+0000 7f211fe00640 1 --2- 172.21.15.72:0/4095721997 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2120105b70 0x7f2120105ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:48.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.228+0000 7f211fe00640 1 --2- 172.21.15.72:0/4095721997 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2120104980 0x7f2120104da0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:48.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.228+0000 7f211fe00640 1 -- 172.21.15.72:0/4095721997 >> 172.21.15.72:0/4095721997 conn(0x7f21200fff10 msgr2=0x7f2120102330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:48.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.228+0000 7f211fe00640 1 -- 172.21.15.72:0/4095721997 shutdown_connections 2024-04-26T23:42:48.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.228+0000 7f211fe00640 1 -- 172.21.15.72:0/4095721997 wait complete. 2024-04-26T23:42:48.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.229+0000 7f211fe00640 1 Processor -- start 2024-04-26T23:42:48.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.229+0000 7f211fe00640 1 -- start start 2024-04-26T23:42:48.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.230+0000 7f211fe00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2120104980 0x7f212019aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:48.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.230+0000 7f211fe00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2120105b70 0x7f212019b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:48.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.230+0000 7f211fe00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2120106530 0x7f21201a2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:48.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.230+0000 7f211fe00640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f2120110500 con 0x7f2120106530 2024-04-26T23:42:48.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.230+0000 7f211fe00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2120110380 con 0x7f2120104980 2024-04-26T23:42:48.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.230+0000 7f211fe00640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f2120110680 con 0x7f2120105b70 2024-04-26T23:42:48.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.230+0000 7f211f400640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2120106530 0x7f21201a2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:48.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.230+0000 7f211ea00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2120104980 0x7f212019aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:48.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.230+0000 7f211ea00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2120104980 0x7f212019aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:33630/0 (socket says 172.21.15.72:33630) 2024-04-26T23:42:48.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.230+0000 7f211f400640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2120106530 0x7f21201a2150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:49002/0 (socket says 172.21.15.72:49002) 2024-04-26T23:42:48.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.230+0000 7f211ea00640 1 -- 172.21.15.72:0/2356232048 learned_addr learned my addr 172.21.15.72:0/2356232048 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:42:48.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.230+0000 7f2117e00640 1 --2- 172.21.15.72:0/2356232048 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2120105b70 0x7f212019b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:48.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.230+0000 7f211ea00640 1 -- 172.21.15.72:0/2356232048 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2120105b70 msgr2=0x7f212019b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:48.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.230+0000 7f211ea00640 1 --2- 172.21.15.72:0/2356232048 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2120105b70 0x7f212019b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:48.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.230+0000 7f211ea00640 1 -- 172.21.15.72:0/2356232048 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2120106530 msgr2=0x7f21201a2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:48.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.230+0000 7f211ea00640 1 --2- 172.21.15.72:0/2356232048 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2120106530 0x7f21201a2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:48.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.230+0000 7f211ea00640 1 -- 172.21.15.72:0/2356232048 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21201a27c0 con 0x7f2120104980 2024-04-26T23:42:48.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.230+0000 7f211f400640 1 --2- 172.21.15.72:0/2356232048 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2120106530 0x7f21201a2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:42:48.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.231+0000 7f211ea00640 1 --2- 172.21.15.72:0/2356232048 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2120104980 0x7f212019aba0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2110031500 tx=0x7f2110003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:48.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.231+0000 7f2117400640 1 -- 172.21.15.72:0/2356232048 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f211003f070 con 0x7f2120104980 2024-04-26T23:42:48.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.231+0000 7f2117400640 1 -- 172.21.15.72:0/2356232048 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2110031c50 con 0x7f2120104980 2024-04-26T23:42:48.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.231+0000 7f211fe00640 1 -- 172.21.15.72:0/2356232048 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21201a2a50 con 0x7f2120104980 2024-04-26T23:42:48.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.231+0000 7f2117400640 1 -- 172.21.15.72:0/2356232048 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2110035030 con 0x7f2120104980 2024-04-26T23:42:48.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.231+0000 7f211fe00640 1 -- 172.21.15.72:0/2356232048 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21201a2f40 con 0x7f2120104980 2024-04-26T23:42:48.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.232+0000 7f211fe00640 1 -- 172.21.15.72:0/2356232048 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f212010bca0 con 0x7f2120104980 2024-04-26T23:42:48.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.234+0000 7f2117400640 1 -- 172.21.15.72:0/2356232048 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2110042070 con 0x7f2120104980 2024-04-26T23:42:48.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.236+0000 7f2117400640 1 --2- 172.21.15.72:0/2356232048 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f21040a3bb0 0x7f21040a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:48.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.236+0000 7f2117400640 1 -- 172.21.15.72:0/2356232048 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f21100f0700 con 0x7f2120104980 2024-04-26T23:42:48.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.236+0000 7f2117e00640 1 --2- 172.21.15.72:0/2356232048 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f21040a3bb0 0x7f21040a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:48.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.237+0000 7f2117e00640 1 --2- 172.21.15.72:0/2356232048 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f21040a3bb0 0x7f21040a6070 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f212019c0b0 tx=0x7f2108005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:48.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.240+0000 7f2117400640 1 -- 172.21.15.72:0/2356232048 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f21100bce90 con 0x7f2120104980 2024-04-26T23:42:48.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.501+0000 7f211fe00640 1 -- 172.21.15.72:0/2356232048 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2120063530 con 0x7f2120104980 2024-04-26T23:42:48.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.502+0000 7f2117400640 1 -- 172.21.15.72:0/2356232048 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8007 (secure 0 0 0) 0x7f21100c1190 con 0x7f2120104980 2024-04-26T23:42:48.506 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:42:48.506 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/2386456166","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2386456166},{"type":"v1","addr":"172.21.15.72:6835","nonce":2386456166}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1528342053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1528342053},{"type":"v1","addr":"172.21.15.136:6833","nonce":1528342053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6833/2427238559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":2427238559},{"type":"v1","addr":"172.21.15.76:6833","nonce":2427238559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24491,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/4050689894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":4050689894},{"type":"v1","addr":"172.21.15.72:6837","nonce":4050689894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3681925367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3681925367},{"type":"v1","addr":"172.21.15.136:6835","nonce":3681925367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4251109085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4251109085},{"type":"v1","addr":"172.21.15.76:6835","nonce":4251109085}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24514,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/3639308555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3639308555},{"type":"v1","addr":"172.21.15.136:6837","nonce":3639308555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/2506930945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2506930945},{"type":"v1","addr":"172.21.15.76:6837","nonce":2506930945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/3938677630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3938677630},{"type":"v1","addr":"172.21.15.136:6839","nonce":3938677630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-26T23:42:48.506 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 10 2024-04-26T23:42:48.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.506+0000 7f211fe00640 1 -- 172.21.15.72:0/2356232048 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f21040a3bb0 msgr2=0x7f21040a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:48.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.506+0000 7f211fe00640 1 --2- 172.21.15.72:0/2356232048 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f21040a3bb0 0x7f21040a6070 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f212019c0b0 tx=0x7f2108005f90 comp rx=0 tx=0).stop 2024-04-26T23:42:48.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.506+0000 7f211fe00640 1 -- 172.21.15.72:0/2356232048 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2120104980 msgr2=0x7f212019aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:48.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.506+0000 7f211fe00640 1 --2- 172.21.15.72:0/2356232048 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2120104980 0x7f212019aba0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2110031500 tx=0x7f2110003c10 comp rx=0 tx=0).stop 2024-04-26T23:42:48.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.506+0000 7f211fe00640 1 -- 172.21.15.72:0/2356232048 shutdown_connections 2024-04-26T23:42:48.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.507+0000 7f211fe00640 1 --2- 172.21.15.72:0/2356232048 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f21040a3bb0 0x7f21040a6070 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:48.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.507+0000 7f211fe00640 1 --2- 172.21.15.72:0/2356232048 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2120106530 0x7f21201a2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:48.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.507+0000 7f211fe00640 1 --2- 172.21.15.72:0/2356232048 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2120105b70 0x7f212019b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:48.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.507+0000 7f211fe00640 1 --2- 172.21.15.72:0/2356232048 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2120104980 0x7f212019aba0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:48.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.507+0000 7f211fe00640 1 -- 172.21.15.72:0/2356232048 >> 172.21.15.72:0/2356232048 conn(0x7f21200fff10 msgr2=0x7f2120108e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:48.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.508+0000 7f211fe00640 1 -- 172.21.15.72:0/2356232048 shutdown_connections 2024-04-26T23:42:48.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:48.508+0000 7f211fe00640 1 -- 172.21.15.72:0/2356232048 wait complete. 2024-04-26T23:42:49.295 INFO:journalctl@ceph.mds.g.smithi072.stdout:Apr 26 23:42:49 smithi072 systemd[1]: Starting Ceph mds.g for 3c8453ae-0425-11ef-bc93-c7b262605968... 2024-04-26T23:42:49.728 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:49 smithi072 ceph-mon[28184]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:49.728 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:49 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/2356232048' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:49.728 INFO:journalctl@ceph.mds.g.smithi072.stdout:Apr 26 23:42:49 smithi072 podman[67697]: 2024-04-26 23:42:49.33865077 +0000 UTC m=+0.032729470 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T23:42:49.728 INFO:journalctl@ceph.mds.g.smithi072.stdout:Apr 26 23:42:49 smithi072 podman[67697]: 2024-04-26 23:42:49.465043058 +0000 UTC m=+0.159121815 container create e329aec6b4c236832ea7582e1823c7a0f91a216ba9f733bb4e26a90b958ef0ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-g, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T23:42:49.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:49 smithi136 ceph-mon[33050]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:49.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:49 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/2356232048' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:49.883 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:49 smithi076 ceph-mon[32774]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:49.883 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:49 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/2356232048' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:50.044 INFO:journalctl@ceph.mds.g.smithi072.stdout:Apr 26 23:42:49 smithi072 podman[67697]: 2024-04-26 23:42:49.797990891 +0000 UTC m=+0.492069672 container init e329aec6b4c236832ea7582e1823c7a0f91a216ba9f733bb4e26a90b958ef0ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-g, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux ) 2024-04-26T23:42:50.044 INFO:journalctl@ceph.mds.g.smithi072.stdout:Apr 26 23:42:49 smithi072 podman[67697]: 2024-04-26 23:42:49.807527403 +0000 UTC m=+0.501606147 container start e329aec6b4c236832ea7582e1823c7a0f91a216ba9f733bb4e26a90b958ef0ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-g, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T23:42:50.044 INFO:journalctl@ceph.mds.g.smithi072.stdout:Apr 26 23:42:49 smithi072 ceph-mds[67723]: -- 172.21.15.72:0/1347156541 <== mon.0 v2:172.21.15.72:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555d9e4b7a40 con 0x555d9e48b000 2024-04-26T23:42:50.044 INFO:journalctl@ceph.mds.g.smithi072.stdout:Apr 26 23:42:49 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-g[67714]: starting mds.g at 2024-04-26T23:42:50.044 INFO:journalctl@ceph.mds.g.smithi072.stdout:Apr 26 23:42:49 smithi072 bash[67697]: e329aec6b4c236832ea7582e1823c7a0f91a216ba9f733bb4e26a90b958ef0ba 2024-04-26T23:42:50.044 INFO:journalctl@ceph.mds.g.smithi072.stdout:Apr 26 23:42:49 smithi072 systemd[1]: Started Ceph mds.g for 3c8453ae-0425-11ef-bc93-c7b262605968. 2024-04-26T23:42:50.534 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:42:50.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:50.712+0000 7f2dcce00640 1 -- 172.21.15.72:0/1887562139 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2dc8105b20 msgr2=0x7f2dc8107f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:50.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:50.712+0000 7f2dcce00640 1 --2- 172.21.15.72:0/1887562139 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2dc8105b20 0x7f2dc8107f10 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2db4009900 tx=0x7f2db402a3f0 comp rx=0 tx=0).stop 2024-04-26T23:42:50.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:50.712+0000 7f2dcce00640 1 -- 172.21.15.72:0/1887562139 shutdown_connections 2024-04-26T23:42:50.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:50.712+0000 7f2dcce00640 1 --2- 172.21.15.72:0/1887562139 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2dc8108450 0x7f2dc810a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:50.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:50.712+0000 7f2dcce00640 1 --2- 172.21.15.72:0/1887562139 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2dc8105b20 0x7f2dc8107f10 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:50.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:50.712+0000 7f2dcce00640 1 --2- 172.21.15.72:0/1887562139 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2dc81031f0 0x7f2dc81055e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:50.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:50.712+0000 7f2dcce00640 1 -- 172.21.15.72:0/1887562139 >> 172.21.15.72:0/1887562139 conn(0x7f2dc80fcbc0 msgr2=0x7f2dc80ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:50.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:50.713+0000 7f2dcce00640 1 -- 172.21.15.72:0/1887562139 shutdown_connections 2024-04-26T23:42:50.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:50.713+0000 7f2dcce00640 1 -- 172.21.15.72:0/1887562139 wait complete. 2024-04-26T23:42:50.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:50.713+0000 7f2dcce00640 1 Processor -- start 2024-04-26T23:42:50.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:50.714+0000 7f2dcce00640 1 -- start start 2024-04-26T23:42:50.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:50.714+0000 7f2dcce00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2dc81031f0 0x7f2dc819abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:50.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:50.714+0000 7f2dcce00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2dc8105b20 0x7f2dc819b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:50.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:50.714+0000 7f2dcce00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2dc8108450 0x7f2dc81a2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:50.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:50.714+0000 7f2dcce00640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f2dc810e300 con 0x7f2dc81031f0 2024-04-26T23:42:50.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:50.714+0000 7f2dcce00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2dc810e180 con 0x7f2dc8108450 2024-04-26T23:42:50.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:50.714+0000 7f2dcce00640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f2dc810e480 con 0x7f2dc8105b20 2024-04-26T23:42:50.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:50.714+0000 7f2dc7400640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2dc81031f0 0x7f2dc819abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:50.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:50.714+0000 7f2dc7400640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2dc81031f0 0x7f2dc819abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:49038/0 (socket says 172.21.15.72:49038) 2024-04-26T23:42:50.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:50.714+0000 7f2dc7400640 1 -- 172.21.15.72:0/4108144474 learned_addr learned my addr 172.21.15.72:0/4108144474 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:42:50.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:50.714+0000 7f2dc7e00640 1 --2- 172.21.15.72:0/4108144474 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2dc8108450 0x7f2dc81a2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:50.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:50.714+0000 7f2dc6a00640 1 --2- 172.21.15.72:0/4108144474 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2dc8105b20 0x7f2dc819b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:50.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:50.715+0000 7f2dc7400640 1 -- 172.21.15.72:0/4108144474 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2dc8105b20 msgr2=0x7f2dc819b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:50.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:50.715+0000 7f2dc7400640 1 --2- 172.21.15.72:0/4108144474 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2dc8105b20 0x7f2dc819b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:50.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:50.715+0000 7f2dc7400640 1 -- 172.21.15.72:0/4108144474 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2dc8108450 msgr2=0x7f2dc81a2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:50.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:50.715+0000 7f2dc7400640 1 --2- 172.21.15.72:0/4108144474 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2dc8108450 0x7f2dc81a2170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:50.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:50.715+0000 7f2dc7400640 1 -- 172.21.15.72:0/4108144474 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dc81a27e0 con 0x7f2dc81031f0 2024-04-26T23:42:50.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:50.715+0000 7f2dc7400640 1 --2- 172.21.15.72:0/4108144474 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2dc81031f0 0x7f2dc819abc0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2db800e9f0 tx=0x7f2db800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:50.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:50.715+0000 7f2dbfe00640 1 -- 172.21.15.72:0/4108144474 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2db800cd80 con 0x7f2dc81031f0 2024-04-26T23:42:50.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:50.715+0000 7f2dcce00640 1 -- 172.21.15.72:0/4108144474 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2dc81a2ad0 con 0x7f2dc81031f0 2024-04-26T23:42:50.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:50.715+0000 7f2dbfe00640 1 -- 172.21.15.72:0/4108144474 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2db80054d0 con 0x7f2dc81031f0 2024-04-26T23:42:50.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:50.715+0000 7f2dbfe00640 1 -- 172.21.15.72:0/4108144474 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2db8010690 con 0x7f2dc81031f0 2024-04-26T23:42:50.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:50.715+0000 7f2dcce00640 1 -- 172.21.15.72:0/4108144474 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2dc81a2fa0 con 0x7f2dc81031f0 2024-04-26T23:42:50.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:50.717+0000 7f2dcce00640 1 -- 172.21.15.72:0/4108144474 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2dc8109fa0 con 0x7f2dc81031f0 2024-04-26T23:42:50.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:50.720+0000 7f2dbfe00640 1 -- 172.21.15.72:0/4108144474 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2db8004e90 con 0x7f2dc81031f0 2024-04-26T23:42:50.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:50.721+0000 7f2dbfe00640 1 --2- 172.21.15.72:0/4108144474 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f2dac0a3bb0 0x7f2dac0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:50.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:50.721+0000 7f2dbfe00640 1 -- 172.21.15.72:0/4108144474 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f2db80d0490 con 0x7f2dc81031f0 2024-04-26T23:42:50.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:50.721+0000 7f2dc6a00640 1 --2- 172.21.15.72:0/4108144474 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f2dac0a3bb0 0x7f2dac0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:50.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:50.722+0000 7f2dc6a00640 1 --2- 172.21.15.72:0/4108144474 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f2dac0a3bb0 0x7f2dac0a6070 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2db4006010 tx=0x7f2db40314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:50.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:50.730+0000 7f2dbfe00640 1 -- 172.21.15.72:0/4108144474 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f2db809d270 con 0x7f2dc81031f0 2024-04-26T23:42:51.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:51.025+0000 7f2dcce00640 1 -- 172.21.15.72:0/4108144474 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2dc802dad0 con 0x7f2dc81031f0 2024-04-26T23:42:51.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:51.027+0000 7f2dbfe00640 1 -- 172.21.15.72:0/4108144474 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8833 (secure 0 0 0) 0x7f2db809cb70 con 0x7f2dc81031f0 2024-04-26T23:42:51.031 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:42:51.032 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/2386456166","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2386456166},{"type":"v1","addr":"172.21.15.72:6835","nonce":2386456166}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1528342053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1528342053},{"type":"v1","addr":"172.21.15.136:6833","nonce":1528342053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6833/2427238559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":2427238559},{"type":"v1","addr":"172.21.15.76:6833","nonce":2427238559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24491,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/4050689894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":4050689894},{"type":"v1","addr":"172.21.15.72:6837","nonce":4050689894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3681925367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3681925367},{"type":"v1","addr":"172.21.15.136:6835","nonce":3681925367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4251109085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4251109085},{"type":"v1","addr":"172.21.15.76:6835","nonce":4251109085}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24514,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/3639308555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3639308555},{"type":"v1","addr":"172.21.15.136:6837","nonce":3639308555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/2506930945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2506930945},{"type":"v1","addr":"172.21.15.76:6837","nonce":2506930945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/3938677630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3938677630},{"type":"v1","addr":"172.21.15.136:6839","nonce":3938677630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24553,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6839/1403162102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6838","nonce":1403162102},{"type":"v1","addr":"172.21.15.72:6839","nonce":1403162102}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[]} 2024-04-26T23:42:51.032 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 11 2024-04-26T23:42:51.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:51.030+0000 7f2dcce00640 1 -- 172.21.15.72:0/4108144474 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f2dac0a3bb0 msgr2=0x7f2dac0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:51.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:51.030+0000 7f2dcce00640 1 --2- 172.21.15.72:0/4108144474 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f2dac0a3bb0 0x7f2dac0a6070 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2db4006010 tx=0x7f2db40314a0 comp rx=0 tx=0).stop 2024-04-26T23:42:51.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:51.030+0000 7f2dcce00640 1 -- 172.21.15.72:0/4108144474 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2dc81031f0 msgr2=0x7f2dc819abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:51.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:51.030+0000 7f2dcce00640 1 --2- 172.21.15.72:0/4108144474 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2dc81031f0 0x7f2dc819abc0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2db800e9f0 tx=0x7f2db800edf0 comp rx=0 tx=0).stop 2024-04-26T23:42:51.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:51.030+0000 7f2dcce00640 1 -- 172.21.15.72:0/4108144474 shutdown_connections 2024-04-26T23:42:51.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:51.030+0000 7f2dcce00640 1 --2- 172.21.15.72:0/4108144474 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f2dac0a3bb0 0x7f2dac0a6070 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:51.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:51.030+0000 7f2dcce00640 1 --2- 172.21.15.72:0/4108144474 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2dc8108450 0x7f2dc81a2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:51.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:51.030+0000 7f2dcce00640 1 --2- 172.21.15.72:0/4108144474 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2dc8105b20 0x7f2dc819b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:51.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:51.030+0000 7f2dcce00640 1 --2- 172.21.15.72:0/4108144474 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2dc81031f0 0x7f2dc819abc0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:51.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:51.030+0000 7f2dcce00640 1 -- 172.21.15.72:0/4108144474 >> 172.21.15.72:0/4108144474 conn(0x7f2dc80fcbc0 msgr2=0x7f2dc8102890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:51.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:51.030+0000 7f2dcce00640 1 -- 172.21.15.72:0/4108144474 shutdown_connections 2024-04-26T23:42:51.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:51.030+0000 7f2dcce00640 1 -- 172.21.15.72:0/4108144474 wait complete. 2024-04-26T23:42:51.153 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:51 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:51.153 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:51 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:51.153 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:51 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:51.153 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:51 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:42:51.153 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:51 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:42:51.153 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:51 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:42:51.154 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:51 smithi076 ceph-mon[32774]: Deploying daemon mds.a on smithi072 2024-04-26T23:42:51.154 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:51 smithi076 ceph-mon[32774]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:51.154 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:51 smithi076 ceph-mon[32774]: mds.? [v2:172.21.15.72:6838/1403162102,v1:172.21.15.72:6839/1403162102] up:boot 2024-04-26T23:42:51.154 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:51 smithi076 ceph-mon[32774]: fsmap 10 up:standby 2024-04-26T23:42:51.154 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:51 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T23:42:51.295 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:51 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:51.295 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:51 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:51.295 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:51 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:51.295 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:51 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:42:51.295 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:51 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:42:51.295 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:51 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:42:51.295 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:51 smithi072 ceph-mon[28184]: Deploying daemon mds.a on smithi072 2024-04-26T23:42:51.296 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:51 smithi072 ceph-mon[28184]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:51.296 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:51 smithi072 ceph-mon[28184]: mds.? [v2:172.21.15.72:6838/1403162102,v1:172.21.15.72:6839/1403162102] up:boot 2024-04-26T23:42:51.296 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:51 smithi072 ceph-mon[28184]: fsmap 10 up:standby 2024-04-26T23:42:51.296 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:51 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T23:42:51.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:51 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:51.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:51 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:51.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:51 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:51.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:51 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:42:51.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:51 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:42:51.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:51 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:42:51.301 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:51 smithi136 ceph-mon[33050]: Deploying daemon mds.a on smithi072 2024-04-26T23:42:51.301 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:51 smithi136 ceph-mon[33050]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:51.301 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:51 smithi136 ceph-mon[33050]: mds.? [v2:172.21.15.72:6838/1403162102,v1:172.21.15.72:6839/1403162102] up:boot 2024-04-26T23:42:51.301 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:51 smithi136 ceph-mon[33050]: fsmap 10 up:standby 2024-04-26T23:42:51.301 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:51 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T23:42:52.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:52 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/4108144474' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:52.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:52 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/4108144474' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:52.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:52 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/4108144474' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:53.052 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:42:53.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:53 smithi072 ceph-mon[28184]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:53.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:53 smithi136 ceph-mon[33050]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:53.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.424+0000 7f3234200640 1 -- 172.21.15.72:0/1845117409 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f322c103b60 msgr2=0x7f322c103fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:53.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.424+0000 7f3234200640 1 --2- 172.21.15.72:0/1845117409 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f322c103b60 0x7f322c103fe0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3218009900 tx=0x7f321802a410 comp rx=0 tx=0).stop 2024-04-26T23:42:53.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.424+0000 7f3234200640 1 -- 172.21.15.72:0/1845117409 shutdown_connections 2024-04-26T23:42:53.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.424+0000 7f3234200640 1 --2- 172.21.15.72:0/1845117409 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f322c103b60 0x7f322c103fe0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:53.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.424+0000 7f3234200640 1 --2- 172.21.15.72:0/1845117409 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f322c104520 0x7f322c10adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:53.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.424+0000 7f3234200640 1 --2- 172.21.15.72:0/1845117409 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f322c102970 0x7f322c102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:53.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.424+0000 7f3234200640 1 -- 172.21.15.72:0/1845117409 >> 172.21.15.72:0/1845117409 conn(0x7f322c0fdee0 msgr2=0x7f322c100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:53.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.424+0000 7f3234200640 1 -- 172.21.15.72:0/1845117409 shutdown_connections 2024-04-26T23:42:53.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.424+0000 7f3234200640 1 -- 172.21.15.72:0/1845117409 wait complete. 2024-04-26T23:42:53.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:53 smithi076 ceph-mon[32774]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:53.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.425+0000 7f3234200640 1 Processor -- start 2024-04-26T23:42:53.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.426+0000 7f3234200640 1 -- start start 2024-04-26T23:42:53.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.426+0000 7f3234200640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f322c102970 0x7f322c19adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:53.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.426+0000 7f3234200640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f322c103b60 0x7f322c19b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:53.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.426+0000 7f3234200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f322c104520 0x7f322c1a2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:53.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.426+0000 7f3234200640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f322c10e4d0 con 0x7f322c103b60 2024-04-26T23:42:53.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.426+0000 7f3234200640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f322c10e350 con 0x7f322c104520 2024-04-26T23:42:53.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.426+0000 7f3234200640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f322c10e650 con 0x7f322c102970 2024-04-26T23:42:53.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.426+0000 7f3230e00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f322c103b60 0x7f322c19b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:53.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.426+0000 7f3231800640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f322c102970 0x7f322c19adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:53.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.426+0000 7f3231800640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f322c102970 0x7f322c19adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.72:53006/0 (socket says 172.21.15.72:53006) 2024-04-26T23:42:53.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.426+0000 7f3231800640 1 -- 172.21.15.72:0/7411746 learned_addr learned my addr 172.21.15.72:0/7411746 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:42:53.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.426+0000 7f3232200640 1 --2- 172.21.15.72:0/7411746 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f322c104520 0x7f322c1a2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:53.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.427+0000 7f3231800640 1 -- 172.21.15.72:0/7411746 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f322c104520 msgr2=0x7f322c1a2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:53.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.427+0000 7f3231800640 1 --2- 172.21.15.72:0/7411746 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f322c104520 0x7f322c1a2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:53.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.427+0000 7f3231800640 1 -- 172.21.15.72:0/7411746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f322c103b60 msgr2=0x7f322c19b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:53.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.427+0000 7f3231800640 1 --2- 172.21.15.72:0/7411746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f322c103b60 0x7f322c19b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:53.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.427+0000 7f3231800640 1 -- 172.21.15.72:0/7411746 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f322c1a29e0 con 0x7f322c102970 2024-04-26T23:42:53.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.427+0000 7f3231800640 1 --2- 172.21.15.72:0/7411746 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f322c102970 0x7f322c19adc0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f322000e9c0 tx=0x7f322000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:53.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.427+0000 7f322a000640 1 -- 172.21.15.72:0/7411746 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f322000cd80 con 0x7f322c102970 2024-04-26T23:42:53.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.427+0000 7f322a000640 1 -- 172.21.15.72:0/7411746 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f32200035f0 con 0x7f322c102970 2024-04-26T23:42:53.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.428+0000 7f322a000640 1 -- 172.21.15.72:0/7411746 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3220010690 con 0x7f322c102970 2024-04-26T23:42:53.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.428+0000 7f3234200640 1 -- 172.21.15.72:0/7411746 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f322c1a2cd0 con 0x7f322c102970 2024-04-26T23:42:53.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.428+0000 7f3234200640 1 -- 172.21.15.72:0/7411746 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f322c1a31a0 con 0x7f322c102970 2024-04-26T23:42:53.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.429+0000 7f3234200640 1 -- 172.21.15.72:0/7411746 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f322c1a3370 con 0x7f322c102970 2024-04-26T23:42:53.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.431+0000 7f322a000640 1 -- 172.21.15.72:0/7411746 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f32200037a0 con 0x7f322c102970 2024-04-26T23:42:53.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.433+0000 7f322a000640 1 --2- 172.21.15.72:0/7411746 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f32140a3c30 0x7f32140a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:53.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.434+0000 7f322a000640 1 -- 172.21.15.72:0/7411746 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f32200d02e0 con 0x7f322c102970 2024-04-26T23:42:53.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.434+0000 7f3230e00640 1 --2- 172.21.15.72:0/7411746 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f32140a3c30 0x7f32140a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:53.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.435+0000 7f3230e00640 1 --2- 172.21.15.72:0/7411746 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f32140a3c30 0x7f32140a60f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3218031500 tx=0x7f3218005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:53.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.435+0000 7f322a000640 1 -- 172.21.15.72:0/7411746 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f322009d0c0 con 0x7f322c102970 2024-04-26T23:42:53.707 INFO:journalctl@ceph.mds.a.smithi072.stdout:Apr 26 23:42:53 smithi072 systemd[1]: Starting Ceph mds.a for 3c8453ae-0425-11ef-bc93-c7b262605968... 2024-04-26T23:42:53.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.706+0000 7f3234200640 1 -- 172.21.15.72:0/7411746 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f322c1a3540 con 0x7f322c102970 2024-04-26T23:42:53.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.707+0000 7f322a000640 1 -- 172.21.15.72:0/7411746 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8833 (secure 0 0 0) 0x7f322009c9c0 con 0x7f322c102970 2024-04-26T23:42:53.711 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:42:53.711 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/2386456166","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2386456166},{"type":"v1","addr":"172.21.15.72:6835","nonce":2386456166}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1528342053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1528342053},{"type":"v1","addr":"172.21.15.136:6833","nonce":1528342053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6833/2427238559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":2427238559},{"type":"v1","addr":"172.21.15.76:6833","nonce":2427238559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24491,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/4050689894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":4050689894},{"type":"v1","addr":"172.21.15.72:6837","nonce":4050689894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3681925367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3681925367},{"type":"v1","addr":"172.21.15.136:6835","nonce":3681925367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4251109085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4251109085},{"type":"v1","addr":"172.21.15.76:6835","nonce":4251109085}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24514,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/3639308555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3639308555},{"type":"v1","addr":"172.21.15.136:6837","nonce":3639308555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/2506930945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2506930945},{"type":"v1","addr":"172.21.15.76:6837","nonce":2506930945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/3938677630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3938677630},{"type":"v1","addr":"172.21.15.136:6839","nonce":3938677630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24553,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6839/1403162102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6838","nonce":1403162102},{"type":"v1","addr":"172.21.15.72:6839","nonce":1403162102}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[]} 2024-04-26T23:42:53.711 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 11 2024-04-26T23:42:53.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.709+0000 7f3234200640 1 -- 172.21.15.72:0/7411746 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f32140a3c30 msgr2=0x7f32140a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:53.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.709+0000 7f3234200640 1 --2- 172.21.15.72:0/7411746 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f32140a3c30 0x7f32140a60f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3218031500 tx=0x7f3218005a20 comp rx=0 tx=0).stop 2024-04-26T23:42:53.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.709+0000 7f3234200640 1 -- 172.21.15.72:0/7411746 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f322c102970 msgr2=0x7f322c19adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:53.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.709+0000 7f3234200640 1 --2- 172.21.15.72:0/7411746 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f322c102970 0x7f322c19adc0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f322000e9c0 tx=0x7f322000ee80 comp rx=0 tx=0).stop 2024-04-26T23:42:53.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.709+0000 7f3234200640 1 -- 172.21.15.72:0/7411746 shutdown_connections 2024-04-26T23:42:53.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.709+0000 7f3234200640 1 --2- 172.21.15.72:0/7411746 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f32140a3c30 0x7f32140a60f0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:53.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.709+0000 7f3234200640 1 --2- 172.21.15.72:0/7411746 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f322c104520 0x7f322c1a2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:53.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.709+0000 7f3234200640 1 --2- 172.21.15.72:0/7411746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f322c103b60 0x7f322c19b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:53.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.709+0000 7f3234200640 1 --2- 172.21.15.72:0/7411746 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f322c102970 0x7f322c19adc0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:53.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.709+0000 7f3234200640 1 -- 172.21.15.72:0/7411746 >> 172.21.15.72:0/7411746 conn(0x7f322c0fdee0 msgr2=0x7f322c074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:53.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.710+0000 7f3234200640 1 -- 172.21.15.72:0/7411746 shutdown_connections 2024-04-26T23:42:53.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:53.710+0000 7f3234200640 1 -- 172.21.15.72:0/7411746 wait complete. 2024-04-26T23:42:54.025 INFO:journalctl@ceph.mds.a.smithi072.stdout:Apr 26 23:42:53 smithi072 podman[68085]: 2024-04-26 23:42:53.611976904 +0000 UTC m=+0.036928542 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T23:42:54.025 INFO:journalctl@ceph.mds.a.smithi072.stdout:Apr 26 23:42:53 smithi072 podman[68085]: 2024-04-26 23:42:53.734066885 +0000 UTC m=+0.159018536 container create 97c724a655b48d7f0ef0873189c364fd980ad547f4356a39593a78778676cce9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-a, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-26T23:42:54.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:54 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/7411746' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:54.294 INFO:journalctl@ceph.mds.a.smithi072.stdout:Apr 26 23:42:54 smithi072 podman[68085]: 2024-04-26 23:42:54.082223048 +0000 UTC m=+0.507174715 container init 97c724a655b48d7f0ef0873189c364fd980ad547f4356a39593a78778676cce9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-a, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-26T23:42:54.294 INFO:journalctl@ceph.mds.a.smithi072.stdout:Apr 26 23:42:54 smithi072 podman[68085]: 2024-04-26 23:42:54.093852514 +0000 UTC m=+0.518804178 container start 97c724a655b48d7f0ef0873189c364fd980ad547f4356a39593a78778676cce9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-a, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-26T23:42:54.294 INFO:journalctl@ceph.mds.a.smithi072.stdout:Apr 26 23:42:54 smithi072 ceph-mds[68107]: -- 172.21.15.72:0/2388322553 <== mon.0 v2:172.21.15.72:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558ae0b71a40 con 0x558ae0b45000 2024-04-26T23:42:54.295 INFO:journalctl@ceph.mds.a.smithi072.stdout:Apr 26 23:42:54 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-a[68103]: starting mds.a at 2024-04-26T23:42:54.295 INFO:journalctl@ceph.mds.a.smithi072.stdout:Apr 26 23:42:54 smithi072 bash[68085]: 97c724a655b48d7f0ef0873189c364fd980ad547f4356a39593a78778676cce9 2024-04-26T23:42:54.295 INFO:journalctl@ceph.mds.a.smithi072.stdout:Apr 26 23:42:54 smithi072 systemd[1]: Started Ceph mds.a for 3c8453ae-0425-11ef-bc93-c7b262605968. 2024-04-26T23:42:54.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:54 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/7411746' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:54.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:54 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/7411746' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:55.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:55 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:55.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:55 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:55.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:55 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:55.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:55 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:42:55.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:55 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:42:55.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:55 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:42:55.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:55 smithi076 ceph-mon[32774]: Deploying daemon mds.b on smithi076 2024-04-26T23:42:55.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:55 smithi076 ceph-mon[32774]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:55.680 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:55 smithi076 ceph-mon[32774]: mds.? [v2:172.21.15.72:6840/1356345077,v1:172.21.15.72:6841/1356345077] up:boot 2024-04-26T23:42:55.680 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:55 smithi076 ceph-mon[32774]: fsmap 11 up:standby 2024-04-26T23:42:55.680 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:55 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T23:42:55.733 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:42:55.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:55 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:55.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:55 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:55.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:55 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:55.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:55 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:42:55.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:55 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:42:55.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:55 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:42:55.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:55 smithi072 ceph-mon[28184]: Deploying daemon mds.b on smithi076 2024-04-26T23:42:55.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:55 smithi072 ceph-mon[28184]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:55.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:55 smithi072 ceph-mon[28184]: mds.? [v2:172.21.15.72:6840/1356345077,v1:172.21.15.72:6841/1356345077] up:boot 2024-04-26T23:42:55.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:55 smithi072 ceph-mon[28184]: fsmap 11 up:standby 2024-04-26T23:42:55.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:55 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T23:42:55.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:55 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:55.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:55 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:55.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:55 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:55.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:55 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T23:42:55.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:55 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T23:42:55.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:55 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:42:55.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:55 smithi136 ceph-mon[33050]: Deploying daemon mds.b on smithi076 2024-04-26T23:42:55.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:55 smithi136 ceph-mon[33050]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:55.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:55 smithi136 ceph-mon[33050]: mds.? [v2:172.21.15.72:6840/1356345077,v1:172.21.15.72:6841/1356345077] up:boot 2024-04-26T23:42:55.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:55 smithi136 ceph-mon[33050]: fsmap 11 up:standby 2024-04-26T23:42:55.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:55 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T23:42:55.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:55.942+0000 7ff5d7000640 1 -- 172.21.15.72:0/2817567051 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5d010a940 msgr2=0x7ff5d01111d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:55.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:55.942+0000 7ff5d7000640 1 --2- 172.21.15.72:0/2817567051 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5d010a940 0x7ff5d01111d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff5c8009be0 tx=0x7ff5c802a6c0 comp rx=0 tx=0).stop 2024-04-26T23:42:55.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:55.943+0000 7ff5d7000640 1 -- 172.21.15.72:0/2817567051 shutdown_connections 2024-04-26T23:42:55.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:55.943+0000 7ff5d7000640 1 --2- 172.21.15.72:0/2817567051 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff5d0109f80 0x7ff5d010a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:55.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:55.943+0000 7ff5d7000640 1 --2- 172.21.15.72:0/2817567051 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5d010a940 0x7ff5d01111d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:55.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:55.943+0000 7ff5d7000640 1 --2- 172.21.15.72:0/2817567051 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff5d0108d90 0x7ff5d01091b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:55.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:55.943+0000 7ff5d7000640 1 -- 172.21.15.72:0/2817567051 >> 172.21.15.72:0/2817567051 conn(0x7ff5d0077d30 msgr2=0x7ff5d007a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:55.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:55.943+0000 7ff5d7000640 1 -- 172.21.15.72:0/2817567051 shutdown_connections 2024-04-26T23:42:55.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:55.943+0000 7ff5d7000640 1 -- 172.21.15.72:0/2817567051 wait complete. 2024-04-26T23:42:55.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:55.944+0000 7ff5d7000640 1 Processor -- start 2024-04-26T23:42:55.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:55.944+0000 7ff5d7000640 1 -- start start 2024-04-26T23:42:55.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:55.944+0000 7ff5d7000640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff5d0108d90 0x7ff5d019ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:55.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:55.944+0000 7ff5d7000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff5d0109f80 0x7ff5d019f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:55.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:55.944+0000 7ff5d7000640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5d010a940 0x7ff5d01a64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:55.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:55.944+0000 7ff5d7000640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d01148c0 con 0x7ff5d0108d90 2024-04-26T23:42:55.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:55.944+0000 7ff5d7000640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d0114740 con 0x7ff5d0109f80 2024-04-26T23:42:55.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:55.944+0000 7ff5d7000640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d0114a40 con 0x7ff5d010a940 2024-04-26T23:42:55.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:55.945+0000 7ff5d5200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff5d0109f80 0x7ff5d019f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:55.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:55.945+0000 7ff5d5c00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff5d0108d90 0x7ff5d019ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:55.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:55.945+0000 7ff5d5200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff5d0109f80 0x7ff5d019f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:33722/0 (socket says 172.21.15.72:33722) 2024-04-26T23:42:55.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:55.945+0000 7ff5d5200640 1 -- 172.21.15.72:0/1411219161 learned_addr learned my addr 172.21.15.72:0/1411219161 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:42:55.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:55.945+0000 7ff5d6600640 1 --2- 172.21.15.72:0/1411219161 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5d010a940 0x7ff5d01a64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:55.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:55.945+0000 7ff5d5200640 1 -- 172.21.15.72:0/1411219161 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5d010a940 msgr2=0x7ff5d01a64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:55.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:55.945+0000 7ff5d5200640 1 --2- 172.21.15.72:0/1411219161 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5d010a940 0x7ff5d01a64e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:55.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:55.945+0000 7ff5d5200640 1 -- 172.21.15.72:0/1411219161 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff5d0108d90 msgr2=0x7ff5d019ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:55.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:55.945+0000 7ff5d5200640 1 --2- 172.21.15.72:0/1411219161 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff5d0108d90 0x7ff5d019ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:55.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:55.945+0000 7ff5d5200640 1 -- 172.21.15.72:0/1411219161 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5d01a6b50 con 0x7ff5d0109f80 2024-04-26T23:42:55.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:55.945+0000 7ff5d5200640 1 --2- 172.21.15.72:0/1411219161 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff5d0109f80 0x7ff5d019f470 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff5c400e9f0 tx=0x7ff5c400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:55.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:55.946+0000 7ff5ce000640 1 -- 172.21.15.72:0/1411219161 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c400ccc0 con 0x7ff5d0109f80 2024-04-26T23:42:55.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:55.946+0000 7ff5ce000640 1 -- 172.21.15.72:0/1411219161 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5c400ce60 con 0x7ff5d0109f80 2024-04-26T23:42:55.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:55.946+0000 7ff5d7000640 1 -- 172.21.15.72:0/1411219161 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5d01a6e40 con 0x7ff5d0109f80 2024-04-26T23:42:55.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:55.946+0000 7ff5ce000640 1 -- 172.21.15.72:0/1411219161 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c4010430 con 0x7ff5d0109f80 2024-04-26T23:42:55.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:55.946+0000 7ff5d7000640 1 -- 172.21.15.72:0/1411219161 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5d01a7300 con 0x7ff5d0109f80 2024-04-26T23:42:55.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:55.947+0000 7ff5d7000640 1 -- 172.21.15.72:0/1411219161 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5d01100b0 con 0x7ff5d0109f80 2024-04-26T23:42:55.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:55.949+0000 7ff5ce000640 1 -- 172.21.15.72:0/1411219161 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ff5c40046c0 con 0x7ff5d0109f80 2024-04-26T23:42:55.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:55.950+0000 7ff5ce000640 1 --2- 172.21.15.72:0/1411219161 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff5b80a3c30 0x7ff5b80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:55.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:55.950+0000 7ff5ce000640 1 -- 172.21.15.72:0/1411219161 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7ff5c40d0340 con 0x7ff5d0109f80 2024-04-26T23:42:55.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:55.950+0000 7ff5d5c00640 1 --2- 172.21.15.72:0/1411219161 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff5b80a3c30 0x7ff5b80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:55.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:55.951+0000 7ff5d5c00640 1 --2- 172.21.15.72:0/1411219161 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff5b80a3c30 0x7ff5b80a60f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0109de0 tx=0x7ff5c0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:55.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:55.954+0000 7ff5ce000640 1 -- 172.21.15.72:0/1411219161 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff5c409d120 con 0x7ff5d0109f80 2024-04-26T23:42:56.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:56.193+0000 7ff5d7000640 1 -- 172.21.15.72:0/1411219161 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff5d00635d0 con 0x7ff5d0109f80 2024-04-26T23:42:56.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:56.195+0000 7ff5ce000640 1 -- 172.21.15.72:0/1411219161 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9659 (secure 0 0 0) 0x7ff5c409ca20 con 0x7ff5d0109f80 2024-04-26T23:42:56.199 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:42:56.199 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/2386456166","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2386456166},{"type":"v1","addr":"172.21.15.72:6835","nonce":2386456166}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1528342053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1528342053},{"type":"v1","addr":"172.21.15.136:6833","nonce":1528342053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6833/2427238559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":2427238559},{"type":"v1","addr":"172.21.15.76:6833","nonce":2427238559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24491,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/4050689894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":4050689894},{"type":"v1","addr":"172.21.15.72:6837","nonce":4050689894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3681925367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3681925367},{"type":"v1","addr":"172.21.15.136:6835","nonce":3681925367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4251109085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4251109085},{"type":"v1","addr":"172.21.15.76:6835","nonce":4251109085}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24514,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/3639308555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3639308555},{"type":"v1","addr":"172.21.15.136:6837","nonce":3639308555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/2506930945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2506930945},{"type":"v1","addr":"172.21.15.76:6837","nonce":2506930945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/3938677630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3938677630},{"type":"v1","addr":"172.21.15.136:6839","nonce":3938677630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24553,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6839/1403162102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6838","nonce":1403162102},{"type":"v1","addr":"172.21.15.72:6839","nonce":1403162102}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24571,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6841/1356345077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6840","nonce":1356345077},{"type":"v1","addr":"172.21.15.72:6841","nonce":1356345077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[]} 2024-04-26T23:42:56.199 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 12 2024-04-26T23:42:56.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:56.197+0000 7ff5d7000640 1 -- 172.21.15.72:0/1411219161 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff5b80a3c30 msgr2=0x7ff5b80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:56.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:56.198+0000 7ff5d7000640 1 --2- 172.21.15.72:0/1411219161 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff5b80a3c30 0x7ff5b80a60f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0109de0 tx=0x7ff5c0009260 comp rx=0 tx=0).stop 2024-04-26T23:42:56.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:56.198+0000 7ff5d7000640 1 -- 172.21.15.72:0/1411219161 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff5d0109f80 msgr2=0x7ff5d019f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:56.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:56.198+0000 7ff5d7000640 1 --2- 172.21.15.72:0/1411219161 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff5d0109f80 0x7ff5d019f470 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff5c400e9f0 tx=0x7ff5c400edf0 comp rx=0 tx=0).stop 2024-04-26T23:42:56.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:56.198+0000 7ff5d7000640 1 -- 172.21.15.72:0/1411219161 shutdown_connections 2024-04-26T23:42:56.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:56.198+0000 7ff5d7000640 1 --2- 172.21.15.72:0/1411219161 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff5b80a3c30 0x7ff5b80a60f0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:56.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:56.198+0000 7ff5d7000640 1 --2- 172.21.15.72:0/1411219161 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff5d0109f80 0x7ff5d019f470 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:56.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:56.198+0000 7ff5d7000640 1 --2- 172.21.15.72:0/1411219161 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5d010a940 0x7ff5d01a64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:56.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:56.198+0000 7ff5d7000640 1 --2- 172.21.15.72:0/1411219161 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff5d0108d90 0x7ff5d019ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:56.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:56.198+0000 7ff5d7000640 1 -- 172.21.15.72:0/1411219161 >> 172.21.15.72:0/1411219161 conn(0x7ff5d0077d30 msgr2=0x7ff5d010d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:56.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:56.198+0000 7ff5d7000640 1 -- 172.21.15.72:0/1411219161 shutdown_connections 2024-04-26T23:42:56.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:56.198+0000 7ff5d7000640 1 -- 172.21.15.72:0/1411219161 wait complete. 2024-04-26T23:42:56.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:56 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/1411219161' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:56.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:56 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/1411219161' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:56.679 INFO:journalctl@ceph.mds.b.smithi076.stdout:Apr 26 23:42:56 smithi076 systemd[1]: Starting Ceph mds.b for 3c8453ae-0425-11ef-bc93-c7b262605968... 2024-04-26T23:42:56.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:56 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/1411219161' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:56.962 INFO:journalctl@ceph.mds.b.smithi076.stdout:Apr 26 23:42:56 smithi076 podman[62238]: 2024-04-26 23:42:56.596678938 +0000 UTC m=+0.017998647 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T23:42:56.962 INFO:journalctl@ceph.mds.b.smithi076.stdout:Apr 26 23:42:56 smithi076 podman[62238]: 2024-04-26 23:42:56.713715284 +0000 UTC m=+0.135035048 container create 66c08b6d66f75919babe496dfc949337489a7d351004f15f44c2ce689b50cfb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T23:42:56.962 INFO:journalctl@ceph.mds.b.smithi076.stdout:Apr 26 23:42:56 smithi076 podman[62238]: 2024-04-26 23:42:56.93333103 +0000 UTC m=+0.354650740 container init 66c08b6d66f75919babe496dfc949337489a7d351004f15f44c2ce689b50cfb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T23:42:56.962 INFO:journalctl@ceph.mds.b.smithi076.stdout:Apr 26 23:42:56 smithi076 podman[62238]: 2024-04-26 23:42:56.935955994 +0000 UTC m=+0.357275709 container start 66c08b6d66f75919babe496dfc949337489a7d351004f15f44c2ce689b50cfb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0) 2024-04-26T23:42:57.375 INFO:journalctl@ceph.mds.b.smithi076.stdout:Apr 26 23:42:56 smithi076 ceph-mds[62257]: -- 172.21.15.76:0/2789841392 <== mon.2 v2:172.21.15.76:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b624d7da40 con 0x55b624d50800 2024-04-26T23:42:57.375 INFO:journalctl@ceph.mds.b.smithi076.stdout:Apr 26 23:42:56 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-b[62253]: starting mds.b at 2024-04-26T23:42:57.375 INFO:journalctl@ceph.mds.b.smithi076.stdout:Apr 26 23:42:57 smithi076 bash[62238]: 66c08b6d66f75919babe496dfc949337489a7d351004f15f44c2ce689b50cfb8 2024-04-26T23:42:57.375 INFO:journalctl@ceph.mds.b.smithi076.stdout:Apr 26 23:42:57 smithi076 systemd[1]: Started Ceph mds.b for 3c8453ae-0425-11ef-bc93-c7b262605968. 2024-04-26T23:42:57.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:57 smithi076 ceph-mon[32774]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:57.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:57 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:57.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:57 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:57.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:57 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:57.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:57 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:57.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:57 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:57.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:57 smithi072 ceph-mon[28184]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:57.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:57 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:57.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:57 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:57.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:57 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:57.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:57 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:57.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:57 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:57.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:57 smithi136 ceph-mon[33050]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:57.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:57 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:57.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:57 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:57.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:57 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:57.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:57 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:57.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:57 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:58.221 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:42:58.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.413+0000 7f43cf000640 1 -- 172.21.15.72:0/54872007 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43c8104b70 msgr2=0x7f43c8104f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:58.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.413+0000 7f43cf000640 1 --2- 172.21.15.72:0/54872007 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43c8104b70 0x7f43c8104f90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f43b8009900 tx=0x7f43b802a410 comp rx=0 tx=0).stop 2024-04-26T23:42:58.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.413+0000 7f43cf000640 1 -- 172.21.15.72:0/54872007 shutdown_connections 2024-04-26T23:42:58.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.413+0000 7f43cf000640 1 --2- 172.21.15.72:0/54872007 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43c8106720 0x7f43c810cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:58.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.413+0000 7f43cf000640 1 --2- 172.21.15.72:0/54872007 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43c8105d60 0x7f43c81061e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:58.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.413+0000 7f43cf000640 1 --2- 172.21.15.72:0/54872007 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43c8104b70 0x7f43c8104f90 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:58.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.413+0000 7f43cf000640 1 -- 172.21.15.72:0/54872007 >> 172.21.15.72:0/54872007 conn(0x7f43c8100120 msgr2=0x7f43c8102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:58.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.413+0000 7f43cf000640 1 -- 172.21.15.72:0/54872007 shutdown_connections 2024-04-26T23:42:58.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.413+0000 7f43cf000640 1 -- 172.21.15.72:0/54872007 wait complete. 2024-04-26T23:42:58.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.414+0000 7f43cf000640 1 Processor -- start 2024-04-26T23:42:58.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.415+0000 7f43cf000640 1 -- start start 2024-04-26T23:42:58.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.415+0000 7f43cf000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43c8104b70 0x7f43c819adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:58.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.415+0000 7f43cf000640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43c8105d60 0x7f43c819b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:58.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.415+0000 7f43cf000640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43c8106720 0x7f43c81a2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:58.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.415+0000 7f43cf000640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f43c81106c0 con 0x7f43c8105d60 2024-04-26T23:42:58.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.415+0000 7f43cf000640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f43c8110540 con 0x7f43c8104b70 2024-04-26T23:42:58.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.415+0000 7f43cf000640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f43c8110840 con 0x7f43c8106720 2024-04-26T23:42:58.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.415+0000 7f43c7400640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43c8105d60 0x7f43c819b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:58.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.415+0000 7f43cd000640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43c8106720 0x7f43c81a2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:58.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.415+0000 7f43c7400640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43c8105d60 0x7f43c819b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48502/0 (socket says 172.21.15.72:48502) 2024-04-26T23:42:58.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.415+0000 7f43c7400640 1 -- 172.21.15.72:0/1515809777 learned_addr learned my addr 172.21.15.72:0/1515809777 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:42:58.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.415+0000 7f43cd000640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43c8106720 0x7f43c81a2370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.72:44760/0 (socket says 172.21.15.72:44760) 2024-04-26T23:42:58.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.415+0000 7f43c7e00640 1 --2- 172.21.15.72:0/1515809777 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43c8104b70 0x7f43c819adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:58.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.416+0000 7f43cd000640 1 -- 172.21.15.72:0/1515809777 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43c8104b70 msgr2=0x7f43c819adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:58.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.416+0000 7f43cd000640 1 --2- 172.21.15.72:0/1515809777 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43c8104b70 0x7f43c819adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:58.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.416+0000 7f43cd000640 1 -- 172.21.15.72:0/1515809777 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43c8105d60 msgr2=0x7f43c819b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:58.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.416+0000 7f43cd000640 1 --2- 172.21.15.72:0/1515809777 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43c8105d60 0x7f43c819b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:58.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.416+0000 7f43cd000640 1 -- 172.21.15.72:0/1515809777 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43c81a29e0 con 0x7f43c8106720 2024-04-26T23:42:58.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.416+0000 7f43c7400640 1 --2- 172.21.15.72:0/1515809777 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43c8105d60 0x7f43c819b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:42:58.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.416+0000 7f43cd000640 1 --2- 172.21.15.72:0/1515809777 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43c8106720 0x7f43c81a2370 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f43c0009fd0 tx=0x7f43c000ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:58.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.416+0000 7f43c4c00640 1 -- 172.21.15.72:0/1515809777 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c000ce20 con 0x7f43c8106720 2024-04-26T23:42:58.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.416+0000 7f43c4c00640 1 -- 172.21.15.72:0/1515809777 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43c0005020 con 0x7f43c8106720 2024-04-26T23:42:58.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.416+0000 7f43c4c00640 1 -- 172.21.15.72:0/1515809777 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c0010690 con 0x7f43c8106720 2024-04-26T23:42:58.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.416+0000 7f43cf000640 1 -- 172.21.15.72:0/1515809777 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43c81a2cd0 con 0x7f43c8106720 2024-04-26T23:42:58.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.416+0000 7f43cf000640 1 -- 172.21.15.72:0/1515809777 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43c81a31d0 con 0x7f43c8106720 2024-04-26T23:42:58.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.418+0000 7f43cf000640 1 -- 172.21.15.72:0/1515809777 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43c810bfb0 con 0x7f43c8106720 2024-04-26T23:42:58.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.419+0000 7f43c4c00640 1 -- 172.21.15.72:0/1515809777 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f43c00054d0 con 0x7f43c8106720 2024-04-26T23:42:58.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.421+0000 7f43c4c00640 1 --2- 172.21.15.72:0/1515809777 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f43ac0a3c30 0x7f43ac0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:58.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.421+0000 7f43c4c00640 1 -- 172.21.15.72:0/1515809777 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f43c00d0670 con 0x7f43c8106720 2024-04-26T23:42:58.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.421+0000 7f43c7e00640 1 --2- 172.21.15.72:0/1515809777 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f43ac0a3c30 0x7f43ac0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:58.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.422+0000 7f43c7e00640 1 --2- 172.21.15.72:0/1515809777 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f43ac0a3c30 0x7f43ac0a60f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f43b8005ab0 tx=0x7f43b8005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:58.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.429+0000 7f43c4c00640 1 -- 172.21.15.72:0/1515809777 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f43c009d450 con 0x7f43c8106720 2024-04-26T23:42:58.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:58 smithi072 ceph-mon[28184]: mds.? [v2:172.21.15.76:6838/3405829058,v1:172.21.15.76:6839/3405829058] up:boot 2024-04-26T23:42:58.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:58 smithi072 ceph-mon[28184]: fsmap 12 up:standby 2024-04-26T23:42:58.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:58 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T23:42:58.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:58 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:42:58.555 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:58 smithi076 ceph-mon[32774]: mds.? [v2:172.21.15.76:6838/3405829058,v1:172.21.15.76:6839/3405829058] up:boot 2024-04-26T23:42:58.555 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:58 smithi076 ceph-mon[32774]: fsmap 12 up:standby 2024-04-26T23:42:58.555 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:58 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T23:42:58.555 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:58 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:42:58.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.682+0000 7f43cf000640 1 -- 172.21.15.72:0/1515809777 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f43c8063560 con 0x7f43c8106720 2024-04-26T23:42:58.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.684+0000 7f43c4c00640 1 -- 172.21.15.72:0/1515809777 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10485 (secure 0 0 0) 0x7f43c009cd50 con 0x7f43c8106720 2024-04-26T23:42:58.688 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:42:58.688 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/2386456166","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2386456166},{"type":"v1","addr":"172.21.15.72:6835","nonce":2386456166}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1528342053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1528342053},{"type":"v1","addr":"172.21.15.136:6833","nonce":1528342053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6833/2427238559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":2427238559},{"type":"v1","addr":"172.21.15.76:6833","nonce":2427238559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24491,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/4050689894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":4050689894},{"type":"v1","addr":"172.21.15.72:6837","nonce":4050689894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3681925367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3681925367},{"type":"v1","addr":"172.21.15.136:6835","nonce":3681925367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4251109085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4251109085},{"type":"v1","addr":"172.21.15.76:6835","nonce":4251109085}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24514,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/3639308555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3639308555},{"type":"v1","addr":"172.21.15.136:6837","nonce":3639308555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/2506930945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2506930945},{"type":"v1","addr":"172.21.15.76:6837","nonce":2506930945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/3938677630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3938677630},{"type":"v1","addr":"172.21.15.136:6839","nonce":3938677630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24553,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6839/1403162102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6838","nonce":1403162102},{"type":"v1","addr":"172.21.15.72:6839","nonce":1403162102}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24571,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6841/1356345077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6840","nonce":1356345077},{"type":"v1","addr":"172.21.15.72:6841","nonce":1356345077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24590,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6839/3405829058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":3405829058},{"type":"v1","addr":"172.21.15.76:6839","nonce":3405829058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-04-26T23:42:58.688 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 13 2024-04-26T23:42:58.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.686+0000 7f43cf000640 1 -- 172.21.15.72:0/1515809777 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f43ac0a3c30 msgr2=0x7f43ac0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:58.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.686+0000 7f43cf000640 1 --2- 172.21.15.72:0/1515809777 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f43ac0a3c30 0x7f43ac0a60f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f43b8005ab0 tx=0x7f43b8005a20 comp rx=0 tx=0).stop 2024-04-26T23:42:58.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.686+0000 7f43cf000640 1 -- 172.21.15.72:0/1515809777 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43c8106720 msgr2=0x7f43c81a2370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:58.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.686+0000 7f43cf000640 1 --2- 172.21.15.72:0/1515809777 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43c8106720 0x7f43c81a2370 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f43c0009fd0 tx=0x7f43c000ef10 comp rx=0 tx=0).stop 2024-04-26T23:42:58.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.686+0000 7f43cf000640 1 -- 172.21.15.72:0/1515809777 shutdown_connections 2024-04-26T23:42:58.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.686+0000 7f43cf000640 1 --2- 172.21.15.72:0/1515809777 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f43ac0a3c30 0x7f43ac0a60f0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:58.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.686+0000 7f43cf000640 1 --2- 172.21.15.72:0/1515809777 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43c8106720 0x7f43c81a2370 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:58.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.686+0000 7f43cf000640 1 --2- 172.21.15.72:0/1515809777 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43c8105d60 0x7f43c819b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:58.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.686+0000 7f43cf000640 1 --2- 172.21.15.72:0/1515809777 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43c8104b70 0x7f43c819adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:58.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.686+0000 7f43cf000640 1 -- 172.21.15.72:0/1515809777 >> 172.21.15.72:0/1515809777 conn(0x7f43c8100120 msgr2=0x7f43c8109180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:58.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.686+0000 7f43cf000640 1 -- 172.21.15.72:0/1515809777 shutdown_connections 2024-04-26T23:42:58.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.686+0000 7f43cf000640 1 -- 172.21.15.72:0/1515809777 wait complete. 2024-04-26T23:42:58.709 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:42:58.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:58 smithi136 ceph-mon[33050]: mds.? [v2:172.21.15.76:6838/3405829058,v1:172.21.15.76:6839/3405829058] up:boot 2024-04-26T23:42:58.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:58 smithi136 ceph-mon[33050]: fsmap 12 up:standby 2024-04-26T23:42:58.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:58 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T23:42:58.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:58 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:42:58.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.888+0000 7fb496000640 1 -- 172.21.15.72:0/2047315147 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb490104340 msgr2=0x7fb49010abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:58.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.888+0000 7fb496000640 1 --2- 172.21.15.72:0/2047315147 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb490104340 0x7fb49010abd0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb488009ca0 tx=0x7fb48802a8c0 comp rx=0 tx=0).stop 2024-04-26T23:42:58.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.888+0000 7fb496000640 1 -- 172.21.15.72:0/2047315147 shutdown_connections 2024-04-26T23:42:58.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.888+0000 7fb496000640 1 --2- 172.21.15.72:0/2047315147 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb490104340 0x7fb49010abd0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:58.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.888+0000 7fb496000640 1 --2- 172.21.15.72:0/2047315147 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb490103980 0x7fb490103e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:58.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.888+0000 7fb496000640 1 --2- 172.21.15.72:0/2047315147 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb490102790 0x7fb490102bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:58.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.888+0000 7fb496000640 1 -- 172.21.15.72:0/2047315147 >> 172.21.15.72:0/2047315147 conn(0x7fb4900fdd00 msgr2=0x7fb490100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:58.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.888+0000 7fb496000640 1 -- 172.21.15.72:0/2047315147 shutdown_connections 2024-04-26T23:42:58.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.888+0000 7fb496000640 1 -- 172.21.15.72:0/2047315147 wait complete. 2024-04-26T23:42:58.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.890+0000 7fb496000640 1 Processor -- start 2024-04-26T23:42:58.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.890+0000 7fb496000640 1 -- start start 2024-04-26T23:42:58.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.890+0000 7fb496000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb490102790 0x7fb49019ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:58.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.890+0000 7fb496000640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb490103980 0x7fb49019b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:58.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.890+0000 7fb496000640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb490104340 0x7fb4901a2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:58.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.891+0000 7fb496000640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fb49010e310 con 0x7fb490103980 2024-04-26T23:42:58.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.891+0000 7fb496000640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb49010e190 con 0x7fb490102790 2024-04-26T23:42:58.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.891+0000 7fb496000640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fb49010e490 con 0x7fb490104340 2024-04-26T23:42:58.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.891+0000 7fb494c00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb490102790 0x7fb49019ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:58.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.891+0000 7fb494c00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb490102790 0x7fb49019ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:32996/0 (socket says 172.21.15.72:32996) 2024-04-26T23:42:58.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.891+0000 7fb494c00640 1 -- 172.21.15.72:0/2684157997 learned_addr learned my addr 172.21.15.72:0/2684157997 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:42:58.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.891+0000 7fb48fe00640 1 --2- 172.21.15.72:0/2684157997 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb490103980 0x7fb49019b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:58.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.891+0000 7fb495600640 1 --2- 172.21.15.72:0/2684157997 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb490104340 0x7fb4901a2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:58.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.891+0000 7fb494c00640 1 -- 172.21.15.72:0/2684157997 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb490104340 msgr2=0x7fb4901a2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:58.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.891+0000 7fb494c00640 1 --2- 172.21.15.72:0/2684157997 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb490104340 0x7fb4901a2110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:58.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.891+0000 7fb494c00640 1 -- 172.21.15.72:0/2684157997 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb490103980 msgr2=0x7fb49019b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:58.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.891+0000 7fb494c00640 1 --2- 172.21.15.72:0/2684157997 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb490103980 0x7fb49019b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:58.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.891+0000 7fb494c00640 1 -- 172.21.15.72:0/2684157997 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4901a2780 con 0x7fb490102790 2024-04-26T23:42:58.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.891+0000 7fb48fe00640 1 --2- 172.21.15.72:0/2684157997 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb490103980 0x7fb49019b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:42:58.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.892+0000 7fb494c00640 1 --2- 172.21.15.72:0/2684157997 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb490102790 0x7fb49019ab60 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb48400e9c0 tx=0x7fb48400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:58.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.892+0000 7fb48d600640 1 -- 172.21.15.72:0/2684157997 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb48400cd80 con 0x7fb490102790 2024-04-26T23:42:58.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.892+0000 7fb48d600640 1 -- 172.21.15.72:0/2684157997 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4840035f0 con 0x7fb490102790 2024-04-26T23:42:58.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.892+0000 7fb496000640 1 -- 172.21.15.72:0/2684157997 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4901a2a70 con 0x7fb490102790 2024-04-26T23:42:58.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.892+0000 7fb48d600640 1 -- 172.21.15.72:0/2684157997 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb484010690 con 0x7fb490102790 2024-04-26T23:42:58.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.892+0000 7fb496000640 1 -- 172.21.15.72:0/2684157997 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4901a2fc0 con 0x7fb490102790 2024-04-26T23:42:58.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.893+0000 7fb496000640 1 -- 172.21.15.72:0/2684157997 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4901a3250 con 0x7fb490102790 2024-04-26T23:42:58.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.895+0000 7fb48d600640 1 -- 172.21.15.72:0/2684157997 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb4840037a0 con 0x7fb490102790 2024-04-26T23:42:58.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.896+0000 7fb48d600640 1 --2- 172.21.15.72:0/2684157997 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fb4780a3bb0 0x7fb4780a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:58.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.896+0000 7fb48d600640 1 -- 172.21.15.72:0/2684157997 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb4840d0360 con 0x7fb490102790 2024-04-26T23:42:58.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.896+0000 7fb48fe00640 1 --2- 172.21.15.72:0/2684157997 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fb4780a3bb0 0x7fb4780a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:58.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.897+0000 7fb48fe00640 1 --2- 172.21.15.72:0/2684157997 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fb4780a3bb0 0x7fb4780a6070 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb49019c070 tx=0x7fb47c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:58.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:58.900+0000 7fb48d600640 1 -- 172.21.15.72:0/2684157997 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb48409d140 con 0x7fb490102790 2024-04-26T23:42:59.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.096+0000 7fb496000640 1 -- 172.21.15.72:0/2684157997 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb4901a3420 con 0x7fb490102790 2024-04-26T23:42:59.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.098+0000 7fb48d600640 1 -- 172.21.15.72:0/2684157997 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18409 (secure 0 0 0) 0x7fb48409ca40 con 0x7fb490102790 2024-04-26T23:42:59.102 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:42:59.102 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":81,"fsid":"3c8453ae-0425-11ef-bc93-c7b262605968","created":"2024-04-26T23:33:48.088300+0000","modified":"2024-04-26T23:42:05.494742+0000","last_up_change":"2024-04-26T23:42:04.482891+0000","last_in_change":"2024-04-26T23:41:44.784415+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:37:44.428727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e20c9ebf-56f6-47c1-a11b-e1e615af042b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6803","nonce":1639298875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6805","nonce":1639298875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6809","nonce":1639298875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6807","nonce":1639298875}]},"public_addr":"172.21.15.72:6803/1639298875","cluster_addr":"172.21.15.72:6805/1639298875","heartbeat_back_addr":"172.21.15.72:6809/1639298875","heartbeat_front_addr":"172.21.15.72:6807/1639298875","state":["exists","up"]},{"osd":1,"uuid":"e6cac469-780c-4d66-b560-b686072371fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6801","nonce":4185342193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6803","nonce":4185342193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6807","nonce":4185342193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6805","nonce":4185342193}]},"public_addr":"172.21.15.76:6801/4185342193","cluster_addr":"172.21.15.76:6803/4185342193","heartbeat_back_addr":"172.21.15.76:6807/4185342193","heartbeat_front_addr":"172.21.15.76:6805/4185342193","state":["exists","up"]},{"osd":2,"uuid":"838a49fc-18de-446c-8336-67b3e05da231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6801","nonce":3033860799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6803","nonce":3033860799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6807","nonce":3033860799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6805","nonce":3033860799}]},"public_addr":"172.21.15.136:6801/3033860799","cluster_addr":"172.21.15.136:6803/3033860799","heartbeat_back_addr":"172.21.15.136:6807/3033860799","heartbeat_front_addr":"172.21.15.136:6805/3033860799","state":["exists","up"]},{"osd":3,"uuid":"e6da89be-5ba7-48a4-9306-f75498660e0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6811","nonce":2071107815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6813","nonce":2071107815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6817","nonce":2071107815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6815","nonce":2071107815}]},"public_addr":"172.21.15.72:6811/2071107815","cluster_addr":"172.21.15.72:6813/2071107815","heartbeat_back_addr":"172.21.15.72:6817/2071107815","heartbeat_front_addr":"172.21.15.72:6815/2071107815","state":["exists","up"]},{"osd":4,"uuid":"0b17c491-d798-488f-892d-521ef2fbc51e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6809","nonce":147449238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6811","nonce":147449238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6815","nonce":147449238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6813","nonce":147449238}]},"public_addr":"172.21.15.76:6809/147449238","cluster_addr":"172.21.15.76:6811/147449238","heartbeat_back_addr":"172.21.15.76:6815/147449238","heartbeat_front_addr":"172.21.15.76:6813/147449238","state":["exists","up"]},{"osd":5,"uuid":"c7e887b3-ab38-4608-9bfd-1ed49956e8af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6809","nonce":2320149212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6811","nonce":2320149212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6815","nonce":2320149212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6813","nonce":2320149212}]},"public_addr":"172.21.15.136:6809/2320149212","cluster_addr":"172.21.15.136:6811/2320149212","heartbeat_back_addr":"172.21.15.136:6815/2320149212","heartbeat_front_addr":"172.21.15.136:6813/2320149212","state":["exists","up"]},{"osd":6,"uuid":"f0bd604b-0f57-4a38-9295-eb200264f1ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6819","nonce":389499207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6821","nonce":389499207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6825","nonce":389499207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6823","nonce":389499207}]},"public_addr":"172.21.15.72:6819/389499207","cluster_addr":"172.21.15.72:6821/389499207","heartbeat_back_addr":"172.21.15.72:6825/389499207","heartbeat_front_addr":"172.21.15.72:6823/389499207","state":["exists","up"]},{"osd":7,"uuid":"1879d4e4-038e-4cd6-a3ec-3a940bcc3797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6817","nonce":375888575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6819","nonce":375888575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6823","nonce":375888575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6821","nonce":375888575}]},"public_addr":"172.21.15.76:6817/375888575","cluster_addr":"172.21.15.76:6819/375888575","heartbeat_back_addr":"172.21.15.76:6823/375888575","heartbeat_front_addr":"172.21.15.76:6821/375888575","state":["exists","up"]},{"osd":8,"uuid":"6f404062-97e7-4ccb-a582-76dc1ffae548","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6817","nonce":3446774052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6819","nonce":3446774052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6823","nonce":3446774052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6821","nonce":3446774052}]},"public_addr":"172.21.15.136:6817/3446774052","cluster_addr":"172.21.15.136:6819/3446774052","heartbeat_back_addr":"172.21.15.136:6823/3446774052","heartbeat_front_addr":"172.21.15.136:6821/3446774052","state":["exists","up"]},{"osd":9,"uuid":"2989ca68-3164-4667-9bcf-230dcbb7b8f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6827","nonce":3423789152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6829","nonce":3423789152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6832","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6833","nonce":3423789152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6831","nonce":3423789152}]},"public_addr":"172.21.15.72:6827/3423789152","cluster_addr":"172.21.15.72:6829/3423789152","heartbeat_back_addr":"172.21.15.72:6833/3423789152","heartbeat_front_addr":"172.21.15.72:6831/3423789152","state":["exists","up"]},{"osd":10,"uuid":"9ec6d46a-0913-4208-97b5-167188ee8cb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6825","nonce":1120991574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6827","nonce":1120991574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6831","nonce":1120991574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6829","nonce":1120991574}]},"public_addr":"172.21.15.76:6825/1120991574","cluster_addr":"172.21.15.76:6827/1120991574","heartbeat_back_addr":"172.21.15.76:6831/1120991574","heartbeat_front_addr":"172.21.15.76:6829/1120991574","state":["exists","up"]},{"osd":11,"uuid":"bdd0774f-6e58-4918-b4cc-afd4f1118d3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6825","nonce":1135995193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6827","nonce":1135995193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6831","nonce":1135995193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6829","nonce":1135995193}]},"public_addr":"172.21.15.136:6825/1135995193","cluster_addr":"172.21.15.136:6827/1135995193","heartbeat_back_addr":"172.21.15.136:6831/1135995193","heartbeat_front_addr":"172.21.15.136:6829/1135995193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:36:52.128788+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:37:17.523654+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:37:41.634677+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:38:15.060797+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:38:41.234111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:39:06.783560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:39:40.884386+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:40:08.340465+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:40:34.344858+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:41:08.525309+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:41:36.782624+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:42:02.756888+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:0/3625071330":"2024-04-27T23:34:58.266218+0000","172.21.15.72:6801/2903979134":"2024-04-27T23:34:58.266218+0000","172.21.15.72:6800/2903979134":"2024-04-27T23:34:58.266218+0000","172.21.15.72:0/1519142576":"2024-04-27T23:34:58.266218+0000","172.21.15.72:0/3669626460":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/945623297":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/3479150073":"2024-04-27T23:34:24.233319+0000","172.21.15.72:6800/4175896290":"2024-04-27T23:34:24.233319+0000","172.21.15.72:6801/4175896290":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/3783175012":"2024-04-27T23:34:58.266218+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:42:59.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.100+0000 7fb496000640 1 -- 172.21.15.72:0/2684157997 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fb4780a3bb0 msgr2=0x7fb4780a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:59.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.100+0000 7fb496000640 1 --2- 172.21.15.72:0/2684157997 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fb4780a3bb0 0x7fb4780a6070 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb49019c070 tx=0x7fb47c005f90 comp rx=0 tx=0).stop 2024-04-26T23:42:59.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.101+0000 7fb496000640 1 -- 172.21.15.72:0/2684157997 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb490102790 msgr2=0x7fb49019ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:59.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.101+0000 7fb496000640 1 --2- 172.21.15.72:0/2684157997 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb490102790 0x7fb49019ab60 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb48400e9c0 tx=0x7fb48400ee80 comp rx=0 tx=0).stop 2024-04-26T23:42:59.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.101+0000 7fb496000640 1 -- 172.21.15.72:0/2684157997 shutdown_connections 2024-04-26T23:42:59.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.101+0000 7fb496000640 1 --2- 172.21.15.72:0/2684157997 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fb4780a3bb0 0x7fb4780a6070 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:59.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.101+0000 7fb496000640 1 --2- 172.21.15.72:0/2684157997 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb490104340 0x7fb4901a2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:59.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.101+0000 7fb496000640 1 --2- 172.21.15.72:0/2684157997 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb490103980 0x7fb49019b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:59.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.101+0000 7fb496000640 1 --2- 172.21.15.72:0/2684157997 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb490102790 0x7fb49019ab60 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:59.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.101+0000 7fb496000640 1 -- 172.21.15.72:0/2684157997 >> 172.21.15.72:0/2684157997 conn(0x7fb4900fdd00 msgr2=0x7fb490074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:59.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.101+0000 7fb496000640 1 -- 172.21.15.72:0/2684157997 shutdown_connections 2024-04-26T23:42:59.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.101+0000 7fb496000640 1 -- 172.21.15.72:0/2684157997 wait complete. 2024-04-26T23:42:59.123 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:37:44.428727+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T23:42:59.123 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:42:59.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.324+0000 7fb546c00640 1 -- 172.21.15.72:0/147297314 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb540102410 msgr2=0x7fb540102890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:59.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.324+0000 7fb546c00640 1 --2- 172.21.15.72:0/147297314 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb540102410 0x7fb540102890 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb538009900 tx=0x7fb53802a410 comp rx=0 tx=0).stop 2024-04-26T23:42:59.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.325+0000 7fb546c00640 1 -- 172.21.15.72:0/147297314 shutdown_connections 2024-04-26T23:42:59.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.325+0000 7fb546c00640 1 --2- 172.21.15.72:0/147297314 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb540102410 0x7fb540102890 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:59.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.325+0000 7fb546c00640 1 --2- 172.21.15.72:0/147297314 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb540101a70 0x7fb540101ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:59.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.325+0000 7fb546c00640 1 --2- 172.21.15.72:0/147297314 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb540101110 0x7fb540101530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:59.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.325+0000 7fb546c00640 1 -- 172.21.15.72:0/147297314 >> 172.21.15.72:0/147297314 conn(0x7fb5400fccd0 msgr2=0x7fb5400ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:59.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.325+0000 7fb546c00640 1 -- 172.21.15.72:0/147297314 shutdown_connections 2024-04-26T23:42:59.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.325+0000 7fb546c00640 1 -- 172.21.15.72:0/147297314 wait complete. 2024-04-26T23:42:59.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.326+0000 7fb546c00640 1 Processor -- start 2024-04-26T23:42:59.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.326+0000 7fb546c00640 1 -- start start 2024-04-26T23:42:59.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.326+0000 7fb546c00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb540101110 0x7fb54019abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:59.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.326+0000 7fb546c00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb540101a70 0x7fb54019b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:59.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.326+0000 7fb546c00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb540102410 0x7fb5401a21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:59.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.326+0000 7fb546c00640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fb5401105b0 con 0x7fb540101110 2024-04-26T23:42:59.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.326+0000 7fb546c00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb540110430 con 0x7fb540101a70 2024-04-26T23:42:59.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.326+0000 7fb546c00640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fb540110730 con 0x7fb540102410 2024-04-26T23:42:59.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.326+0000 7fb544e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb540101a70 0x7fb54019b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:59.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.326+0000 7fb545800640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb540101110 0x7fb54019abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:59.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.326+0000 7fb544e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb540101a70 0x7fb54019b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:33018/0 (socket says 172.21.15.72:33018) 2024-04-26T23:42:59.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.326+0000 7fb544e00640 1 -- 172.21.15.72:0/565019489 learned_addr learned my addr 172.21.15.72:0/565019489 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:42:59.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.326+0000 7fb546200640 1 --2- 172.21.15.72:0/565019489 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb540102410 0x7fb5401a21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:59.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.327+0000 7fb544e00640 1 -- 172.21.15.72:0/565019489 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb540102410 msgr2=0x7fb5401a21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:59.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.327+0000 7fb544e00640 1 --2- 172.21.15.72:0/565019489 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb540102410 0x7fb5401a21a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:59.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.327+0000 7fb544e00640 1 -- 172.21.15.72:0/565019489 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb540101110 msgr2=0x7fb54019abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:59.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.327+0000 7fb544e00640 1 --2- 172.21.15.72:0/565019489 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb540101110 0x7fb54019abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:59.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.327+0000 7fb544e00640 1 -- 172.21.15.72:0/565019489 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5401a2810 con 0x7fb540101a70 2024-04-26T23:42:59.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.327+0000 7fb545800640 1 --2- 172.21.15.72:0/565019489 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb540101110 0x7fb54019abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:42:59.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.327+0000 7fb544e00640 1 --2- 172.21.15.72:0/565019489 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb540101a70 0x7fb54019b130 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb52c00e9f0 tx=0x7fb52c00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:59.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.327+0000 7fb53e000640 1 -- 172.21.15.72:0/565019489 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb52c00cd80 con 0x7fb540101a70 2024-04-26T23:42:59.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.327+0000 7fb53e000640 1 -- 172.21.15.72:0/565019489 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb52c0054d0 con 0x7fb540101a70 2024-04-26T23:42:59.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.327+0000 7fb53e000640 1 -- 172.21.15.72:0/565019489 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb52c010690 con 0x7fb540101a70 2024-04-26T23:42:59.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.327+0000 7fb546c00640 1 -- 172.21.15.72:0/565019489 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5401a2b00 con 0x7fb540101a70 2024-04-26T23:42:59.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.328+0000 7fb546c00640 1 -- 172.21.15.72:0/565019489 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5401a3050 con 0x7fb540101a70 2024-04-26T23:42:59.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.329+0000 7fb546c00640 1 -- 172.21.15.72:0/565019489 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb54010bff0 con 0x7fb540101a70 2024-04-26T23:42:59.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.331+0000 7fb53e000640 1 -- 172.21.15.72:0/565019489 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb52c004e90 con 0x7fb540101a70 2024-04-26T23:42:59.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.332+0000 7fb53e000640 1 --2- 172.21.15.72:0/565019489 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fb5280a3c30 0x7fb5280a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:59.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.333+0000 7fb53e000640 1 -- 172.21.15.72:0/565019489 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb52c0d0340 con 0x7fb540101a70 2024-04-26T23:42:59.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.333+0000 7fb545800640 1 --2- 172.21.15.72:0/565019489 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fb5280a3c30 0x7fb5280a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:59.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.333+0000 7fb545800640 1 --2- 172.21.15.72:0/565019489 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fb5280a3c30 0x7fb5280a60f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb534005fd0 tx=0x7fb534009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:59.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.335+0000 7fb53e000640 1 -- 172.21.15.72:0/565019489 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb52c09d120 con 0x7fb540101a70 2024-04-26T23:42:59.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.562+0000 7fb546c00640 1 -- 172.21.15.72:0/565019489 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb54002dc60 con 0x7fb540101a70 2024-04-26T23:42:59.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.562+0000 7fb53e000640 1 -- 172.21.15.72:0/565019489 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7fb52c09ca20 con 0x7fb540101a70 2024-04-26T23:42:59.566 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-04-26T23:42:59.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.565+0000 7fb546c00640 1 -- 172.21.15.72:0/565019489 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fb5280a3c30 msgr2=0x7fb5280a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:59.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.565+0000 7fb546c00640 1 --2- 172.21.15.72:0/565019489 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fb5280a3c30 0x7fb5280a60f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb534005fd0 tx=0x7fb534009500 comp rx=0 tx=0).stop 2024-04-26T23:42:59.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.565+0000 7fb546c00640 1 -- 172.21.15.72:0/565019489 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb540101a70 msgr2=0x7fb54019b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:59.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.565+0000 7fb546c00640 1 --2- 172.21.15.72:0/565019489 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb540101a70 0x7fb54019b130 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb52c00e9f0 tx=0x7fb52c00edf0 comp rx=0 tx=0).stop 2024-04-26T23:42:59.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.565+0000 7fb546c00640 1 -- 172.21.15.72:0/565019489 shutdown_connections 2024-04-26T23:42:59.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.565+0000 7fb546c00640 1 --2- 172.21.15.72:0/565019489 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fb5280a3c30 0x7fb5280a60f0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:59.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.565+0000 7fb546c00640 1 --2- 172.21.15.72:0/565019489 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb540101a70 0x7fb54019b130 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:59.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.565+0000 7fb546c00640 1 --2- 172.21.15.72:0/565019489 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb540102410 0x7fb5401a21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:59.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.565+0000 7fb546c00640 1 --2- 172.21.15.72:0/565019489 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb540101110 0x7fb54019abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:59.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.565+0000 7fb546c00640 1 -- 172.21.15.72:0/565019489 >> 172.21.15.72:0/565019489 conn(0x7fb5400fccd0 msgr2=0x7fb5400fe3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:59.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.565+0000 7fb546c00640 1 -- 172.21.15.72:0/565019489 shutdown_connections 2024-04-26T23:42:59.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.565+0000 7fb546c00640 1 -- 172.21.15.72:0/565019489 wait complete. 2024-04-26T23:42:59.586 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-26T23:42:59.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:59 smithi076 ceph-mon[32774]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:59.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:59 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:59.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:59 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/1515809777' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:59.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:59 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/2684157997' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:42:59.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:59 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:59.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:59 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:59.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:59 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:59.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:42:59 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:59.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.761+0000 7f992e600640 1 -- 172.21.15.72:0/3349336466 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9928075330 msgr2=0x7f9928075790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:59.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.761+0000 7f992e600640 1 --2- 172.21.15.72:0/3349336466 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9928075330 0x7f9928075790 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9914009900 tx=0x7f991402a480 comp rx=0 tx=0).stop 2024-04-26T23:42:59.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.761+0000 7f992e600640 1 -- 172.21.15.72:0/3349336466 shutdown_connections 2024-04-26T23:42:59.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.761+0000 7f992e600640 1 --2- 172.21.15.72:0/3349336466 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9928075330 0x7f9928075790 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:59.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.761+0000 7f992e600640 1 --2- 172.21.15.72:0/3349336466 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f992810a660 0x7f992810cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:59.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.761+0000 7f992e600640 1 --2- 172.21.15.72:0/3349336466 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9928076910 0x7f9928074d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:59.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.761+0000 7f992e600640 1 -- 172.21.15.72:0/3349336466 >> 172.21.15.72:0/3349336466 conn(0x7f99280fdd40 msgr2=0x7f9928100180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:59.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.761+0000 7f992e600640 1 -- 172.21.15.72:0/3349336466 shutdown_connections 2024-04-26T23:42:59.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.761+0000 7f992e600640 1 -- 172.21.15.72:0/3349336466 wait complete. 2024-04-26T23:42:59.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.762+0000 7f992e600640 1 Processor -- start 2024-04-26T23:42:59.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.762+0000 7f992e600640 1 -- start start 2024-04-26T23:42:59.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.763+0000 7f992e600640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9928075330 0x7f992819aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:59.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.763+0000 7f992e600640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9928076910 0x7f992819b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:59.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.763+0000 7f992e600640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f992810a660 0x7f99281a21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:59.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.763+0000 7f992e600640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f9928110550 con 0x7f9928076910 2024-04-26T23:42:59.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.763+0000 7f992e600640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f99281103d0 con 0x7f9928075330 2024-04-26T23:42:59.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.763+0000 7f992e600640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f99281106d0 con 0x7f992810a660 2024-04-26T23:42:59.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.763+0000 7f992d200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9928075330 0x7f992819aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:59.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.763+0000 7f9927e00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9928076910 0x7f992819b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:59.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.763+0000 7f992d200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9928075330 0x7f992819aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:33036/0 (socket says 172.21.15.72:33036) 2024-04-26T23:42:59.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.763+0000 7f992d200640 1 -- 172.21.15.72:0/615986246 learned_addr learned my addr 172.21.15.72:0/615986246 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:42:59.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.763+0000 7f992dc00640 1 --2- 172.21.15.72:0/615986246 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f992810a660 0x7f99281a21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:59.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.763+0000 7f992d200640 1 -- 172.21.15.72:0/615986246 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f992810a660 msgr2=0x7f99281a21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:59.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.763+0000 7f992d200640 1 --2- 172.21.15.72:0/615986246 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f992810a660 0x7f99281a21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:59.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.763+0000 7f992d200640 1 -- 172.21.15.72:0/615986246 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9928076910 msgr2=0x7f992819b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:59.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.763+0000 7f992d200640 1 --2- 172.21.15.72:0/615986246 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9928076910 0x7f992819b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:59.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.763+0000 7f992d200640 1 -- 172.21.15.72:0/615986246 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99281a2860 con 0x7f9928075330 2024-04-26T23:42:59.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.764+0000 7f992d200640 1 --2- 172.21.15.72:0/615986246 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9928075330 0x7f992819aca0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f991800eb90 tx=0x7f991800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:59.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.764+0000 7f9925600640 1 -- 172.21.15.72:0/615986246 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9918004d20 con 0x7f9928075330 2024-04-26T23:42:59.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.764+0000 7f9925600640 1 -- 172.21.15.72:0/615986246 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f991800ee90 con 0x7f9928075330 2024-04-26T23:42:59.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.764+0000 7f9925600640 1 -- 172.21.15.72:0/615986246 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9918010670 con 0x7f9928075330 2024-04-26T23:42:59.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.764+0000 7f992e600640 1 -- 172.21.15.72:0/615986246 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99281a2b50 con 0x7f9928075330 2024-04-26T23:42:59.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.764+0000 7f992e600640 1 -- 172.21.15.72:0/615986246 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99281a3050 con 0x7f9928075330 2024-04-26T23:42:59.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.765+0000 7f992e600640 1 -- 172.21.15.72:0/615986246 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99080052c0 con 0x7f9928075330 2024-04-26T23:42:59.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.767+0000 7f9925600640 1 -- 172.21.15.72:0/615986246 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9918005020 con 0x7f9928075330 2024-04-26T23:42:59.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.769+0000 7f9925600640 1 --2- 172.21.15.72:0/615986246 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f990c0a3bb0 0x7f990c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:42:59.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.769+0000 7f9925600640 1 -- 172.21.15.72:0/615986246 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f99180d0700 con 0x7f9928075330 2024-04-26T23:42:59.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.770+0000 7f9927e00640 1 --2- 172.21.15.72:0/615986246 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f990c0a3bb0 0x7f990c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:42:59.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.770+0000 7f9927e00640 1 --2- 172.21.15.72:0/615986246 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f990c0a3bb0 0x7f990c0a6070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9914031500 tx=0x7f9914005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:42:59.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.772+0000 7f9925600640 1 -- 172.21.15.72:0/615986246 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f991809d4e0 con 0x7f9928075330 2024-04-26T23:42:59.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:59 smithi072 ceph-mon[28184]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:59.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:59 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:59.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:59 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/1515809777' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:59.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:59 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/2684157997' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:42:59.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:59 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:59.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:59 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:59.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:59 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:59.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:42:59 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:59.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:59 smithi136 ceph-mon[33050]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:42:59.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:59 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:59.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:59 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/1515809777' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:42:59.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:59 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/2684157997' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:42:59.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:59 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:59.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:59 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:59.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:59 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:59.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:42:59 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:42:59.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.985+0000 7f992e600640 1 -- 172.21.15.72:0/615986246 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f99080055b0 con 0x7f9928075330 2024-04-26T23:42:59.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.985+0000 7f9925600640 1 -- 172.21.15.72:0/615986246 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) ==== 84+0+65 (secure 0 0 0) 0x7f991809cde0 con 0x7f9928075330 2024-04-26T23:42:59.989 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:42:59.989 INFO:teuthology.orchestra.run.smithi072.stdout:[ 2024-04-26T23:42:59.989 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-04-26T23:42:59.989 INFO:teuthology.orchestra.run.smithi072.stdout: "poolnum": 1, 2024-04-26T23:42:59.989 INFO:teuthology.orchestra.run.smithi072.stdout: "poolname": ".mgr" 2024-04-26T23:42:59.989 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-04-26T23:42:59.990 INFO:teuthology.orchestra.run.smithi072.stdout:] 2024-04-26T23:42:59.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.987+0000 7f992e600640 1 -- 172.21.15.72:0/615986246 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f990c0a3bb0 msgr2=0x7f990c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:59.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.987+0000 7f992e600640 1 --2- 172.21.15.72:0/615986246 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f990c0a3bb0 0x7f990c0a6070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9914031500 tx=0x7f9914005a20 comp rx=0 tx=0).stop 2024-04-26T23:42:59.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.987+0000 7f992e600640 1 -- 172.21.15.72:0/615986246 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9928075330 msgr2=0x7f992819aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:42:59.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.987+0000 7f992e600640 1 --2- 172.21.15.72:0/615986246 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9928075330 0x7f992819aca0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f991800eb90 tx=0x7f991800c6a0 comp rx=0 tx=0).stop 2024-04-26T23:42:59.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.988+0000 7f992e600640 1 -- 172.21.15.72:0/615986246 shutdown_connections 2024-04-26T23:42:59.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.988+0000 7f992e600640 1 --2- 172.21.15.72:0/615986246 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f990c0a3bb0 0x7f990c0a6070 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:59.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.988+0000 7f992e600640 1 --2- 172.21.15.72:0/615986246 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f992810a660 0x7f99281a21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:59.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.988+0000 7f992e600640 1 --2- 172.21.15.72:0/615986246 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9928076910 0x7f992819b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:59.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.988+0000 7f992e600640 1 --2- 172.21.15.72:0/615986246 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9928075330 0x7f992819aca0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:42:59.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.988+0000 7f992e600640 1 -- 172.21.15.72:0/615986246 >> 172.21.15.72:0/615986246 conn(0x7f99280fdd40 msgr2=0x7f99281065f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:42:59.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.988+0000 7f992e600640 1 -- 172.21.15.72:0/615986246 shutdown_connections 2024-04-26T23:42:59.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:42:59.988+0000 7f992e600640 1 -- 172.21.15.72:0/615986246 wait complete. 2024-04-26T23:43:00.009 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-26T23:43:00.009 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-26T23:43:00.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:00.192+0000 7f4c52600640 1 -- 172.21.15.72:0/3878943126 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4c4c108650 msgr2=0x7f4c4c10aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:00.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:00.192+0000 7f4c52600640 1 --2- 172.21.15.72:0/3878943126 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4c4c108650 0x7f4c4c10aaa0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4c44009be0 tx=0x7f4c4402a6c0 comp rx=0 tx=0).stop 2024-04-26T23:43:00.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:00.192+0000 7f4c52600640 1 -- 172.21.15.72:0/3878943126 shutdown_connections 2024-04-26T23:43:00.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:00.192+0000 7f4c52600640 1 --2- 172.21.15.72:0/3878943126 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4c4c108650 0x7f4c4c10aaa0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:00.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:00.192+0000 7f4c52600640 1 --2- 172.21.15.72:0/3878943126 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c4c105d20 0x7f4c4c108110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:00.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:00.192+0000 7f4c52600640 1 --2- 172.21.15.72:0/3878943126 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4c4c1033f0 0x7f4c4c1057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:00.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:00.192+0000 7f4c52600640 1 -- 172.21.15.72:0/3878943126 >> 172.21.15.72:0/3878943126 conn(0x7f4c4c0fcde0 msgr2=0x7f4c4c0ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:00.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:00.193+0000 7f4c52600640 1 -- 172.21.15.72:0/3878943126 shutdown_connections 2024-04-26T23:43:00.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:00.193+0000 7f4c52600640 1 -- 172.21.15.72:0/3878943126 wait complete. 2024-04-26T23:43:00.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:00.194+0000 7f4c52600640 1 Processor -- start 2024-04-26T23:43:00.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:00.194+0000 7f4c52600640 1 -- start start 2024-04-26T23:43:00.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:00.194+0000 7f4c52600640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4c4c1033f0 0x7f4c4c102d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:00.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:00.194+0000 7f4c52600640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c4c105d20 0x7f4c4c1013a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:00.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:00.194+0000 7f4c52600640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4c4c108650 0x7f4c4c1018e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:00.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:00.194+0000 7f4c52600640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f4c4c10e4a0 con 0x7f4c4c105d20 2024-04-26T23:43:00.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:00.194+0000 7f4c52600640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f4c4c10e320 con 0x7f4c4c1033f0 2024-04-26T23:43:00.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:00.194+0000 7f4c52600640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f4c4c10e620 con 0x7f4c4c108650 2024-04-26T23:43:00.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:00.194+0000 7f4c4b400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4c4c1033f0 0x7f4c4c102d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:00.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:00.195+0000 7f4c4b400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4c4c1033f0 0x7f4c4c102d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:33056/0 (socket says 172.21.15.72:33056) 2024-04-26T23:43:00.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:00.195+0000 7f4c4b400640 1 -- 172.21.15.72:0/495259989 learned_addr learned my addr 172.21.15.72:0/495259989 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:43:00.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:00.195+0000 7f4c4aa00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c4c105d20 0x7f4c4c1013a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:00.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:00.195+0000 7f4c4be00640 1 --2- 172.21.15.72:0/495259989 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4c4c108650 0x7f4c4c1018e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:00.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:00.195+0000 7f4c4b400640 1 -- 172.21.15.72:0/495259989 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4c4c108650 msgr2=0x7f4c4c1018e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:00.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:00.195+0000 7f4c4b400640 1 --2- 172.21.15.72:0/495259989 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4c4c108650 0x7f4c4c1018e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:00.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:00.195+0000 7f4c4b400640 1 -- 172.21.15.72:0/495259989 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c4c105d20 msgr2=0x7f4c4c1013a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:00.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:00.195+0000 7f4c4b400640 1 --2- 172.21.15.72:0/495259989 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c4c105d20 0x7f4c4c1013a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:00.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:00.195+0000 7f4c4b400640 1 -- 172.21.15.72:0/495259989 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c4c102170 con 0x7f4c4c1033f0 2024-04-26T23:43:00.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:00.195+0000 7f4c4b400640 1 --2- 172.21.15.72:0/495259989 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4c4c1033f0 0x7f4c4c102d60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4c3400e9c0 tx=0x7f4c3400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:00.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:00.196+0000 7f4c43e00640 1 -- 172.21.15.72:0/495259989 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c3400cde0 con 0x7f4c4c1033f0 2024-04-26T23:43:00.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:00.196+0000 7f4c52600640 1 -- 172.21.15.72:0/495259989 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c4c1a2cc0 con 0x7f4c4c1033f0 2024-04-26T23:43:00.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:00.196+0000 7f4c43e00640 1 -- 172.21.15.72:0/495259989 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c340035f0 con 0x7f4c4c1033f0 2024-04-26T23:43:00.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:00.196+0000 7f4c43e00640 1 -- 172.21.15.72:0/495259989 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c34010690 con 0x7f4c4c1033f0 2024-04-26T23:43:00.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:00.196+0000 7f4c52600640 1 -- 172.21.15.72:0/495259989 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c4c1a31c0 con 0x7f4c4c1033f0 2024-04-26T23:43:00.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:00.197+0000 7f4c52600640 1 -- 172.21.15.72:0/495259989 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c4c0fe8d0 con 0x7f4c4c1033f0 2024-04-26T23:43:00.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:00.199+0000 7f4c43e00640 1 -- 172.21.15.72:0/495259989 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f4c34004000 con 0x7f4c4c1033f0 2024-04-26T23:43:00.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:00.200+0000 7f4c43e00640 1 --2- 172.21.15.72:0/495259989 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f4c300a3bb0 0x7f4c300a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:00.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:00.200+0000 7f4c43e00640 1 -- 172.21.15.72:0/495259989 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f4c340d0340 con 0x7f4c4c1033f0 2024-04-26T23:43:00.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:00.200+0000 7f4c4aa00640 1 --2- 172.21.15.72:0/495259989 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f4c300a3bb0 0x7f4c300a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:00.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:00.201+0000 7f4c4aa00640 1 --2- 172.21.15.72:0/495259989 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f4c300a3bb0 0x7f4c300a6070 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c102ad0 tx=0x7f4c3c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:00.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:00.204+0000 7f4c43e00640 1 -- 172.21.15.72:0/495259989 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4c34009510 con 0x7f4c4c1033f0 2024-04-26T23:43:00.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:00.414+0000 7f4c52600640 1 -- 172.21.15.72:0/495259989 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f4c4c063530 con 0x7f4c4c1033f0 2024-04-26T23:43:00.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:00 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/565019489' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T23:43:00.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:00 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/615986246' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-26T23:43:00.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:00 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:43:00.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:00 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:43:00.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:00 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:43:00.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:00 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:43:00.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:00 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:43:00.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:00 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/565019489' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T23:43:00.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:00 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/615986246' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-26T23:43:00.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:00 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:43:00.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:00 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:43:00.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:00 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:43:00.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:00 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:43:00.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:00 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:43:00.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:00 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/565019489' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T23:43:00.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:00 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/615986246' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-26T23:43:00.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:00 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:43:00.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:00 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:43:00.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:00 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:43:00.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:00 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:43:00.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:00 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:43:01.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.423+0000 7f4c43e00640 1 -- 172.21.15.72:0/495259989 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) ==== 138+0+0 (secure 0 0 0) 0x7f4c3409d120 con 0x7f4c4c1033f0 2024-04-26T23:43:01.431 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_metadata' created 2024-04-26T23:43:01.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.429+0000 7f4c52600640 1 -- 172.21.15.72:0/495259989 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f4c300a3bb0 msgr2=0x7f4c300a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:01.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.429+0000 7f4c52600640 1 --2- 172.21.15.72:0/495259989 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f4c300a3bb0 0x7f4c300a6070 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c102ad0 tx=0x7f4c3c005f90 comp rx=0 tx=0).stop 2024-04-26T23:43:01.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.429+0000 7f4c52600640 1 -- 172.21.15.72:0/495259989 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4c4c1033f0 msgr2=0x7f4c4c102d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:01.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.429+0000 7f4c52600640 1 --2- 172.21.15.72:0/495259989 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4c4c1033f0 0x7f4c4c102d60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4c3400e9c0 tx=0x7f4c3400ee80 comp rx=0 tx=0).stop 2024-04-26T23:43:01.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.430+0000 7f4c52600640 1 -- 172.21.15.72:0/495259989 shutdown_connections 2024-04-26T23:43:01.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.430+0000 7f4c52600640 1 --2- 172.21.15.72:0/495259989 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f4c300a3bb0 0x7f4c300a6070 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:01.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.430+0000 7f4c52600640 1 --2- 172.21.15.72:0/495259989 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4c4c108650 0x7f4c4c1018e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:01.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.430+0000 7f4c52600640 1 --2- 172.21.15.72:0/495259989 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c4c105d20 0x7f4c4c1013a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:01.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.430+0000 7f4c52600640 1 --2- 172.21.15.72:0/495259989 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4c4c1033f0 0x7f4c4c102d60 secure :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4c3400e9c0 tx=0x7f4c3400ee80 comp rx=0 tx=0).stop 2024-04-26T23:43:01.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.430+0000 7f4c52600640 1 -- 172.21.15.72:0/495259989 >> 172.21.15.72:0/495259989 conn(0x7f4c4c0fcde0 msgr2=0x7f4c4c103db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:01.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.430+0000 7f4c52600640 1 -- 172.21.15.72:0/495259989 shutdown_connections 2024-04-26T23:43:01.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.430+0000 7f4c52600640 1 -- 172.21.15.72:0/495259989 wait complete. 2024-04-26T23:43:01.455 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-04-26T23:43:01.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.639+0000 7f07ff400640 1 -- 172.21.15.72:0/3608750301 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07f81033f0 msgr2=0x7f07f81057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:01.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.639+0000 7f07ff400640 1 --2- 172.21.15.72:0/3608750301 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07f81033f0 0x7f07f81057e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f07e8009900 tx=0x7f07e802a410 comp rx=0 tx=0).stop 2024-04-26T23:43:01.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.639+0000 7f07ff400640 1 -- 172.21.15.72:0/3608750301 shutdown_connections 2024-04-26T23:43:01.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.639+0000 7f07ff400640 1 --2- 172.21.15.72:0/3608750301 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f07f8108650 0x7f07f810aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:01.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.639+0000 7f07ff400640 1 --2- 172.21.15.72:0/3608750301 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f07f8105d20 0x7f07f8108110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:01.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.639+0000 7f07ff400640 1 --2- 172.21.15.72:0/3608750301 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07f81033f0 0x7f07f81057e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:01.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.639+0000 7f07ff400640 1 -- 172.21.15.72:0/3608750301 >> 172.21.15.72:0/3608750301 conn(0x7f07f80fcde0 msgr2=0x7f07f80ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:01.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.639+0000 7f07ff400640 1 -- 172.21.15.72:0/3608750301 shutdown_connections 2024-04-26T23:43:01.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.639+0000 7f07ff400640 1 -- 172.21.15.72:0/3608750301 wait complete. 2024-04-26T23:43:01.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.640+0000 7f07ff400640 1 Processor -- start 2024-04-26T23:43:01.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.640+0000 7f07ff400640 1 -- start start 2024-04-26T23:43:01.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.640+0000 7f07ff400640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07f81033f0 0x7f07f819ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:01.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.640+0000 7f07ff400640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f07f8105d20 0x7f07f819b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:01.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.640+0000 7f07ff400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f07f8108650 0x7f07f81a2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:01.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.640+0000 7f07ff400640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f07f810e4f0 con 0x7f07f81033f0 2024-04-26T23:43:01.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.640+0000 7f07ff400640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f07f810e370 con 0x7f07f8108650 2024-04-26T23:43:01.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.640+0000 7f07ff400640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f07f810e670 con 0x7f07f8105d20 2024-04-26T23:43:01.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.641+0000 7f07fca00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07f81033f0 0x7f07f819ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:01.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.641+0000 7f07f7e00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f07f8105d20 0x7f07f819b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:01.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.641+0000 7f07fd400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f07f8108650 0x7f07f81a2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:01.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.641+0000 7f07fca00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07f81033f0 0x7f07f819ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48598/0 (socket says 172.21.15.72:48598) 2024-04-26T23:43:01.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.641+0000 7f07fca00640 1 -- 172.21.15.72:0/1329917052 learned_addr learned my addr 172.21.15.72:0/1329917052 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:43:01.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.641+0000 7f07fd400640 1 -- 172.21.15.72:0/1329917052 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f07f8105d20 msgr2=0x7f07f819b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:01.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.641+0000 7f07fd400640 1 --2- 172.21.15.72:0/1329917052 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f07f8105d20 0x7f07f819b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:01.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.641+0000 7f07fd400640 1 -- 172.21.15.72:0/1329917052 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07f81033f0 msgr2=0x7f07f819ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:01.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.641+0000 7f07fd400640 1 --2- 172.21.15.72:0/1329917052 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07f81033f0 0x7f07f819ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:01.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.641+0000 7f07fd400640 1 -- 172.21.15.72:0/1329917052 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07f81a2a00 con 0x7f07f8108650 2024-04-26T23:43:01.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.641+0000 7f07fd400640 1 --2- 172.21.15.72:0/1329917052 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f07f8108650 0x7f07f81a2390 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f07f0009fd0 tx=0x7f07f000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:01.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.642+0000 7f07f5600640 1 -- 172.21.15.72:0/1329917052 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07f000ce20 con 0x7f07f8108650 2024-04-26T23:43:01.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.642+0000 7f07ff400640 1 -- 172.21.15.72:0/1329917052 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07f81a2cf0 con 0x7f07f8108650 2024-04-26T23:43:01.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.642+0000 7f07f5600640 1 -- 172.21.15.72:0/1329917052 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07f0005020 con 0x7f07f8108650 2024-04-26T23:43:01.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.642+0000 7f07f5600640 1 -- 172.21.15.72:0/1329917052 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07f0010690 con 0x7f07f8108650 2024-04-26T23:43:01.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.642+0000 7f07ff400640 1 -- 172.21.15.72:0/1329917052 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07f81a31e0 con 0x7f07f8108650 2024-04-26T23:43:01.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.643+0000 7f07ff400640 1 -- 172.21.15.72:0/1329917052 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07f810a1a0 con 0x7f07f8108650 2024-04-26T23:43:01.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.645+0000 7f07f5600640 1 -- 172.21.15.72:0/1329917052 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f07f00054d0 con 0x7f07f8108650 2024-04-26T23:43:01.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.646+0000 7f07f5600640 1 --2- 172.21.15.72:0/1329917052 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f07dc0a3bb0 0x7f07dc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:01.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.647+0000 7f07f5600640 1 -- 172.21.15.72:0/1329917052 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7497+0+0 (secure 0 0 0) 0x7f07f00d0830 con 0x7f07f8108650 2024-04-26T23:43:01.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.647+0000 7f07fca00640 1 --2- 172.21.15.72:0/1329917052 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f07dc0a3bb0 0x7f07dc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:01.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.647+0000 7f07fca00640 1 --2- 172.21.15.72:0/1329917052 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f07dc0a3bb0 0x7f07dc0a6070 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f07e80096c0 tx=0x7f07e8005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:01.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.650+0000 7f07f5600640 1 -- 172.21.15.72:0/1329917052 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f07f009d4a0 con 0x7f07f8108650 2024-04-26T23:43:01.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:01 smithi076 ceph-mon[32774]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:43:01.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:01 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/495259989' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T23:43:01.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:01 smithi076 ceph-mon[32774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T23:43:01.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:01 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:43:01.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:01 smithi072 ceph-mon[28184]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:43:01.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:01 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/495259989' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T23:43:01.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:01 smithi072 ceph-mon[28184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T23:43:01.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:01 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:43:01.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:01 smithi136 ceph-mon[33050]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:43:01.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:01 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/495259989' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T23:43:01.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:01 smithi136 ceph-mon[33050]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T23:43:01.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:01 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:43:01.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:01.858+0000 7f07ff400640 1 -- 172.21.15.72:0/1329917052 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7f07f8063530 con 0x7f07f8108650 2024-04-26T23:43:02.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.429+0000 7f07f5600640 1 -- 172.21.15.72:0/1329917052 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v83) ==== 171+0+0 (secure 0 0 0) 0x7f07f009cda0 con 0x7f07f8108650 2024-04-26T23:43:02.441 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_data' created 2024-04-26T23:43:02.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.437+0000 7f07ff400640 1 -- 172.21.15.72:0/1329917052 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f07dc0a3bb0 msgr2=0x7f07dc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:02.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.437+0000 7f07ff400640 1 --2- 172.21.15.72:0/1329917052 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f07dc0a3bb0 0x7f07dc0a6070 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f07e80096c0 tx=0x7f07e8005a20 comp rx=0 tx=0).stop 2024-04-26T23:43:02.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.437+0000 7f07ff400640 1 -- 172.21.15.72:0/1329917052 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f07f8108650 msgr2=0x7f07f81a2390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:02.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.437+0000 7f07ff400640 1 --2- 172.21.15.72:0/1329917052 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f07f8108650 0x7f07f81a2390 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f07f0009fd0 tx=0x7f07f000ef10 comp rx=0 tx=0).stop 2024-04-26T23:43:02.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.437+0000 7f07ff400640 1 -- 172.21.15.72:0/1329917052 shutdown_connections 2024-04-26T23:43:02.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.437+0000 7f07ff400640 1 --2- 172.21.15.72:0/1329917052 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f07dc0a3bb0 0x7f07dc0a6070 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:02.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.437+0000 7f07ff400640 1 --2- 172.21.15.72:0/1329917052 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f07f8108650 0x7f07f81a2390 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:02.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.437+0000 7f07ff400640 1 --2- 172.21.15.72:0/1329917052 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f07f8105d20 0x7f07f819b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:02.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.437+0000 7f07ff400640 1 --2- 172.21.15.72:0/1329917052 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07f81033f0 0x7f07f819ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:02.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.437+0000 7f07ff400640 1 -- 172.21.15.72:0/1329917052 >> 172.21.15.72:0/1329917052 conn(0x7f07f80fcde0 msgr2=0x7f07f8102ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:02.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.438+0000 7f07ff400640 1 -- 172.21.15.72:0/1329917052 shutdown_connections 2024-04-26T23:43:02.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.438+0000 7f07ff400640 1 -- 172.21.15.72:0/1329917052 wait complete. 2024-04-26T23:43:02.482 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-26T23:43:02.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.652+0000 7f3a33400640 1 -- 172.21.15.72:0/3875311902 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a2c104bb0 msgr2=0x7f3a2c104fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:02.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.652+0000 7f3a33400640 1 --2- 172.21.15.72:0/3875311902 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a2c104bb0 0x7f3a2c104fd0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3a20009900 tx=0x7f3a2002a410 comp rx=0 tx=0).stop 2024-04-26T23:43:02.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.652+0000 7f3a33400640 1 -- 172.21.15.72:0/3875311902 shutdown_connections 2024-04-26T23:43:02.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.652+0000 7f3a33400640 1 --2- 172.21.15.72:0/3875311902 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a2c106760 0x7f3a2c10cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:02.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.652+0000 7f3a33400640 1 --2- 172.21.15.72:0/3875311902 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a2c105da0 0x7f3a2c106220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:02.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.652+0000 7f3a33400640 1 --2- 172.21.15.72:0/3875311902 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a2c104bb0 0x7f3a2c104fd0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:02.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.652+0000 7f3a33400640 1 -- 172.21.15.72:0/3875311902 >> 172.21.15.72:0/3875311902 conn(0x7f3a2c100120 msgr2=0x7f3a2c102560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:02.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.652+0000 7f3a33400640 1 -- 172.21.15.72:0/3875311902 shutdown_connections 2024-04-26T23:43:02.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.652+0000 7f3a33400640 1 -- 172.21.15.72:0/3875311902 wait complete. 2024-04-26T23:43:02.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.653+0000 7f3a33400640 1 Processor -- start 2024-04-26T23:43:02.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.653+0000 7f3a33400640 1 -- start start 2024-04-26T23:43:02.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.654+0000 7f3a33400640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a2c104bb0 0x7f3a2c19ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:02.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.654+0000 7f3a33400640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a2c105da0 0x7f3a2c19b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:02.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.654+0000 7f3a33400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a2c106760 0x7f3a2c1a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:02.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.654+0000 7f3a33400640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f3a2c110720 con 0x7f3a2c104bb0 2024-04-26T23:43:02.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.654+0000 7f3a33400640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3a2c1105a0 con 0x7f3a2c106760 2024-04-26T23:43:02.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.654+0000 7f3a33400640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f3a2c1108a0 con 0x7f3a2c105da0 2024-04-26T23:43:02.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.654+0000 7f3a30a00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a2c104bb0 0x7f3a2c19ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:02.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.654+0000 7f3a30a00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a2c104bb0 0x7f3a2c19ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48620/0 (socket says 172.21.15.72:48620) 2024-04-26T23:43:02.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.654+0000 7f3a30a00640 1 -- 172.21.15.72:0/107677120 learned_addr learned my addr 172.21.15.72:0/107677120 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:43:02.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.654+0000 7f3a2be00640 1 --2- 172.21.15.72:0/107677120 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a2c105da0 0x7f3a2c19b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:02.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.654+0000 7f3a31400640 1 --2- 172.21.15.72:0/107677120 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a2c106760 0x7f3a2c1a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:02.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.654+0000 7f3a2be00640 1 -- 172.21.15.72:0/107677120 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a2c106760 msgr2=0x7f3a2c1a2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:02.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.654+0000 7f3a2be00640 1 --2- 172.21.15.72:0/107677120 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a2c106760 0x7f3a2c1a2410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:02.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.654+0000 7f3a2be00640 1 -- 172.21.15.72:0/107677120 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a2c104bb0 msgr2=0x7f3a2c19ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:02.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.654+0000 7f3a2be00640 1 --2- 172.21.15.72:0/107677120 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a2c104bb0 0x7f3a2c19ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:02.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.655+0000 7f3a2be00640 1 -- 172.21.15.72:0/107677120 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a2c1a2a80 con 0x7f3a2c105da0 2024-04-26T23:43:02.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.655+0000 7f3a2be00640 1 --2- 172.21.15.72:0/107677120 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a2c105da0 0x7f3a2c19b3a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3a1800e9f0 tx=0x7f3a1800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:02.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.655+0000 7f3a29600640 1 -- 172.21.15.72:0/107677120 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a1800ccc0 con 0x7f3a2c105da0 2024-04-26T23:43:02.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.655+0000 7f3a33400640 1 -- 172.21.15.72:0/107677120 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a2c1a2d70 con 0x7f3a2c105da0 2024-04-26T23:43:02.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.655+0000 7f3a29600640 1 -- 172.21.15.72:0/107677120 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a18005020 con 0x7f3a2c105da0 2024-04-26T23:43:02.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.655+0000 7f3a29600640 1 -- 172.21.15.72:0/107677120 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a18004a00 con 0x7f3a2c105da0 2024-04-26T23:43:02.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.655+0000 7f3a33400640 1 -- 172.21.15.72:0/107677120 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a2c1a3210 con 0x7f3a2c105da0 2024-04-26T23:43:02.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.656+0000 7f3a33400640 1 -- 172.21.15.72:0/107677120 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a2c10bf40 con 0x7f3a2c105da0 2024-04-26T23:43:02.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.659+0000 7f3a29600640 1 -- 172.21.15.72:0/107677120 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3a18004ba0 con 0x7f3a2c105da0 2024-04-26T23:43:02.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.660+0000 7f3a29600640 1 --2- 172.21.15.72:0/107677120 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f3a140a3bb0 0x7f3a140a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:02.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.660+0000 7f3a29600640 1 -- 172.21.15.72:0/107677120 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7879+0+0 (secure 0 0 0) 0x7f3a180d0270 con 0x7f3a2c105da0 2024-04-26T23:43:02.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.661+0000 7f3a30a00640 1 --2- 172.21.15.72:0/107677120 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f3a140a3bb0 0x7f3a140a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:02.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.661+0000 7f3a30a00640 1 --2- 172.21.15.72:0/107677120 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f3a140a3bb0 0x7f3a140a6070 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3a20000c00 tx=0x7f3a20005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:02.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.665+0000 7f3a29600640 1 -- 172.21.15.72:0/107677120 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f3a1809cd60 con 0x7f3a2c105da0 2024-04-26T23:43:02.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:02 smithi072 ceph-mon[28184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-26T23:43:02.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:02 smithi072 ceph-mon[28184]: osdmap e82: 12 total, 12 up, 12 in 2024-04-26T23:43:02.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:02 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/1329917052' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-26T23:43:02.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:02 smithi072 ceph-mon[28184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-26T23:43:02.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:02 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:43:02.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:02 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:43:02.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:02 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:43:02.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:02 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:43:02.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:02 smithi136 ceph-mon[33050]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-26T23:43:02.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:02 smithi136 ceph-mon[33050]: osdmap e82: 12 total, 12 up, 12 in 2024-04-26T23:43:02.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:02 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/1329917052' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-26T23:43:02.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:02 smithi136 ceph-mon[33050]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-26T23:43:02.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:02 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:43:02.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:02 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:43:02.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:02 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:43:02.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:02 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:43:02.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:02 smithi076 ceph-mon[32774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-26T23:43:02.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:02 smithi076 ceph-mon[32774]: osdmap e82: 12 total, 12 up, 12 in 2024-04-26T23:43:02.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:02 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/1329917052' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-26T23:43:02.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:02 smithi076 ceph-mon[32774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-26T23:43:02.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:02 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:43:02.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:02 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:43:02.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:02 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:43:02.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:02 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:43:02.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:02.926+0000 7f3a33400640 1 -- 172.21.15.72:0/107677120 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f3a2c02dad0 con 0x7f3a2c105da0 2024-04-26T23:43:03.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.459+0000 7f3a29600640 1 -- 172.21.15.72:0/107677120 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f3a1809c660 con 0x7f3a2c105da0 2024-04-26T23:43:03.464 INFO:teuthology.orchestra.run.smithi072.stderr:new fs with metadata pool 2 and data pool 3 2024-04-26T23:43:03.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.463+0000 7f3a33400640 1 -- 172.21.15.72:0/107677120 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f3a140a3bb0 msgr2=0x7f3a140a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:03.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.463+0000 7f3a33400640 1 --2- 172.21.15.72:0/107677120 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f3a140a3bb0 0x7f3a140a6070 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3a20000c00 tx=0x7f3a20005a20 comp rx=0 tx=0).stop 2024-04-26T23:43:03.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.463+0000 7f3a33400640 1 -- 172.21.15.72:0/107677120 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a2c105da0 msgr2=0x7f3a2c19b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:03.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.463+0000 7f3a33400640 1 --2- 172.21.15.72:0/107677120 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a2c105da0 0x7f3a2c19b3a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3a1800e9f0 tx=0x7f3a1800eeb0 comp rx=0 tx=0).stop 2024-04-26T23:43:03.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.463+0000 7f3a33400640 1 -- 172.21.15.72:0/107677120 shutdown_connections 2024-04-26T23:43:03.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.463+0000 7f3a33400640 1 --2- 172.21.15.72:0/107677120 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f3a140a3bb0 0x7f3a140a6070 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:03.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.463+0000 7f3a33400640 1 --2- 172.21.15.72:0/107677120 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a2c106760 0x7f3a2c1a2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:03.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.463+0000 7f3a33400640 1 --2- 172.21.15.72:0/107677120 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a2c105da0 0x7f3a2c19b3a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:03.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.463+0000 7f3a33400640 1 --2- 172.21.15.72:0/107677120 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a2c104bb0 0x7f3a2c19ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:03.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.463+0000 7f3a33400640 1 -- 172.21.15.72:0/107677120 >> 172.21.15.72:0/107677120 conn(0x7f3a2c100120 msgr2=0x7f3a2c109110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:03.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.464+0000 7f3a33400640 1 -- 172.21.15.72:0/107677120 shutdown_connections 2024-04-26T23:43:03.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.464+0000 7f3a33400640 1 -- 172.21.15.72:0/107677120 wait complete. 2024-04-26T23:43:03.492 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-04-26T23:43:03.492 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-04-26T23:43:03.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.668+0000 7ffb79200640 1 -- 172.21.15.72:0/3455867535 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffb74104b70 msgr2=0x7ffb74104f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:03.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.668+0000 7ffb79200640 1 --2- 172.21.15.72:0/3455867535 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffb74104b70 0x7ffb74104f90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ffb64009860 tx=0x7ffb6402a340 comp rx=0 tx=0).stop 2024-04-26T23:43:03.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.668+0000 7ffb79200640 1 -- 172.21.15.72:0/3455867535 shutdown_connections 2024-04-26T23:43:03.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.668+0000 7ffb79200640 1 --2- 172.21.15.72:0/3455867535 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffb74106720 0x7ffb7410cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:03.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.668+0000 7ffb79200640 1 --2- 172.21.15.72:0/3455867535 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffb74105d60 0x7ffb741061e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:03.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.668+0000 7ffb79200640 1 --2- 172.21.15.72:0/3455867535 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffb74104b70 0x7ffb74104f90 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:03.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.668+0000 7ffb79200640 1 -- 172.21.15.72:0/3455867535 >> 172.21.15.72:0/3455867535 conn(0x7ffb74100120 msgr2=0x7ffb74102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:03.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.668+0000 7ffb79200640 1 -- 172.21.15.72:0/3455867535 shutdown_connections 2024-04-26T23:43:03.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.669+0000 7ffb79200640 1 -- 172.21.15.72:0/3455867535 wait complete. 2024-04-26T23:43:03.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.669+0000 7ffb79200640 1 Processor -- start 2024-04-26T23:43:03.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.670+0000 7ffb79200640 1 -- start start 2024-04-26T23:43:03.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.670+0000 7ffb79200640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffb74104b70 0x7ffb7419add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:03.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.670+0000 7ffb79200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffb74105d60 0x7ffb7419b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:03.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.670+0000 7ffb79200640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffb74106720 0x7ffb741a2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:03.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.670+0000 7ffb79200640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7ffb74110690 con 0x7ffb74106720 2024-04-26T23:43:03.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.670+0000 7ffb79200640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ffb74110510 con 0x7ffb74105d60 2024-04-26T23:43:03.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.670+0000 7ffb79200640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ffb74110810 con 0x7ffb74104b70 2024-04-26T23:43:03.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.670+0000 7ffb71400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffb74105d60 0x7ffb7419b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:03.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.670+0000 7ffb71400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffb74105d60 0x7ffb7419b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:33092/0 (socket says 172.21.15.72:33092) 2024-04-26T23:43:03.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.670+0000 7ffb72800640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffb74106720 0x7ffb741a2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:03.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.670+0000 7ffb71400640 1 -- 172.21.15.72:0/337896062 learned_addr learned my addr 172.21.15.72:0/337896062 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:43:03.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.670+0000 7ffb71e00640 1 --2- 172.21.15.72:0/337896062 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffb74104b70 0x7ffb7419add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:03.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.670+0000 7ffb71400640 1 -- 172.21.15.72:0/337896062 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffb74104b70 msgr2=0x7ffb7419add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:03.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.670+0000 7ffb71400640 1 --2- 172.21.15.72:0/337896062 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffb74104b70 0x7ffb7419add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:03.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.671+0000 7ffb71400640 1 -- 172.21.15.72:0/337896062 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffb74106720 msgr2=0x7ffb741a2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:03.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.671+0000 7ffb71400640 1 --2- 172.21.15.72:0/337896062 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffb74106720 0x7ffb741a2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:03.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.671+0000 7ffb71400640 1 -- 172.21.15.72:0/337896062 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb741a29f0 con 0x7ffb74105d60 2024-04-26T23:43:03.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.671+0000 7ffb71400640 1 --2- 172.21.15.72:0/337896062 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffb74105d60 0x7ffb7419b310 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ffb6000e9f0 tx=0x7ffb6000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:03.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.671+0000 7ffb6aa00640 1 -- 172.21.15.72:0/337896062 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb6000cd80 con 0x7ffb74105d60 2024-04-26T23:43:03.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.671+0000 7ffb79200640 1 -- 172.21.15.72:0/337896062 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb741a2ce0 con 0x7ffb74105d60 2024-04-26T23:43:03.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.671+0000 7ffb6aa00640 1 -- 172.21.15.72:0/337896062 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffb600054d0 con 0x7ffb74105d60 2024-04-26T23:43:03.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.671+0000 7ffb6aa00640 1 -- 172.21.15.72:0/337896062 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb60010690 con 0x7ffb74105d60 2024-04-26T23:43:03.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.671+0000 7ffb79200640 1 -- 172.21.15.72:0/337896062 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb741a3180 con 0x7ffb74105d60 2024-04-26T23:43:03.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.672+0000 7ffb79200640 1 -- 172.21.15.72:0/337896062 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb7410c000 con 0x7ffb74105d60 2024-04-26T23:43:03.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.675+0000 7ffb6aa00640 1 -- 172.21.15.72:0/337896062 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ffb60004e90 con 0x7ffb74105d60 2024-04-26T23:43:03.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.676+0000 7ffb6aa00640 1 --2- 172.21.15.72:0/337896062 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ffb580a3bb0 0x7ffb580a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:03.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.676+0000 7ffb6aa00640 1 -- 172.21.15.72:0/337896062 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7ffb600d0730 con 0x7ffb74105d60 2024-04-26T23:43:03.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.677+0000 7ffb71e00640 1 --2- 172.21.15.72:0/337896062 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ffb580a3bb0 0x7ffb580a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:03.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.677+0000 7ffb71e00640 1 --2- 172.21.15.72:0/337896062 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ffb580a3bb0 0x7ffb580a6070 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ffb64009830 tx=0x7ffb64030450 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:03.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.681+0000 7ffb6aa00640 1 -- 172.21.15.72:0/337896062 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ffb6009d1c0 con 0x7ffb74105d60 2024-04-26T23:43:03.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:03 smithi072 ceph-mon[28184]: pgmap v314: 33 pgs: 2 creating+peering, 1 active+clean, 30 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:43:03.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:03 smithi072 ceph-mon[28184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-26T23:43:03.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:03 smithi072 ceph-mon[28184]: osdmap e83: 12 total, 12 up, 12 in 2024-04-26T23:43:03.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:03 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/107677120' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T23:43:03.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:03 smithi072 ceph-mon[28184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T23:43:03.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:03 smithi072 ceph-mon[28184]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-26T23:43:03.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:03 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:43:03.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:03 smithi072 ceph-mon[28184]: osdmap e84: 12 total, 12 up, 12 in 2024-04-26T23:43:03.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:03 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:43:03.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:03 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:43:03.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:03 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:43:03.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:03 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:43:03.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:03 smithi072 ceph-mon[28184]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T23:43:03.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:03 smithi136 ceph-mon[33050]: pgmap v314: 33 pgs: 2 creating+peering, 1 active+clean, 30 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:43:03.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:03 smithi136 ceph-mon[33050]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-26T23:43:03.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:03 smithi136 ceph-mon[33050]: osdmap e83: 12 total, 12 up, 12 in 2024-04-26T23:43:03.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:03 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/107677120' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T23:43:03.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:03 smithi136 ceph-mon[33050]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T23:43:03.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:03 smithi136 ceph-mon[33050]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-26T23:43:03.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:03 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:43:03.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:03 smithi136 ceph-mon[33050]: osdmap e84: 12 total, 12 up, 12 in 2024-04-26T23:43:03.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:03 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:43:03.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:03 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:43:03.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:03 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:43:03.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:03 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:43:03.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:03 smithi136 ceph-mon[33050]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T23:43:03.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:03.891+0000 7ffb79200640 1 -- 172.21.15.72:0/337896062 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7ffb7402dad0 con 0x7ffb74105d60 2024-04-26T23:43:03.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:03 smithi076 ceph-mon[32774]: pgmap v314: 33 pgs: 2 creating+peering, 1 active+clean, 30 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:43:03.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:03 smithi076 ceph-mon[32774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-26T23:43:03.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:03 smithi076 ceph-mon[32774]: osdmap e83: 12 total, 12 up, 12 in 2024-04-26T23:43:03.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:03 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/107677120' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T23:43:03.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:03 smithi076 ceph-mon[32774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T23:43:03.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:03 smithi076 ceph-mon[32774]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-26T23:43:03.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:03 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:43:03.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:03 smithi076 ceph-mon[32774]: osdmap e84: 12 total, 12 up, 12 in 2024-04-26T23:43:03.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:03 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:43:03.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:03 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:43:03.930 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:03 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:43:03.930 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:03 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:43:03.930 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:03 smithi076 ceph-mon[32774]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T23:43:04.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.275+0000 7ffb6aa00640 1 -- 172.21.15.72:0/337896062 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v85) ==== 157+0+0 (secure 0 0 0) 0x7ffb6009cac0 con 0x7ffb74105d60 2024-04-26T23:43:04.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.279+0000 7ffb79200640 1 -- 172.21.15.72:0/337896062 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ffb580a3bb0 msgr2=0x7ffb580a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:04.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.279+0000 7ffb79200640 1 --2- 172.21.15.72:0/337896062 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ffb580a3bb0 0x7ffb580a6070 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ffb64009830 tx=0x7ffb64030450 comp rx=0 tx=0).stop 2024-04-26T23:43:04.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.279+0000 7ffb79200640 1 -- 172.21.15.72:0/337896062 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffb74105d60 msgr2=0x7ffb7419b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:04.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.279+0000 7ffb79200640 1 --2- 172.21.15.72:0/337896062 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffb74105d60 0x7ffb7419b310 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ffb6000e9f0 tx=0x7ffb6000edf0 comp rx=0 tx=0).stop 2024-04-26T23:43:04.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.279+0000 7ffb79200640 1 -- 172.21.15.72:0/337896062 shutdown_connections 2024-04-26T23:43:04.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.279+0000 7ffb79200640 1 --2- 172.21.15.72:0/337896062 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ffb580a3bb0 0x7ffb580a6070 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:04.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.279+0000 7ffb79200640 1 --2- 172.21.15.72:0/337896062 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffb74105d60 0x7ffb7419b310 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:04.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.279+0000 7ffb79200640 1 --2- 172.21.15.72:0/337896062 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffb74106720 0x7ffb741a2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:04.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.279+0000 7ffb79200640 1 --2- 172.21.15.72:0/337896062 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffb74104b70 0x7ffb7419add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:04.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.279+0000 7ffb79200640 1 -- 172.21.15.72:0/337896062 >> 172.21.15.72:0/337896062 conn(0x7ffb74100120 msgr2=0x7ffb741091d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:04.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.279+0000 7ffb79200640 1 -- 172.21.15.72:0/337896062 shutdown_connections 2024-04-26T23:43:04.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.279+0000 7ffb79200640 1 -- 172.21.15.72:0/337896062 wait complete. 2024-04-26T23:43:04.304 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-04-26T23:43:04.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.519+0000 7fe4f2800640 1 -- 172.21.15.72:0/1063166447 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe4ec076910 msgr2=0x7fe4ec074d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:04.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.519+0000 7fe4f2800640 1 --2- 172.21.15.72:0/1063166447 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe4ec076910 0x7fe4ec074d60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe4e4009ca0 tx=0x7fe4e402a950 comp rx=0 tx=0).stop 2024-04-26T23:43:04.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.519+0000 7fe4f2800640 1 -- 172.21.15.72:0/1063166447 shutdown_connections 2024-04-26T23:43:04.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.519+0000 7fe4f2800640 1 --2- 172.21.15.72:0/1063166447 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe4ec10a810 0x7fe4ec10ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:04.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.519+0000 7fe4f2800640 1 --2- 172.21.15.72:0/1063166447 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe4ec075330 0x7fe4ec075790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:04.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.519+0000 7fe4f2800640 1 --2- 172.21.15.72:0/1063166447 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe4ec076910 0x7fe4ec074d60 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:04.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.520+0000 7fe4f2800640 1 -- 172.21.15.72:0/1063166447 >> 172.21.15.72:0/1063166447 conn(0x7fe4ec0fdf10 msgr2=0x7fe4ec100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:04.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.520+0000 7fe4f2800640 1 -- 172.21.15.72:0/1063166447 shutdown_connections 2024-04-26T23:43:04.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.520+0000 7fe4f2800640 1 -- 172.21.15.72:0/1063166447 wait complete. 2024-04-26T23:43:04.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.521+0000 7fe4f2800640 1 Processor -- start 2024-04-26T23:43:04.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.521+0000 7fe4f2800640 1 -- start start 2024-04-26T23:43:04.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.521+0000 7fe4f2800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe4ec075330 0x7fe4ec19adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:04.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.521+0000 7fe4f2800640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe4ec076910 0x7fe4ec19b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:04.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.521+0000 7fe4f2800640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe4ec10a810 0x7fe4ec1a2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:04.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.521+0000 7fe4f2800640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fe4ec1106f0 con 0x7fe4ec076910 2024-04-26T23:43:04.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.521+0000 7fe4f2800640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe4ec110570 con 0x7fe4ec075330 2024-04-26T23:43:04.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.521+0000 7fe4f2800640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fe4ec110870 con 0x7fe4ec10a810 2024-04-26T23:43:04.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.521+0000 7fe4eaa00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe4ec076910 0x7fe4ec19b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:04.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.521+0000 7fe4ebe00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe4ec10a810 0x7fe4ec1a2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:04.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.522+0000 7fe4eb400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe4ec075330 0x7fe4ec19adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:04.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.522+0000 7fe4eaa00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe4ec076910 0x7fe4ec19b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48648/0 (socket says 172.21.15.72:48648) 2024-04-26T23:43:04.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.522+0000 7fe4ebe00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe4ec10a810 0x7fe4ec1a2370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.72:44908/0 (socket says 172.21.15.72:44908) 2024-04-26T23:43:04.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.522+0000 7fe4eaa00640 1 -- 172.21.15.72:0/999799133 learned_addr learned my addr 172.21.15.72:0/999799133 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:43:04.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.522+0000 7fe4ebe00640 1 -- 172.21.15.72:0/999799133 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe4ec075330 msgr2=0x7fe4ec19adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:04.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.522+0000 7fe4ebe00640 1 --2- 172.21.15.72:0/999799133 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe4ec075330 0x7fe4ec19adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:04.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.522+0000 7fe4ebe00640 1 -- 172.21.15.72:0/999799133 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe4ec076910 msgr2=0x7fe4ec19b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:04.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.522+0000 7fe4ebe00640 1 --2- 172.21.15.72:0/999799133 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe4ec076910 0x7fe4ec19b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:04.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.522+0000 7fe4ebe00640 1 -- 172.21.15.72:0/999799133 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4ec1a29e0 con 0x7fe4ec10a810 2024-04-26T23:43:04.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.522+0000 7fe4eaa00640 1 --2- 172.21.15.72:0/999799133 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe4ec076910 0x7fe4ec19b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:43:04.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.522+0000 7fe4ebe00640 1 --2- 172.21.15.72:0/999799133 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe4ec10a810 0x7fe4ec1a2370 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe4d400e970 tx=0x7fe4d400ee30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:04.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.523+0000 7fe4e3e00640 1 -- 172.21.15.72:0/999799133 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d400ccc0 con 0x7fe4ec10a810 2024-04-26T23:43:04.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.523+0000 7fe4e3e00640 1 -- 172.21.15.72:0/999799133 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4d40035f0 con 0x7fe4ec10a810 2024-04-26T23:43:04.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.523+0000 7fe4f2800640 1 -- 172.21.15.72:0/999799133 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4ec1a2cd0 con 0x7fe4ec10a810 2024-04-26T23:43:04.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.523+0000 7fe4e3e00640 1 -- 172.21.15.72:0/999799133 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d400ce60 con 0x7fe4ec10a810 2024-04-26T23:43:04.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.523+0000 7fe4f2800640 1 -- 172.21.15.72:0/999799133 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4ec1a31d0 con 0x7fe4ec10a810 2024-04-26T23:43:04.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.524+0000 7fe4f2800640 1 -- 172.21.15.72:0/999799133 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4ec1095d0 con 0x7fe4ec10a810 2024-04-26T23:43:04.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.526+0000 7fe4e3e00640 1 -- 172.21.15.72:0/999799133 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe4d4004020 con 0x7fe4ec10a810 2024-04-26T23:43:04.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.527+0000 7fe4e3e00640 1 --2- 172.21.15.72:0/999799133 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fe4d00a3bb0 0x7fe4d00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:04.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.527+0000 7fe4e3e00640 1 -- 172.21.15.72:0/999799133 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8277+0+0 (secure 0 0 0) 0x7fe4d40d06b0 con 0x7fe4ec10a810 2024-04-26T23:43:04.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.527+0000 7fe4eb400640 1 --2- 172.21.15.72:0/999799133 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fe4d00a3bb0 0x7fe4d00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:04.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.528+0000 7fe4eb400640 1 --2- 172.21.15.72:0/999799133 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fe4d00a3bb0 0x7fe4d00a6070 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe4e4000f80 tx=0x7fe4e402f870 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:04.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.531+0000 7fe4e3e00640 1 -- 172.21.15.72:0/999799133 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe4d409d000 con 0x7fe4ec10a810 2024-04-26T23:43:04.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:04.734+0000 7fe4f2800640 1 -- 172.21.15.72:0/999799133 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fe4ec0635d0 con 0x7fe4ec10a810 2024-04-26T23:43:04.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:04 smithi072 ceph-mon[28184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-26T23:43:04.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:04 smithi072 ceph-mon[28184]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-26T23:43:04.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:04 smithi072 ceph-mon[28184]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-26T23:43:04.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:04 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/337896062' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T23:43:04.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:04 smithi072 ceph-mon[28184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T23:43:04.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:04 smithi072 ceph-mon[28184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-26T23:43:04.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:04 smithi072 ceph-mon[28184]: osdmap e85: 12 total, 12 up, 12 in 2024-04-26T23:43:04.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:04 smithi136 ceph-mon[33050]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-26T23:43:04.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:04 smithi136 ceph-mon[33050]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-26T23:43:04.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:04 smithi136 ceph-mon[33050]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-26T23:43:04.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:04 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/337896062' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T23:43:04.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:04 smithi136 ceph-mon[33050]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T23:43:04.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:04 smithi136 ceph-mon[33050]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-26T23:43:04.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:04 smithi136 ceph-mon[33050]: osdmap e85: 12 total, 12 up, 12 in 2024-04-26T23:43:04.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:04 smithi076 ceph-mon[32774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-26T23:43:04.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:04 smithi076 ceph-mon[32774]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-26T23:43:04.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:04 smithi076 ceph-mon[32774]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-26T23:43:04.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:04 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/337896062' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T23:43:04.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:04 smithi076 ceph-mon[32774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T23:43:04.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:04 smithi076 ceph-mon[32774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-26T23:43:04.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:04 smithi076 ceph-mon[32774]: osdmap e85: 12 total, 12 up, 12 in 2024-04-26T23:43:05.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:05 smithi072 ceph-mon[28184]: pgmap v318: 97 pgs: 2 creating+peering, 40 active+clean, 55 unknown; 578 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 4 op/s 2024-04-26T23:43:05.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:05 smithi072 ceph-mon[28184]: mds.? [v2:172.21.15.76:6838/3405829058,v1:172.21.15.76:6839/3405829058] up:active 2024-04-26T23:43:05.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:05 smithi072 ceph-mon[28184]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T23:43:05.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:05 smithi072 ceph-mon[28184]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T23:43:05.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:05 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/999799133' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T23:43:05.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:05 smithi072 ceph-mon[28184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T23:43:05.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:05 smithi072 ceph-mon[28184]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-26T23:43:05.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:05 smithi072 ceph-mon[28184]: Cluster is now healthy 2024-04-26T23:43:05.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:05 smithi136 ceph-mon[33050]: pgmap v318: 97 pgs: 2 creating+peering, 40 active+clean, 55 unknown; 578 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 4 op/s 2024-04-26T23:43:05.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:05 smithi136 ceph-mon[33050]: mds.? [v2:172.21.15.76:6838/3405829058,v1:172.21.15.76:6839/3405829058] up:active 2024-04-26T23:43:05.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:05 smithi136 ceph-mon[33050]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T23:43:05.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:05 smithi136 ceph-mon[33050]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T23:43:05.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:05 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/999799133' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T23:43:05.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:05 smithi136 ceph-mon[33050]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T23:43:05.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:05 smithi136 ceph-mon[33050]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-26T23:43:05.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:05 smithi136 ceph-mon[33050]: Cluster is now healthy 2024-04-26T23:43:05.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:05 smithi076 ceph-mon[32774]: pgmap v318: 97 pgs: 2 creating+peering, 40 active+clean, 55 unknown; 578 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 4 op/s 2024-04-26T23:43:05.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:05 smithi076 ceph-mon[32774]: mds.? [v2:172.21.15.76:6838/3405829058,v1:172.21.15.76:6839/3405829058] up:active 2024-04-26T23:43:05.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:05 smithi076 ceph-mon[32774]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T23:43:05.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:05 smithi076 ceph-mon[32774]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T23:43:05.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:05 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/999799133' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T23:43:05.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:05 smithi076 ceph-mon[32774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T23:43:05.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:05 smithi076 ceph-mon[32774]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-26T23:43:05.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:05 smithi076 ceph-mon[32774]: Cluster is now healthy 2024-04-26T23:43:06.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.512+0000 7fe4e3e00640 1 -- 172.21.15.72:0/999799133 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v87) ==== 228+0+0 (secure 0 0 0) 0x7fe4d409c900 con 0x7fe4ec10a810 2024-04-26T23:43:06.516 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_data_ec' created 2024-04-26T23:43:06.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.518+0000 7fe4f2800640 1 -- 172.21.15.72:0/999799133 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fe4d00a3bb0 msgr2=0x7fe4d00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:06.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.518+0000 7fe4f2800640 1 --2- 172.21.15.72:0/999799133 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fe4d00a3bb0 0x7fe4d00a6070 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe4e4000f80 tx=0x7fe4e402f870 comp rx=0 tx=0).stop 2024-04-26T23:43:06.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.518+0000 7fe4f2800640 1 -- 172.21.15.72:0/999799133 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe4ec10a810 msgr2=0x7fe4ec1a2370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:06.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.518+0000 7fe4f2800640 1 --2- 172.21.15.72:0/999799133 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe4ec10a810 0x7fe4ec1a2370 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe4d400e970 tx=0x7fe4d400ee30 comp rx=0 tx=0).stop 2024-04-26T23:43:06.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.519+0000 7fe4f2800640 1 -- 172.21.15.72:0/999799133 shutdown_connections 2024-04-26T23:43:06.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.519+0000 7fe4f2800640 1 --2- 172.21.15.72:0/999799133 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fe4d00a3bb0 0x7fe4d00a6070 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:06.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.519+0000 7fe4f2800640 1 --2- 172.21.15.72:0/999799133 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe4ec10a810 0x7fe4ec1a2370 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:06.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.519+0000 7fe4f2800640 1 --2- 172.21.15.72:0/999799133 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe4ec076910 0x7fe4ec19b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:06.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.519+0000 7fe4f2800640 1 --2- 172.21.15.72:0/999799133 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe4ec075330 0x7fe4ec19adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:06.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.519+0000 7fe4f2800640 1 -- 172.21.15.72:0/999799133 >> 172.21.15.72:0/999799133 conn(0x7fe4ec0fdf10 msgr2=0x7fe4ec1067a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:06.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.520+0000 7fe4f2800640 1 -- 172.21.15.72:0/999799133 shutdown_connections 2024-04-26T23:43:06.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.521+0000 7fe4f2800640 1 -- 172.21.15.72:0/999799133 wait complete. 2024-04-26T23:43:06.547 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-04-26T23:43:06.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.730+0000 7f1fef400640 1 -- 172.21.15.72:0/4181098631 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1fe8078530 msgr2=0x7f1fe8078990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:06.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.730+0000 7f1fef400640 1 --2- 172.21.15.72:0/4181098631 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1fe8078530 0x7f1fe8078990 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1fd4009900 tx=0x7f1fd402a3f0 comp rx=0 tx=0).stop 2024-04-26T23:43:06.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.730+0000 7f1fef400640 1 -- 172.21.15.72:0/4181098631 shutdown_connections 2024-04-26T23:43:06.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.730+0000 7f1fef400640 1 --2- 172.21.15.72:0/4181098631 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1fe8078ed0 0x7f1fe8079770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:06.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.730+0000 7f1fef400640 1 --2- 172.21.15.72:0/4181098631 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1fe8078530 0x7f1fe8078990 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:06.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.730+0000 7f1fef400640 1 --2- 172.21.15.72:0/4181098631 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1fe80772f0 0x7f1fe8077710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:06.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.730+0000 7f1fef400640 1 -- 172.21.15.72:0/4181098631 >> 172.21.15.72:0/4181098631 conn(0x7f1fe8100120 msgr2=0x7f1fe8102560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:06.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.730+0000 7f1fef400640 1 -- 172.21.15.72:0/4181098631 shutdown_connections 2024-04-26T23:43:06.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.730+0000 7f1fef400640 1 -- 172.21.15.72:0/4181098631 wait complete. 2024-04-26T23:43:06.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.731+0000 7f1fef400640 1 Processor -- start 2024-04-26T23:43:06.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.731+0000 7f1fef400640 1 -- start start 2024-04-26T23:43:06.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.732+0000 7f1fef400640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1fe80772f0 0x7f1fe819f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:06.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.732+0000 7f1fef400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1fe8078530 0x7f1fe819f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:06.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.732+0000 7f1fef400640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1fe8078ed0 0x7f1fe81a6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:06.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.732+0000 7f1fef400640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe8114b30 con 0x7f1fe80772f0 2024-04-26T23:43:06.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.732+0000 7f1fef400640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe81149b0 con 0x7f1fe8078530 2024-04-26T23:43:06.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.732+0000 7f1fef400640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe8114cb0 con 0x7f1fe8078ed0 2024-04-26T23:43:06.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.732+0000 7f1feca00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1fe80772f0 0x7f1fe819f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:06.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.732+0000 7f1feca00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1fe80772f0 0x7f1fe819f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48656/0 (socket says 172.21.15.72:48656) 2024-04-26T23:43:06.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.732+0000 7f1feca00640 1 -- 172.21.15.72:0/346012808 learned_addr learned my addr 172.21.15.72:0/346012808 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:43:06.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.732+0000 7f1fe7e00640 1 --2- 172.21.15.72:0/346012808 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1fe8078530 0x7f1fe819f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:06.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.732+0000 7f1fed400640 1 --2- 172.21.15.72:0/346012808 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1fe8078ed0 0x7f1fe81a6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:06.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.732+0000 7f1feca00640 1 -- 172.21.15.72:0/346012808 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1fe8078ed0 msgr2=0x7f1fe81a6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:06.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.732+0000 7f1feca00640 1 --2- 172.21.15.72:0/346012808 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1fe8078ed0 0x7f1fe81a6790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:06.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.732+0000 7f1feca00640 1 -- 172.21.15.72:0/346012808 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1fe8078530 msgr2=0x7f1fe819f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:06.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.732+0000 7f1feca00640 1 --2- 172.21.15.72:0/346012808 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1fe8078530 0x7f1fe819f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:06.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.732+0000 7f1feca00640 1 -- 172.21.15.72:0/346012808 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fe81a6e00 con 0x7f1fe80772f0 2024-04-26T23:43:06.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.733+0000 7f1feca00640 1 --2- 172.21.15.72:0/346012808 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1fe80772f0 0x7f1fe819f1e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc00ea60 tx=0x7f1fdc00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:06.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.733+0000 7f1fe5600640 1 -- 172.21.15.72:0/346012808 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc00cde0 con 0x7f1fe80772f0 2024-04-26T23:43:06.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.733+0000 7f1fef400640 1 -- 172.21.15.72:0/346012808 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fe81a70f0 con 0x7f1fe80772f0 2024-04-26T23:43:06.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.733+0000 7f1fe5600640 1 -- 172.21.15.72:0/346012808 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fdc0035f0 con 0x7f1fe80772f0 2024-04-26T23:43:06.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.733+0000 7f1fe5600640 1 -- 172.21.15.72:0/346012808 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc010690 con 0x7f1fe80772f0 2024-04-26T23:43:06.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.733+0000 7f1fef400640 1 -- 172.21.15.72:0/346012808 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fe81a7610 con 0x7f1fe80772f0 2024-04-26T23:43:06.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.734+0000 7f1fef400640 1 -- 172.21.15.72:0/346012808 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fe8111520 con 0x7f1fe80772f0 2024-04-26T23:43:06.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.737+0000 7f1fe5600640 1 -- 172.21.15.72:0/346012808 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f1fdc0037a0 con 0x7f1fe80772f0 2024-04-26T23:43:06.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.738+0000 7f1fe5600640 1 --2- 172.21.15.72:0/346012808 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f1fd00a3bb0 0x7f1fd00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:06.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.738+0000 7f1fe5600640 1 -- 172.21.15.72:0/346012808 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8770+0+0 (secure 0 0 0) 0x7f1fdc0d0b70 con 0x7f1fe80772f0 2024-04-26T23:43:06.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.738+0000 7f1fe7e00640 1 --2- 172.21.15.72:0/346012808 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f1fd00a3bb0 0x7f1fd00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:06.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.739+0000 7f1fe7e00640 1 --2- 172.21.15.72:0/346012808 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f1fd00a3bb0 0x7f1fd00a6070 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1fd4031500 tx=0x7f1fd4005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:06.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.742+0000 7f1fe5600640 1 -- 172.21.15.72:0/346012808 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1fdc054060 con 0x7f1fe80772f0 2024-04-26T23:43:06.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:06 smithi072 ceph-mon[28184]: osdmap e86: 12 total, 12 up, 12 in 2024-04-26T23:43:06.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:06 smithi136 ceph-mon[33050]: osdmap e86: 12 total, 12 up, 12 in 2024-04-26T23:43:06.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:06 smithi076 ceph-mon[32774]: osdmap e86: 12 total, 12 up, 12 in 2024-04-26T23:43:06.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:06.945+0000 7f1fef400640 1 -- 172.21.15.72:0/346012808 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f1fe802dad0 con 0x7f1fe80772f0 2024-04-26T23:43:07.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.507+0000 7f1fe5600640 1 -- 172.21.15.72:0/346012808 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v88) ==== 169+0+0 (secure 0 0 0) 0x7f1fdc09d2d0 con 0x7f1fe80772f0 2024-04-26T23:43:07.510 INFO:teuthology.orchestra.run.smithi072.stderr:set pool 4 allow_ec_overwrites to true 2024-04-26T23:43:07.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.515+0000 7f1fef400640 1 -- 172.21.15.72:0/346012808 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f1fd00a3bb0 msgr2=0x7f1fd00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:07.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.515+0000 7f1fef400640 1 --2- 172.21.15.72:0/346012808 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f1fd00a3bb0 0x7f1fd00a6070 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1fd4031500 tx=0x7f1fd4005a20 comp rx=0 tx=0).stop 2024-04-26T23:43:07.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.515+0000 7f1fef400640 1 -- 172.21.15.72:0/346012808 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1fe80772f0 msgr2=0x7f1fe819f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:07.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.515+0000 7f1fef400640 1 --2- 172.21.15.72:0/346012808 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1fe80772f0 0x7f1fe819f1e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc00ea60 tx=0x7f1fdc00ef20 comp rx=0 tx=0).stop 2024-04-26T23:43:07.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.515+0000 7f1fef400640 1 -- 172.21.15.72:0/346012808 shutdown_connections 2024-04-26T23:43:07.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.515+0000 7f1fef400640 1 --2- 172.21.15.72:0/346012808 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f1fd00a3bb0 0x7f1fd00a6070 secure :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1fd4031500 tx=0x7f1fd4005a20 comp rx=0 tx=0).stop 2024-04-26T23:43:07.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.515+0000 7f1fef400640 1 --2- 172.21.15.72:0/346012808 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1fe8078530 0x7f1fe819f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:07.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.515+0000 7f1fef400640 1 --2- 172.21.15.72:0/346012808 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1fe8078ed0 0x7f1fe81a6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:07.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.515+0000 7f1fef400640 1 --2- 172.21.15.72:0/346012808 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1fe80772f0 0x7f1fe819f1e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:07.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.515+0000 7f1fef400640 1 -- 172.21.15.72:0/346012808 >> 172.21.15.72:0/346012808 conn(0x7f1fe8100120 msgr2=0x7f1fe8103480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:07.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.517+0000 7f1fef400640 1 -- 172.21.15.72:0/346012808 shutdown_connections 2024-04-26T23:43:07.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.517+0000 7f1fef400640 1 -- 172.21.15.72:0/346012808 wait complete. 2024-04-26T23:43:07.557 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-04-26T23:43:07.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.730+0000 7fe479c00640 1 -- 172.21.15.72:0/4196209801 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe474105d80 msgr2=0x7fe474106200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:07.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.730+0000 7fe479c00640 1 --2- 172.21.15.72:0/4196209801 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe474105d80 0x7fe474106200 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe45c009900 tx=0x7fe45c02a3f0 comp rx=0 tx=0).stop 2024-04-26T23:43:07.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.730+0000 7fe479c00640 1 -- 172.21.15.72:0/4196209801 shutdown_connections 2024-04-26T23:43:07.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.730+0000 7fe479c00640 1 --2- 172.21.15.72:0/4196209801 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe474106740 0x7fe47410cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:07.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.730+0000 7fe479c00640 1 --2- 172.21.15.72:0/4196209801 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe474105d80 0x7fe474106200 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:07.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.730+0000 7fe479c00640 1 --2- 172.21.15.72:0/4196209801 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe474104b90 0x7fe474104fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:07.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.730+0000 7fe479c00640 1 -- 172.21.15.72:0/4196209801 >> 172.21.15.72:0/4196209801 conn(0x7fe474100120 msgr2=0x7fe474102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:07.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.730+0000 7fe479c00640 1 -- 172.21.15.72:0/4196209801 shutdown_connections 2024-04-26T23:43:07.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.730+0000 7fe479c00640 1 -- 172.21.15.72:0/4196209801 wait complete. 2024-04-26T23:43:07.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.731+0000 7fe479c00640 1 Processor -- start 2024-04-26T23:43:07.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.731+0000 7fe479c00640 1 -- start start 2024-04-26T23:43:07.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.731+0000 7fe479c00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe474104b90 0x7fe47419adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:07.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.731+0000 7fe479c00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe474105d80 0x7fe47419b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:07.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.731+0000 7fe479c00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe474106740 0x7fe4741a23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:07.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.731+0000 7fe479c00640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fe4741106a0 con 0x7fe474105d80 2024-04-26T23:43:07.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.732+0000 7fe479c00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe474110520 con 0x7fe474106740 2024-04-26T23:43:07.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.732+0000 7fe479c00640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fe474110820 con 0x7fe474104b90 2024-04-26T23:43:07.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.732+0000 7fe46a000640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe474105d80 0x7fe47419b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:07.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.732+0000 7fe46a000640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe474105d80 0x7fe47419b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43228/0 (socket says 172.21.15.72:43228) 2024-04-26T23:43:07.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.732+0000 7fe46a000640 1 -- 172.21.15.72:0/3420675397 learned_addr learned my addr 172.21.15.72:0/3420675397 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:43:07.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.732+0000 7fe472a00640 1 --2- 172.21.15.72:0/3420675397 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe474104b90 0x7fe47419adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:07.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.732+0000 7fe473400640 1 --2- 172.21.15.72:0/3420675397 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe474106740 0x7fe4741a23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:07.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.732+0000 7fe46a000640 1 -- 172.21.15.72:0/3420675397 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe474104b90 msgr2=0x7fe47419adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:07.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.732+0000 7fe46a000640 1 --2- 172.21.15.72:0/3420675397 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe474104b90 0x7fe47419adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:07.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.732+0000 7fe46a000640 1 -- 172.21.15.72:0/3420675397 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe474106740 msgr2=0x7fe4741a23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:07.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.732+0000 7fe46a000640 1 --2- 172.21.15.72:0/3420675397 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe474106740 0x7fe4741a23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:07.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.732+0000 7fe46a000640 1 -- 172.21.15.72:0/3420675397 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4741a2a10 con 0x7fe474105d80 2024-04-26T23:43:07.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.732+0000 7fe46a000640 1 --2- 172.21.15.72:0/3420675397 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe474105d80 0x7fe47419b330 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe45c005940 tx=0x7fe45c031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:07.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.733+0000 7fe46be00640 1 -- 172.21.15.72:0/3420675397 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe45c03f030 con 0x7fe474105d80 2024-04-26T23:43:07.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.733+0000 7fe46be00640 1 -- 172.21.15.72:0/3420675397 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe45c003aa0 con 0x7fe474105d80 2024-04-26T23:43:07.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.733+0000 7fe479c00640 1 -- 172.21.15.72:0/3420675397 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4741a2ca0 con 0x7fe474105d80 2024-04-26T23:43:07.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.733+0000 7fe46be00640 1 -- 172.21.15.72:0/3420675397 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe45c035070 con 0x7fe474105d80 2024-04-26T23:43:07.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.733+0000 7fe479c00640 1 -- 172.21.15.72:0/3420675397 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4741a3190 con 0x7fe474105d80 2024-04-26T23:43:07.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.734+0000 7fe479c00640 1 -- 172.21.15.72:0/3420675397 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe47410bf50 con 0x7fe474105d80 2024-04-26T23:43:07.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.736+0000 7fe46be00640 1 -- 172.21.15.72:0/3420675397 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe45c042030 con 0x7fe474105d80 2024-04-26T23:43:07.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.737+0000 7fe46be00640 1 --2- 172.21.15.72:0/3420675397 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fe4580a3c30 0x7fe4580a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:07.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.738+0000 7fe46be00640 1 -- 172.21.15.72:0/3420675397 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8770+0+0 (secure 0 0 0) 0x7fe45c0f0b90 con 0x7fe474105d80 2024-04-26T23:43:07.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.738+0000 7fe472a00640 1 --2- 172.21.15.72:0/3420675397 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fe4580a3c30 0x7fe4580a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:07.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.738+0000 7fe472a00640 1 --2- 172.21.15.72:0/3420675397 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fe4580a3c30 0x7fe4580a60f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe474105be0 tx=0x7fe464005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:07.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.742+0000 7fe46be00640 1 -- 172.21.15.72:0/3420675397 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe45c0bcca0 con 0x7fe474105d80 2024-04-26T23:43:07.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:07 smithi072 ceph-mon[28184]: pgmap v320: 97 pgs: 2 creating+peering, 87 active+clean, 8 unknown; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 11 op/s 2024-04-26T23:43:07.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:07 smithi072 ceph-mon[28184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-26T23:43:07.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:07 smithi072 ceph-mon[28184]: osdmap e87: 12 total, 12 up, 12 in 2024-04-26T23:43:07.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:07 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/346012808' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-26T23:43:07.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:07 smithi136 ceph-mon[33050]: pgmap v320: 97 pgs: 2 creating+peering, 87 active+clean, 8 unknown; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 11 op/s 2024-04-26T23:43:07.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:07 smithi136 ceph-mon[33050]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-26T23:43:07.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:07 smithi136 ceph-mon[33050]: osdmap e87: 12 total, 12 up, 12 in 2024-04-26T23:43:07.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:07 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/346012808' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-26T23:43:07.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:07 smithi076 ceph-mon[32774]: pgmap v320: 97 pgs: 2 creating+peering, 87 active+clean, 8 unknown; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 11 op/s 2024-04-26T23:43:07.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:07 smithi076 ceph-mon[32774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-26T23:43:07.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:07 smithi076 ceph-mon[32774]: osdmap e87: 12 total, 12 up, 12 in 2024-04-26T23:43:07.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:07 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/346012808' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-26T23:43:07.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:07.989+0000 7fe479c00640 1 -- 172.21.15.72:0/3420675397 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7fe474063530 con 0x7fe474105d80 2024-04-26T23:43:08.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.537+0000 7fe46be00640 1 -- 172.21.15.72:0/3420675397 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7fe45c0c0fa0 con 0x7fe474105d80 2024-04-26T23:43:08.540 INFO:teuthology.orchestra.run.smithi072.stderr:added data pool 4 to fsmap 2024-04-26T23:43:08.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.542+0000 7fe479c00640 1 -- 172.21.15.72:0/3420675397 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fe4580a3c30 msgr2=0x7fe4580a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:08.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.542+0000 7fe479c00640 1 --2- 172.21.15.72:0/3420675397 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fe4580a3c30 0x7fe4580a60f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe474105be0 tx=0x7fe464005f90 comp rx=0 tx=0).stop 2024-04-26T23:43:08.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.542+0000 7fe479c00640 1 -- 172.21.15.72:0/3420675397 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe474105d80 msgr2=0x7fe47419b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:08.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.542+0000 7fe479c00640 1 --2- 172.21.15.72:0/3420675397 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe474105d80 0x7fe47419b330 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe45c005940 tx=0x7fe45c031a50 comp rx=0 tx=0).stop 2024-04-26T23:43:08.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.542+0000 7fe479c00640 1 -- 172.21.15.72:0/3420675397 shutdown_connections 2024-04-26T23:43:08.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.542+0000 7fe479c00640 1 --2- 172.21.15.72:0/3420675397 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fe4580a3c30 0x7fe4580a60f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:08.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.542+0000 7fe479c00640 1 --2- 172.21.15.72:0/3420675397 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe474106740 0x7fe4741a23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:08.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.542+0000 7fe479c00640 1 --2- 172.21.15.72:0/3420675397 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe474105d80 0x7fe47419b330 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:08.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.542+0000 7fe479c00640 1 --2- 172.21.15.72:0/3420675397 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe474104b90 0x7fe47419adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:08.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.542+0000 7fe479c00640 1 -- 172.21.15.72:0/3420675397 >> 172.21.15.72:0/3420675397 conn(0x7fe474100120 msgr2=0x7fe474109120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:08.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.542+0000 7fe479c00640 1 -- 172.21.15.72:0/3420675397 shutdown_connections 2024-04-26T23:43:08.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.542+0000 7fe479c00640 1 -- 172.21.15.72:0/3420675397 wait complete. 2024-04-26T23:43:08.570 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:43:08.679 INFO:journalctl@ceph.mds.b.smithi076.stdout:Apr 26 23:43:08 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-b[62253]: 2024-04-26T23:43:08.377+0000 7ff94af66700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-26T23:43:08.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:08 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/346012808' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-26T23:43:08.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:08 smithi076 ceph-mon[32774]: osdmap e88: 12 total, 12 up, 12 in 2024-04-26T23:43:08.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:08 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/3420675397' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-26T23:43:08.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:08 smithi076 ceph-mon[32774]: osdmap e89: 12 total, 12 up, 12 in 2024-04-26T23:43:08.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.770+0000 7f02b3e00640 1 -- 172.21.15.72:0/2261304010 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02b4109f80 msgr2=0x7f02b410a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:08.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.770+0000 7f02b3e00640 1 --2- 172.21.15.72:0/2261304010 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02b4109f80 0x7f02b410a400 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f02a0009900 tx=0x7f02a002a3f0 comp rx=0 tx=0).stop 2024-04-26T23:43:08.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.770+0000 7f02b3e00640 1 -- 172.21.15.72:0/2261304010 shutdown_connections 2024-04-26T23:43:08.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.770+0000 7f02b3e00640 1 --2- 172.21.15.72:0/2261304010 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f02b410a940 0x7f02b41111d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:08.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.770+0000 7f02b3e00640 1 --2- 172.21.15.72:0/2261304010 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02b4109f80 0x7f02b410a400 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:08.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.770+0000 7f02b3e00640 1 --2- 172.21.15.72:0/2261304010 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f02b4108d90 0x7f02b41091b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:08.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.770+0000 7f02b3e00640 1 -- 172.21.15.72:0/2261304010 >> 172.21.15.72:0/2261304010 conn(0x7f02b4077d30 msgr2=0x7f02b407a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:08.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.770+0000 7f02b3e00640 1 -- 172.21.15.72:0/2261304010 shutdown_connections 2024-04-26T23:43:08.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.770+0000 7f02b3e00640 1 -- 172.21.15.72:0/2261304010 wait complete. 2024-04-26T23:43:08.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.771+0000 7f02b3e00640 1 Processor -- start 2024-04-26T23:43:08.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.772+0000 7f02b3e00640 1 -- start start 2024-04-26T23:43:08.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.772+0000 7f02b3e00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02b4108d90 0x7f02b419ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:08.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.772+0000 7f02b3e00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f02b4109f80 0x7f02b419f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:08.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.772+0000 7f02b3e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f02b410a940 0x7f02b41a6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:08.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.772+0000 7f02b3e00640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f02b41148f0 con 0x7f02b4108d90 2024-04-26T23:43:08.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.772+0000 7f02b3e00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f02b4114770 con 0x7f02b410a940 2024-04-26T23:43:08.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.772+0000 7f02b3e00640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f02b4114a70 con 0x7f02b4109f80 2024-04-26T23:43:08.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.772+0000 7f02b2a00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02b4108d90 0x7f02b419ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:08.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.772+0000 7f02b2a00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02b4108d90 0x7f02b419ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43250/0 (socket says 172.21.15.72:43250) 2024-04-26T23:43:08.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.772+0000 7f02b2a00640 1 -- 172.21.15.72:0/1027562616 learned_addr learned my addr 172.21.15.72:0/1027562616 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:43:08.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.772+0000 7f02b3400640 1 --2- 172.21.15.72:0/1027562616 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f02b410a940 0x7f02b41a6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:08.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.772+0000 7f02b2000640 1 --2- 172.21.15.72:0/1027562616 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f02b4109f80 0x7f02b419f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:08.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.772+0000 7f02b2a00640 1 -- 172.21.15.72:0/1027562616 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f02b4109f80 msgr2=0x7f02b419f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:08.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.772+0000 7f02b2a00640 1 --2- 172.21.15.72:0/1027562616 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f02b4109f80 0x7f02b419f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:08.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.772+0000 7f02b2a00640 1 -- 172.21.15.72:0/1027562616 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f02b410a940 msgr2=0x7f02b41a6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:08.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.772+0000 7f02b2a00640 1 --2- 172.21.15.72:0/1027562616 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f02b410a940 0x7f02b41a6510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:08.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.772+0000 7f02b2a00640 1 -- 172.21.15.72:0/1027562616 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02b41a6b80 con 0x7f02b4108d90 2024-04-26T23:43:08.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.773+0000 7f02b2a00640 1 --2- 172.21.15.72:0/1027562616 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02b4108d90 0x7f02b419ef60 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f02a400e9c0 tx=0x7f02a400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:08.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.773+0000 7f02ab400640 1 -- 172.21.15.72:0/1027562616 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a400cde0 con 0x7f02b4108d90 2024-04-26T23:43:08.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.773+0000 7f02ab400640 1 -- 172.21.15.72:0/1027562616 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02a40108e0 con 0x7f02b4108d90 2024-04-26T23:43:08.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.773+0000 7f02b3e00640 1 -- 172.21.15.72:0/1027562616 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02b41a6e70 con 0x7f02b4108d90 2024-04-26T23:43:08.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.773+0000 7f02ab400640 1 -- 172.21.15.72:0/1027562616 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a4018960 con 0x7f02b4108d90 2024-04-26T23:43:08.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.773+0000 7f02b3e00640 1 -- 172.21.15.72:0/1027562616 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02b41a72d0 con 0x7f02b4108d90 2024-04-26T23:43:08.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.774+0000 7f02b3e00640 1 -- 172.21.15.72:0/1027562616 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02b41100b0 con 0x7f02b4108d90 2024-04-26T23:43:08.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.776+0000 7f02ab400640 1 -- 172.21.15.72:0/1027562616 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f02a401f020 con 0x7f02b4108d90 2024-04-26T23:43:08.781 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.778+0000 7f02ab400640 1 --2- 172.21.15.72:0/1027562616 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f02980a3bb0 0x7f02980a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:08.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.779+0000 7f02ab400640 1 -- 172.21.15.72:0/1027562616 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f02a40d0960 con 0x7f02b4108d90 2024-04-26T23:43:08.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.779+0000 7f02b2000640 1 --2- 172.21.15.72:0/1027562616 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f02980a3bb0 0x7f02980a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:08.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.779+0000 7f02b2000640 1 --2- 172.21.15.72:0/1027562616 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f02980a3bb0 0x7f02980a6070 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f02a0000c00 tx=0x7f02a00314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:08.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:08.783+0000 7f02ab400640 1 -- 172.21.15.72:0/1027562616 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f02a4054060 con 0x7f02b4108d90 2024-04-26T23:43:08.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:08 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/346012808' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-26T23:43:08.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:08 smithi072 ceph-mon[28184]: osdmap e88: 12 total, 12 up, 12 in 2024-04-26T23:43:08.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:08 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/3420675397' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-26T23:43:08.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:08 smithi072 ceph-mon[28184]: osdmap e89: 12 total, 12 up, 12 in 2024-04-26T23:43:08.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:08 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/346012808' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-26T23:43:08.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:08 smithi136 ceph-mon[33050]: osdmap e88: 12 total, 12 up, 12 in 2024-04-26T23:43:08.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:08 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/3420675397' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-26T23:43:08.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:08 smithi136 ceph-mon[33050]: osdmap e89: 12 total, 12 up, 12 in 2024-04-26T23:43:09.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.019+0000 7f02b3e00640 1 -- 172.21.15.72:0/1027562616 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f02b402dd80 con 0x7f02b4108d90 2024-04-26T23:43:09.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.020+0000 7f02ab400640 1 -- 172.21.15.72:0/1027562616 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11803 (secure 0 0 0) 0x7f02a409d100 con 0x7f02b4108d90 2024-04-26T23:43:09.023 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:43:09.023 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/2386456166","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2386456166},{"type":"v1","addr":"172.21.15.72:6835","nonce":2386456166}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1528342053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1528342053},{"type":"v1","addr":"172.21.15.136:6833","nonce":1528342053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6833/2427238559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":2427238559},{"type":"v1","addr":"172.21.15.76:6833","nonce":2427238559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24491,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/4050689894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":4050689894},{"type":"v1","addr":"172.21.15.72:6837","nonce":4050689894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3681925367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3681925367},{"type":"v1","addr":"172.21.15.136:6835","nonce":3681925367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4251109085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4251109085},{"type":"v1","addr":"172.21.15.76:6835","nonce":4251109085}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24514,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/3639308555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3639308555},{"type":"v1","addr":"172.21.15.136:6837","nonce":3639308555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/2506930945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2506930945},{"type":"v1","addr":"172.21.15.76:6837","nonce":2506930945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/3938677630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3938677630},{"type":"v1","addr":"172.21.15.136:6839","nonce":3938677630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24553,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6839/1403162102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6838","nonce":1403162102},{"type":"v1","addr":"172.21.15.72:6839","nonce":1403162102}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24571,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6841/1356345077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6840","nonce":1356345077},{"type":"v1","addr":"172.21.15.72:6841","nonce":1356345077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:43:02.928649+0000","modified":"2024-04-26T23:43:07.991051+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24590},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.76:6839/3405829058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":3405829058},{"type":"v1","addr":"172.21.15.76:6839","nonce":3405829058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24590,"qdb_cluster":[24590]},"id":1}]} 2024-04-26T23:43:09.023 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 17 2024-04-26T23:43:09.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.022+0000 7f02b3e00640 1 -- 172.21.15.72:0/1027562616 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f02980a3bb0 msgr2=0x7f02980a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:09.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.022+0000 7f02b3e00640 1 --2- 172.21.15.72:0/1027562616 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f02980a3bb0 0x7f02980a6070 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f02a0000c00 tx=0x7f02a00314a0 comp rx=0 tx=0).stop 2024-04-26T23:43:09.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.023+0000 7f02b3e00640 1 -- 172.21.15.72:0/1027562616 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02b4108d90 msgr2=0x7f02b419ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:09.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.023+0000 7f02b3e00640 1 --2- 172.21.15.72:0/1027562616 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02b4108d90 0x7f02b419ef60 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f02a400e9c0 tx=0x7f02a400ee80 comp rx=0 tx=0).stop 2024-04-26T23:43:09.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.023+0000 7f02b3e00640 1 -- 172.21.15.72:0/1027562616 shutdown_connections 2024-04-26T23:43:09.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.023+0000 7f02b3e00640 1 --2- 172.21.15.72:0/1027562616 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f02980a3bb0 0x7f02980a6070 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:09.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.023+0000 7f02b3e00640 1 --2- 172.21.15.72:0/1027562616 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f02b410a940 0x7f02b41a6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:09.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.023+0000 7f02b3e00640 1 --2- 172.21.15.72:0/1027562616 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f02b4109f80 0x7f02b419f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:09.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.023+0000 7f02b3e00640 1 --2- 172.21.15.72:0/1027562616 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02b4108d90 0x7f02b419ef60 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:09.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.023+0000 7f02b3e00640 1 -- 172.21.15.72:0/1027562616 >> 172.21.15.72:0/1027562616 conn(0x7f02b4077d30 msgr2=0x7f02b410d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:09.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.023+0000 7f02b3e00640 1 -- 172.21.15.72:0/1027562616 shutdown_connections 2024-04-26T23:43:09.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.023+0000 7f02b3e00640 1 -- 172.21.15.72:0/1027562616 wait complete. 2024-04-26T23:43:09.046 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:43:09.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.235+0000 7f8e7e800640 1 -- 172.21.15.72:0/1313093530 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e7810a180 msgr2=0x7f8e7810a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:09.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.235+0000 7f8e7e800640 1 --2- 172.21.15.72:0/1313093530 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e7810a180 0x7f8e7810a600 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8e60009860 tx=0x7f8e6002a340 comp rx=0 tx=0).stop 2024-04-26T23:43:09.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.236+0000 7f8e7e800640 1 -- 172.21.15.72:0/1313093530 shutdown_connections 2024-04-26T23:43:09.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.236+0000 7f8e7e800640 1 --2- 172.21.15.72:0/1313093530 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8e7810ab40 0x7f8e781113d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:09.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.236+0000 7f8e7e800640 1 --2- 172.21.15.72:0/1313093530 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e7810a180 0x7f8e7810a600 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:09.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.236+0000 7f8e7e800640 1 --2- 172.21.15.72:0/1313093530 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8e78108f90 0x7f8e781093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:09.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.236+0000 7f8e7e800640 1 -- 172.21.15.72:0/1313093530 >> 172.21.15.72:0/1313093530 conn(0x7f8e78077f40 msgr2=0x7f8e7807a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:09.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.236+0000 7f8e7e800640 1 -- 172.21.15.72:0/1313093530 shutdown_connections 2024-04-26T23:43:09.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.236+0000 7f8e7e800640 1 -- 172.21.15.72:0/1313093530 wait complete. 2024-04-26T23:43:09.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.237+0000 7f8e7e800640 1 Processor -- start 2024-04-26T23:43:09.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.237+0000 7f8e7e800640 1 -- start start 2024-04-26T23:43:09.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.237+0000 7f8e7e800640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8e78108f90 0x7f8e7819f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:09.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.237+0000 7f8e7e800640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e7810a180 0x7f8e7819f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:09.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.237+0000 7f8e7e800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8e7810ab40 0x7f8e781a6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:09.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.237+0000 7f8e7e800640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f8e78114ad0 con 0x7f8e7810a180 2024-04-26T23:43:09.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.237+0000 7f8e7e800640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8e78114950 con 0x7f8e7810ab40 2024-04-26T23:43:09.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.237+0000 7f8e7e800640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f8e78114c50 con 0x7f8e78108f90 2024-04-26T23:43:09.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.237+0000 7f8e6ea00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e7810a180 0x7f8e7819f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:09.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.237+0000 7f8e6ea00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e7810a180 0x7f8e7819f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43272/0 (socket says 172.21.15.72:43272) 2024-04-26T23:43:09.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.237+0000 7f8e6ea00640 1 -- 172.21.15.72:0/1811599247 learned_addr learned my addr 172.21.15.72:0/1811599247 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:43:09.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.237+0000 7f8e77e00640 1 --2- 172.21.15.72:0/1811599247 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8e7810ab40 0x7f8e781a6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:09.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.238+0000 7f8e77400640 1 --2- 172.21.15.72:0/1811599247 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8e78108f90 0x7f8e7819f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:09.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.238+0000 7f8e6ea00640 1 -- 172.21.15.72:0/1811599247 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8e78108f90 msgr2=0x7f8e7819f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:09.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.238+0000 7f8e6ea00640 1 --2- 172.21.15.72:0/1811599247 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8e78108f90 0x7f8e7819f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:09.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.238+0000 7f8e6ea00640 1 -- 172.21.15.72:0/1811599247 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8e7810ab40 msgr2=0x7f8e781a6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:09.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.238+0000 7f8e6ea00640 1 --2- 172.21.15.72:0/1811599247 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8e7810ab40 0x7f8e781a6750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:09.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.238+0000 7f8e6ea00640 1 -- 172.21.15.72:0/1811599247 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e781a6dc0 con 0x7f8e7810a180 2024-04-26T23:43:09.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.238+0000 7f8e6ea00640 1 --2- 172.21.15.72:0/1811599247 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e7810a180 0x7f8e7819f6e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8e600057c0 tx=0x7f8e60032da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:09.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.238+0000 7f8e74c00640 1 -- 172.21.15.72:0/1811599247 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e60037040 con 0x7f8e7810a180 2024-04-26T23:43:09.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.238+0000 7f8e74c00640 1 -- 172.21.15.72:0/1811599247 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e60030bd0 con 0x7f8e7810a180 2024-04-26T23:43:09.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.238+0000 7f8e7e800640 1 -- 172.21.15.72:0/1811599247 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e781a7050 con 0x7f8e7810a180 2024-04-26T23:43:09.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.238+0000 7f8e74c00640 1 -- 172.21.15.72:0/1811599247 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e60005280 con 0x7f8e7810a180 2024-04-26T23:43:09.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.239+0000 7f8e7e800640 1 -- 172.21.15.72:0/1811599247 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e781a74e0 con 0x7f8e7810a180 2024-04-26T23:43:09.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.240+0000 7f8e7e800640 1 -- 172.21.15.72:0/1811599247 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e78110300 con 0x7f8e7810a180 2024-04-26T23:43:09.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.242+0000 7f8e74c00640 1 -- 172.21.15.72:0/1811599247 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8e60043020 con 0x7f8e7810a180 2024-04-26T23:43:09.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.243+0000 7f8e74c00640 1 --2- 172.21.15.72:0/1811599247 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f8e5c0a3bb0 0x7f8e5c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:09.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.243+0000 7f8e74c00640 1 -- 172.21.15.72:0/1811599247 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f8e600efcd0 con 0x7f8e7810a180 2024-04-26T23:43:09.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.243+0000 7f8e77400640 1 --2- 172.21.15.72:0/1811599247 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f8e5c0a3bb0 0x7f8e5c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:09.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.244+0000 7f8e77400640 1 --2- 172.21.15.72:0/1811599247 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f8e5c0a3bb0 0x7f8e5c0a6070 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8e680098d0 tx=0x7f8e68010040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:09.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.249+0000 7f8e74c00640 1 -- 172.21.15.72:0/1811599247 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8e600bc410 con 0x7f8e7810a180 2024-04-26T23:43:09.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.456+0000 7f8e7e800640 1 -- 172.21.15.72:0/1811599247 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8e780635d0 con 0x7f8e7810a180 2024-04-26T23:43:09.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.458+0000 7f8e74c00640 1 -- 172.21.15.72:0/1811599247 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24081 (secure 0 0 0) 0x7f8e600bbd10 con 0x7f8e7810a180 2024-04-26T23:43:09.462 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:43:09.463 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"fsid":"3c8453ae-0425-11ef-bc93-c7b262605968","created":"2024-04-26T23:33:48.088300+0000","modified":"2024-04-26T23:43:07.990780+0000","last_up_change":"2024-04-26T23:42:04.482891+0000","last_in_change":"2024-04-26T23:41:44.784415+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:37:44.428727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:43:00.431829+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:43:01.872977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:43:05.512050+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e20c9ebf-56f6-47c1-a11b-e1e615af042b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6803","nonce":1639298875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6805","nonce":1639298875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6809","nonce":1639298875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6807","nonce":1639298875}]},"public_addr":"172.21.15.72:6803/1639298875","cluster_addr":"172.21.15.72:6805/1639298875","heartbeat_back_addr":"172.21.15.72:6809/1639298875","heartbeat_front_addr":"172.21.15.72:6807/1639298875","state":["exists","up"]},{"osd":1,"uuid":"e6cac469-780c-4d66-b560-b686072371fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6801","nonce":4185342193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6803","nonce":4185342193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6807","nonce":4185342193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6805","nonce":4185342193}]},"public_addr":"172.21.15.76:6801/4185342193","cluster_addr":"172.21.15.76:6803/4185342193","heartbeat_back_addr":"172.21.15.76:6807/4185342193","heartbeat_front_addr":"172.21.15.76:6805/4185342193","state":["exists","up"]},{"osd":2,"uuid":"838a49fc-18de-446c-8336-67b3e05da231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6801","nonce":3033860799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6803","nonce":3033860799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6807","nonce":3033860799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6805","nonce":3033860799}]},"public_addr":"172.21.15.136:6801/3033860799","cluster_addr":"172.21.15.136:6803/3033860799","heartbeat_back_addr":"172.21.15.136:6807/3033860799","heartbeat_front_addr":"172.21.15.136:6805/3033860799","state":["exists","up"]},{"osd":3,"uuid":"e6da89be-5ba7-48a4-9306-f75498660e0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6811","nonce":2071107815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6813","nonce":2071107815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6817","nonce":2071107815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6815","nonce":2071107815}]},"public_addr":"172.21.15.72:6811/2071107815","cluster_addr":"172.21.15.72:6813/2071107815","heartbeat_back_addr":"172.21.15.72:6817/2071107815","heartbeat_front_addr":"172.21.15.72:6815/2071107815","state":["exists","up"]},{"osd":4,"uuid":"0b17c491-d798-488f-892d-521ef2fbc51e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6809","nonce":147449238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6811","nonce":147449238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6815","nonce":147449238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6813","nonce":147449238}]},"public_addr":"172.21.15.76:6809/147449238","cluster_addr":"172.21.15.76:6811/147449238","heartbeat_back_addr":"172.21.15.76:6815/147449238","heartbeat_front_addr":"172.21.15.76:6813/147449238","state":["exists","up"]},{"osd":5,"uuid":"c7e887b3-ab38-4608-9bfd-1ed49956e8af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6809","nonce":2320149212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6811","nonce":2320149212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6815","nonce":2320149212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6813","nonce":2320149212}]},"public_addr":"172.21.15.136:6809/2320149212","cluster_addr":"172.21.15.136:6811/2320149212","heartbeat_back_addr":"172.21.15.136:6815/2320149212","heartbeat_front_addr":"172.21.15.136:6813/2320149212","state":["exists","up"]},{"osd":6,"uuid":"f0bd604b-0f57-4a38-9295-eb200264f1ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6819","nonce":389499207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6821","nonce":389499207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6825","nonce":389499207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6823","nonce":389499207}]},"public_addr":"172.21.15.72:6819/389499207","cluster_addr":"172.21.15.72:6821/389499207","heartbeat_back_addr":"172.21.15.72:6825/389499207","heartbeat_front_addr":"172.21.15.72:6823/389499207","state":["exists","up"]},{"osd":7,"uuid":"1879d4e4-038e-4cd6-a3ec-3a940bcc3797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6817","nonce":375888575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6819","nonce":375888575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6823","nonce":375888575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6821","nonce":375888575}]},"public_addr":"172.21.15.76:6817/375888575","cluster_addr":"172.21.15.76:6819/375888575","heartbeat_back_addr":"172.21.15.76:6823/375888575","heartbeat_front_addr":"172.21.15.76:6821/375888575","state":["exists","up"]},{"osd":8,"uuid":"6f404062-97e7-4ccb-a582-76dc1ffae548","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6817","nonce":3446774052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6819","nonce":3446774052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6823","nonce":3446774052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6821","nonce":3446774052}]},"public_addr":"172.21.15.136:6817/3446774052","cluster_addr":"172.21.15.136:6819/3446774052","heartbeat_back_addr":"172.21.15.136:6823/3446774052","heartbeat_front_addr":"172.21.15.136:6821/3446774052","state":["exists","up"]},{"osd":9,"uuid":"2989ca68-3164-4667-9bcf-230dcbb7b8f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6827","nonce":3423789152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6829","nonce":3423789152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6832","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6833","nonce":3423789152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6831","nonce":3423789152}]},"public_addr":"172.21.15.72:6827/3423789152","cluster_addr":"172.21.15.72:6829/3423789152","heartbeat_back_addr":"172.21.15.72:6833/3423789152","heartbeat_front_addr":"172.21.15.72:6831/3423789152","state":["exists","up"]},{"osd":10,"uuid":"9ec6d46a-0913-4208-97b5-167188ee8cb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6825","nonce":1120991574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6827","nonce":1120991574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6831","nonce":1120991574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6829","nonce":1120991574}]},"public_addr":"172.21.15.76:6825/1120991574","cluster_addr":"172.21.15.76:6827/1120991574","heartbeat_back_addr":"172.21.15.76:6831/1120991574","heartbeat_front_addr":"172.21.15.76:6829/1120991574","state":["exists","up"]},{"osd":11,"uuid":"bdd0774f-6e58-4918-b4cc-afd4f1118d3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6825","nonce":1135995193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6827","nonce":1135995193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6831","nonce":1135995193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6829","nonce":1135995193}]},"public_addr":"172.21.15.136:6825/1135995193","cluster_addr":"172.21.15.136:6827/1135995193","heartbeat_back_addr":"172.21.15.136:6831/1135995193","heartbeat_front_addr":"172.21.15.136:6829/1135995193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:36:52.128788+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:37:17.523654+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:37:41.634677+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:38:15.060797+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:38:41.234111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:39:06.783560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:39:40.884386+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:40:08.340465+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:40:34.344858+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:41:08.525309+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:41:36.782624+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:42:02.756888+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:0/3625071330":"2024-04-27T23:34:58.266218+0000","172.21.15.72:6801/2903979134":"2024-04-27T23:34:58.266218+0000","172.21.15.72:6800/2903979134":"2024-04-27T23:34:58.266218+0000","172.21.15.72:0/1519142576":"2024-04-27T23:34:58.266218+0000","172.21.15.72:0/3669626460":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/945623297":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/3479150073":"2024-04-27T23:34:24.233319+0000","172.21.15.72:6800/4175896290":"2024-04-27T23:34:24.233319+0000","172.21.15.72:6801/4175896290":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/3783175012":"2024-04-27T23:34:58.266218+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:43:09.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.461+0000 7f8e7e800640 1 -- 172.21.15.72:0/1811599247 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f8e5c0a3bb0 msgr2=0x7f8e5c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:09.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.461+0000 7f8e7e800640 1 --2- 172.21.15.72:0/1811599247 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f8e5c0a3bb0 0x7f8e5c0a6070 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8e680098d0 tx=0x7f8e68010040 comp rx=0 tx=0).stop 2024-04-26T23:43:09.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.461+0000 7f8e7e800640 1 -- 172.21.15.72:0/1811599247 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e7810a180 msgr2=0x7f8e7819f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:09.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.461+0000 7f8e7e800640 1 --2- 172.21.15.72:0/1811599247 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e7810a180 0x7f8e7819f6e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8e600057c0 tx=0x7f8e60032da0 comp rx=0 tx=0).stop 2024-04-26T23:43:09.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.461+0000 7f8e7e800640 1 -- 172.21.15.72:0/1811599247 shutdown_connections 2024-04-26T23:43:09.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.461+0000 7f8e7e800640 1 --2- 172.21.15.72:0/1811599247 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f8e5c0a3bb0 0x7f8e5c0a6070 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:09.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.461+0000 7f8e7e800640 1 --2- 172.21.15.72:0/1811599247 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8e7810ab40 0x7f8e781a6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:09.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.461+0000 7f8e7e800640 1 --2- 172.21.15.72:0/1811599247 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e7810a180 0x7f8e7819f6e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:09.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.461+0000 7f8e7e800640 1 --2- 172.21.15.72:0/1811599247 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8e78108f90 0x7f8e7819f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:09.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.461+0000 7f8e7e800640 1 -- 172.21.15.72:0/1811599247 >> 172.21.15.72:0/1811599247 conn(0x7f8e78077f40 msgr2=0x7f8e7810d4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:09.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.461+0000 7f8e7e800640 1 -- 172.21.15.72:0/1811599247 shutdown_connections 2024-04-26T23:43:09.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.461+0000 7f8e7e800640 1 -- 172.21.15.72:0/1811599247 wait complete. 2024-04-26T23:43:09.485 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:43:09.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.694+0000 7f93ba000640 1 -- 172.21.15.72:0/3063158254 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f93b4075390 msgr2=0x7f93b4075810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:09.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.694+0000 7f93ba000640 1 --2- 172.21.15.72:0/3063158254 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f93b4075390 0x7f93b4075810 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f93a0009900 tx=0x7f93a002a3f0 comp rx=0 tx=0).stop 2024-04-26T23:43:09.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.694+0000 7f93ba000640 1 -- 172.21.15.72:0/3063158254 shutdown_connections 2024-04-26T23:43:09.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.694+0000 7f93ba000640 1 --2- 172.21.15.72:0/3063158254 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f93b410a630 0x7f93b410cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:09.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.694+0000 7f93ba000640 1 --2- 172.21.15.72:0/3063158254 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f93b4075390 0x7f93b4075810 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:09.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.694+0000 7f93ba000640 1 --2- 172.21.15.72:0/3063158254 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f93b4076970 0x7f93b4074dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:09.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.694+0000 7f93ba000640 1 -- 172.21.15.72:0/3063158254 >> 172.21.15.72:0/3063158254 conn(0x7f93b40fdd00 msgr2=0x7f93b4100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:09.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.694+0000 7f93ba000640 1 -- 172.21.15.72:0/3063158254 shutdown_connections 2024-04-26T23:43:09.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.694+0000 7f93ba000640 1 -- 172.21.15.72:0/3063158254 wait complete. 2024-04-26T23:43:09.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.695+0000 7f93ba000640 1 Processor -- start 2024-04-26T23:43:09.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.695+0000 7f93ba000640 1 -- start start 2024-04-26T23:43:09.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.695+0000 7f93ba000640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f93b4075390 0x7f93b419ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:09.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.695+0000 7f93ba000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f93b4076970 0x7f93b419b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:09.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.695+0000 7f93ba000640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f93b410a630 0x7f93b41a2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:09.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.695+0000 7f93ba000640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f93b41104d0 con 0x7f93b4075390 2024-04-26T23:43:09.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.695+0000 7f93ba000640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f93b4110350 con 0x7f93b4076970 2024-04-26T23:43:09.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.695+0000 7f93ba000640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f93b4110650 con 0x7f93b410a630 2024-04-26T23:43:09.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.696+0000 7f93b8c00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f93b4075390 0x7f93b419ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:09.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.696+0000 7f93b3e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f93b4076970 0x7f93b419b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:09.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.696+0000 7f93b3e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f93b4076970 0x7f93b419b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:34726/0 (socket says 172.21.15.72:34726) 2024-04-26T23:43:09.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.696+0000 7f93b3e00640 1 -- 172.21.15.72:0/3723235722 learned_addr learned my addr 172.21.15.72:0/3723235722 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:43:09.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.696+0000 7f93b9600640 1 --2- 172.21.15.72:0/3723235722 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f93b410a630 0x7f93b41a2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:09.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.696+0000 7f93b3e00640 1 -- 172.21.15.72:0/3723235722 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f93b410a630 msgr2=0x7f93b41a2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:09.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.696+0000 7f93b3e00640 1 --2- 172.21.15.72:0/3723235722 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f93b410a630 0x7f93b41a2120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:09.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.696+0000 7f93b3e00640 1 -- 172.21.15.72:0/3723235722 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f93b4075390 msgr2=0x7f93b419ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:09.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.696+0000 7f93b3e00640 1 --2- 172.21.15.72:0/3723235722 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f93b4075390 0x7f93b419ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:09.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.696+0000 7f93b3e00640 1 -- 172.21.15.72:0/3723235722 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93b41a2790 con 0x7f93b4076970 2024-04-26T23:43:09.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.696+0000 7f93b3e00640 1 --2- 172.21.15.72:0/3723235722 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f93b4076970 0x7f93b419b0b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f93a0005ab0 tx=0x7f93a0003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:09.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.697+0000 7f93b1600640 1 -- 172.21.15.72:0/3723235722 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93a003f070 con 0x7f93b4076970 2024-04-26T23:43:09.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.697+0000 7f93ba000640 1 -- 172.21.15.72:0/3723235722 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93b41a2a20 con 0x7f93b4076970 2024-04-26T23:43:09.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.697+0000 7f93b1600640 1 -- 172.21.15.72:0/3723235722 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93a0031c50 con 0x7f93b4076970 2024-04-26T23:43:09.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.697+0000 7f93b1600640 1 -- 172.21.15.72:0/3723235722 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93a0035030 con 0x7f93b4076970 2024-04-26T23:43:09.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.697+0000 7f93ba000640 1 -- 172.21.15.72:0/3723235722 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93b41a2f10 con 0x7f93b4076970 2024-04-26T23:43:09.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.698+0000 7f93ba000640 1 -- 172.21.15.72:0/3723235722 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93b4109360 con 0x7f93b4076970 2024-04-26T23:43:09.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.700+0000 7f93b1600640 1 -- 172.21.15.72:0/3723235722 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f93a0042070 con 0x7f93b4076970 2024-04-26T23:43:09.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.701+0000 7f93b1600640 1 --2- 172.21.15.72:0/3723235722 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f939c0a3bb0 0x7f939c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:09.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.701+0000 7f93b1600640 1 -- 172.21.15.72:0/3723235722 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f93a00f0dd0 con 0x7f93b4076970 2024-04-26T23:43:09.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.701+0000 7f93b8c00640 1 --2- 172.21.15.72:0/3723235722 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f939c0a3bb0 0x7f939c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:09.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.702+0000 7f93b8c00640 1 --2- 172.21.15.72:0/3723235722 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f939c0a3bb0 0x7f939c0a6070 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f93a8009bd0 tx=0x7f93a8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:09.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.705+0000 7f93b1600640 1 -- 172.21.15.72:0/3723235722 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f93a00bcec0 con 0x7f93b4076970 2024-04-26T23:43:09.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:09 smithi072 ceph-mon[28184]: pgmap v324: 129 pgs: 2 creating+peering, 97 active+clean, 30 unknown; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 6 op/s 2024-04-26T23:43:09.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:09 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/3420675397' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-26T23:43:09.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:09 smithi072 ceph-mon[28184]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T23:43:09.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:09 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/1027562616' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:43:09.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:09 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/1811599247' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:43:09.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:09 smithi136 ceph-mon[33050]: pgmap v324: 129 pgs: 2 creating+peering, 97 active+clean, 30 unknown; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 6 op/s 2024-04-26T23:43:09.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:09 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/3420675397' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-26T23:43:09.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:09 smithi136 ceph-mon[33050]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T23:43:09.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:09 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/1027562616' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:43:09.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:09 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/1811599247' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:43:09.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.909+0000 7f93ba000640 1 -- 172.21.15.72:0/3723235722 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f93b40635d0 con 0x7f93b4076970 2024-04-26T23:43:09.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.911+0000 7f93b1600640 1 -- 172.21.15.72:0/3723235722 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24081 (secure 0 0 0) 0x7f93a00c11c0 con 0x7f93b4076970 2024-04-26T23:43:09.915 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:43:09.915 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"fsid":"3c8453ae-0425-11ef-bc93-c7b262605968","created":"2024-04-26T23:33:48.088300+0000","modified":"2024-04-26T23:43:07.990780+0000","last_up_change":"2024-04-26T23:42:04.482891+0000","last_in_change":"2024-04-26T23:41:44.784415+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:37:44.428727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:43:00.431829+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:43:01.872977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:43:05.512050+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e20c9ebf-56f6-47c1-a11b-e1e615af042b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6803","nonce":1639298875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6805","nonce":1639298875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6809","nonce":1639298875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6807","nonce":1639298875}]},"public_addr":"172.21.15.72:6803/1639298875","cluster_addr":"172.21.15.72:6805/1639298875","heartbeat_back_addr":"172.21.15.72:6809/1639298875","heartbeat_front_addr":"172.21.15.72:6807/1639298875","state":["exists","up"]},{"osd":1,"uuid":"e6cac469-780c-4d66-b560-b686072371fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6801","nonce":4185342193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6803","nonce":4185342193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6807","nonce":4185342193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6805","nonce":4185342193}]},"public_addr":"172.21.15.76:6801/4185342193","cluster_addr":"172.21.15.76:6803/4185342193","heartbeat_back_addr":"172.21.15.76:6807/4185342193","heartbeat_front_addr":"172.21.15.76:6805/4185342193","state":["exists","up"]},{"osd":2,"uuid":"838a49fc-18de-446c-8336-67b3e05da231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6801","nonce":3033860799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6803","nonce":3033860799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6807","nonce":3033860799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6805","nonce":3033860799}]},"public_addr":"172.21.15.136:6801/3033860799","cluster_addr":"172.21.15.136:6803/3033860799","heartbeat_back_addr":"172.21.15.136:6807/3033860799","heartbeat_front_addr":"172.21.15.136:6805/3033860799","state":["exists","up"]},{"osd":3,"uuid":"e6da89be-5ba7-48a4-9306-f75498660e0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6811","nonce":2071107815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6813","nonce":2071107815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6817","nonce":2071107815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6815","nonce":2071107815}]},"public_addr":"172.21.15.72:6811/2071107815","cluster_addr":"172.21.15.72:6813/2071107815","heartbeat_back_addr":"172.21.15.72:6817/2071107815","heartbeat_front_addr":"172.21.15.72:6815/2071107815","state":["exists","up"]},{"osd":4,"uuid":"0b17c491-d798-488f-892d-521ef2fbc51e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6809","nonce":147449238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6811","nonce":147449238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6815","nonce":147449238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6813","nonce":147449238}]},"public_addr":"172.21.15.76:6809/147449238","cluster_addr":"172.21.15.76:6811/147449238","heartbeat_back_addr":"172.21.15.76:6815/147449238","heartbeat_front_addr":"172.21.15.76:6813/147449238","state":["exists","up"]},{"osd":5,"uuid":"c7e887b3-ab38-4608-9bfd-1ed49956e8af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6809","nonce":2320149212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6811","nonce":2320149212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6815","nonce":2320149212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6813","nonce":2320149212}]},"public_addr":"172.21.15.136:6809/2320149212","cluster_addr":"172.21.15.136:6811/2320149212","heartbeat_back_addr":"172.21.15.136:6815/2320149212","heartbeat_front_addr":"172.21.15.136:6813/2320149212","state":["exists","up"]},{"osd":6,"uuid":"f0bd604b-0f57-4a38-9295-eb200264f1ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6819","nonce":389499207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6821","nonce":389499207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6825","nonce":389499207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6823","nonce":389499207}]},"public_addr":"172.21.15.72:6819/389499207","cluster_addr":"172.21.15.72:6821/389499207","heartbeat_back_addr":"172.21.15.72:6825/389499207","heartbeat_front_addr":"172.21.15.72:6823/389499207","state":["exists","up"]},{"osd":7,"uuid":"1879d4e4-038e-4cd6-a3ec-3a940bcc3797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6817","nonce":375888575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6819","nonce":375888575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6823","nonce":375888575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6821","nonce":375888575}]},"public_addr":"172.21.15.76:6817/375888575","cluster_addr":"172.21.15.76:6819/375888575","heartbeat_back_addr":"172.21.15.76:6823/375888575","heartbeat_front_addr":"172.21.15.76:6821/375888575","state":["exists","up"]},{"osd":8,"uuid":"6f404062-97e7-4ccb-a582-76dc1ffae548","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6817","nonce":3446774052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6819","nonce":3446774052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6823","nonce":3446774052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6821","nonce":3446774052}]},"public_addr":"172.21.15.136:6817/3446774052","cluster_addr":"172.21.15.136:6819/3446774052","heartbeat_back_addr":"172.21.15.136:6823/3446774052","heartbeat_front_addr":"172.21.15.136:6821/3446774052","state":["exists","up"]},{"osd":9,"uuid":"2989ca68-3164-4667-9bcf-230dcbb7b8f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6827","nonce":3423789152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6829","nonce":3423789152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6832","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6833","nonce":3423789152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6831","nonce":3423789152}]},"public_addr":"172.21.15.72:6827/3423789152","cluster_addr":"172.21.15.72:6829/3423789152","heartbeat_back_addr":"172.21.15.72:6833/3423789152","heartbeat_front_addr":"172.21.15.72:6831/3423789152","state":["exists","up"]},{"osd":10,"uuid":"9ec6d46a-0913-4208-97b5-167188ee8cb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6825","nonce":1120991574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6827","nonce":1120991574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6831","nonce":1120991574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6829","nonce":1120991574}]},"public_addr":"172.21.15.76:6825/1120991574","cluster_addr":"172.21.15.76:6827/1120991574","heartbeat_back_addr":"172.21.15.76:6831/1120991574","heartbeat_front_addr":"172.21.15.76:6829/1120991574","state":["exists","up"]},{"osd":11,"uuid":"bdd0774f-6e58-4918-b4cc-afd4f1118d3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6825","nonce":1135995193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6827","nonce":1135995193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6831","nonce":1135995193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6829","nonce":1135995193}]},"public_addr":"172.21.15.136:6825/1135995193","cluster_addr":"172.21.15.136:6827/1135995193","heartbeat_back_addr":"172.21.15.136:6831/1135995193","heartbeat_front_addr":"172.21.15.136:6829/1135995193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:36:52.128788+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:37:17.523654+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:37:41.634677+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:38:15.060797+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:38:41.234111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:39:06.783560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:39:40.884386+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:40:08.340465+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:40:34.344858+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:41:08.525309+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:41:36.782624+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:42:02.756888+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:0/3625071330":"2024-04-27T23:34:58.266218+0000","172.21.15.72:6801/2903979134":"2024-04-27T23:34:58.266218+0000","172.21.15.72:6800/2903979134":"2024-04-27T23:34:58.266218+0000","172.21.15.72:0/1519142576":"2024-04-27T23:34:58.266218+0000","172.21.15.72:0/3669626460":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/945623297":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/3479150073":"2024-04-27T23:34:24.233319+0000","172.21.15.72:6800/4175896290":"2024-04-27T23:34:24.233319+0000","172.21.15.72:6801/4175896290":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/3783175012":"2024-04-27T23:34:58.266218+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:43:09.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.914+0000 7f93ba000640 1 -- 172.21.15.72:0/3723235722 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f939c0a3bb0 msgr2=0x7f939c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:09.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.914+0000 7f93ba000640 1 --2- 172.21.15.72:0/3723235722 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f939c0a3bb0 0x7f939c0a6070 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f93a8009bd0 tx=0x7f93a8005f90 comp rx=0 tx=0).stop 2024-04-26T23:43:09.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.914+0000 7f93ba000640 1 -- 172.21.15.72:0/3723235722 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f93b4076970 msgr2=0x7f93b419b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:09.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.914+0000 7f93ba000640 1 --2- 172.21.15.72:0/3723235722 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f93b4076970 0x7f93b419b0b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f93a0005ab0 tx=0x7f93a0003b90 comp rx=0 tx=0).stop 2024-04-26T23:43:09.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.914+0000 7f93ba000640 1 -- 172.21.15.72:0/3723235722 shutdown_connections 2024-04-26T23:43:09.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.914+0000 7f93ba000640 1 --2- 172.21.15.72:0/3723235722 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f939c0a3bb0 0x7f939c0a6070 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:09.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.914+0000 7f93ba000640 1 --2- 172.21.15.72:0/3723235722 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f93b4076970 0x7f93b419b0b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:09.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.914+0000 7f93ba000640 1 --2- 172.21.15.72:0/3723235722 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f93b410a630 0x7f93b41a2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:09.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.914+0000 7f93ba000640 1 --2- 172.21.15.72:0/3723235722 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f93b4075390 0x7f93b419ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:09.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.914+0000 7f93ba000640 1 -- 172.21.15.72:0/3723235722 >> 172.21.15.72:0/3723235722 conn(0x7f93b40fdd00 msgr2=0x7f93b4106530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:09.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.914+0000 7f93ba000640 1 -- 172.21.15.72:0/3723235722 shutdown_connections 2024-04-26T23:43:09.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:09.914+0000 7f93ba000640 1 -- 172.21.15.72:0/3723235722 wait complete. 2024-04-26T23:43:09.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:09 smithi076 ceph-mon[32774]: pgmap v324: 129 pgs: 2 creating+peering, 97 active+clean, 30 unknown; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 6 op/s 2024-04-26T23:43:09.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:09 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/3420675397' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-26T23:43:09.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:09 smithi076 ceph-mon[32774]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T23:43:09.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:09 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/1027562616' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:43:09.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:09 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/1811599247' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:43:09.949 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:43:10.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.148+0000 7f289ca00640 1 -- 172.21.15.72:0/3355115116 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f289810a940 msgr2=0x7f28981111d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:10.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.148+0000 7f289ca00640 1 --2- 172.21.15.72:0/3355115116 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f289810a940 0x7f28981111d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2890009ca0 tx=0x7f289002a8b0 comp rx=0 tx=0).stop 2024-04-26T23:43:10.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.149+0000 7f289ca00640 1 -- 172.21.15.72:0/3355115116 shutdown_connections 2024-04-26T23:43:10.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.149+0000 7f289ca00640 1 --2- 172.21.15.72:0/3355115116 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f289810a940 0x7f28981111d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:10.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.149+0000 7f289ca00640 1 --2- 172.21.15.72:0/3355115116 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2898109f80 0x7f289810a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:10.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.149+0000 7f289ca00640 1 --2- 172.21.15.72:0/3355115116 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2898108d90 0x7f28981091b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:10.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.149+0000 7f289ca00640 1 -- 172.21.15.72:0/3355115116 >> 172.21.15.72:0/3355115116 conn(0x7f2898077d30 msgr2=0x7f289807a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:10.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.149+0000 7f289ca00640 1 -- 172.21.15.72:0/3355115116 shutdown_connections 2024-04-26T23:43:10.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.149+0000 7f289ca00640 1 -- 172.21.15.72:0/3355115116 wait complete. 2024-04-26T23:43:10.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.150+0000 7f289ca00640 1 Processor -- start 2024-04-26T23:43:10.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.150+0000 7f289ca00640 1 -- start start 2024-04-26T23:43:10.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.150+0000 7f289ca00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2898108d90 0x7f289819eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:10.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.150+0000 7f289ca00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2898109f80 0x7f289819f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:10.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.150+0000 7f289ca00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f289810a940 0x7f28981a64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:10.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.150+0000 7f289ca00640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f2898114910 con 0x7f2898109f80 2024-04-26T23:43:10.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.150+0000 7f289ca00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2898114790 con 0x7f289810a940 2024-04-26T23:43:10.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.150+0000 7f289ca00640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f2898114a90 con 0x7f2898108d90 2024-04-26T23:43:10.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.151+0000 7f288de00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2898109f80 0x7f289819f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:10.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.151+0000 7f2896800640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2898108d90 0x7f289819eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:10.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.151+0000 7f2896800640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2898108d90 0x7f289819eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.72:35790/0 (socket says 172.21.15.72:35790) 2024-04-26T23:43:10.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.151+0000 7f288de00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2898109f80 0x7f289819f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43308/0 (socket says 172.21.15.72:43308) 2024-04-26T23:43:10.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.151+0000 7f2896800640 1 -- 172.21.15.72:0/2441519532 learned_addr learned my addr 172.21.15.72:0/2441519532 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:43:10.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.151+0000 7f2897200640 1 --2- 172.21.15.72:0/2441519532 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f289810a940 0x7f28981a64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:10.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.151+0000 7f2896800640 1 -- 172.21.15.72:0/2441519532 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f289810a940 msgr2=0x7f28981a64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:10.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.151+0000 7f2896800640 1 --2- 172.21.15.72:0/2441519532 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f289810a940 0x7f28981a64a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:10.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.151+0000 7f2896800640 1 -- 172.21.15.72:0/2441519532 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2898109f80 msgr2=0x7f289819f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:10.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.151+0000 7f2896800640 1 --2- 172.21.15.72:0/2441519532 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2898109f80 0x7f289819f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:10.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.151+0000 7f2896800640 1 -- 172.21.15.72:0/2441519532 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28981a6b10 con 0x7f2898108d90 2024-04-26T23:43:10.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.151+0000 7f288de00640 1 --2- 172.21.15.72:0/2441519532 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2898109f80 0x7f289819f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:43:10.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.152+0000 7f2896800640 1 --2- 172.21.15.72:0/2441519532 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2898108d90 0x7f289819eef0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f288800e9c0 tx=0x7f288800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:10.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.152+0000 7f288fe00640 1 -- 172.21.15.72:0/2441519532 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f288800cd80 con 0x7f2898108d90 2024-04-26T23:43:10.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.152+0000 7f288fe00640 1 -- 172.21.15.72:0/2441519532 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28880035f0 con 0x7f2898108d90 2024-04-26T23:43:10.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.152+0000 7f289ca00640 1 -- 172.21.15.72:0/2441519532 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28981a6e00 con 0x7f2898108d90 2024-04-26T23:43:10.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.152+0000 7f288fe00640 1 -- 172.21.15.72:0/2441519532 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2888010690 con 0x7f2898108d90 2024-04-26T23:43:10.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.152+0000 7f289ca00640 1 -- 172.21.15.72:0/2441519532 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28981a7260 con 0x7f2898108d90 2024-04-26T23:43:10.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.153+0000 7f289ca00640 1 -- 172.21.15.72:0/2441519532 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28981100b0 con 0x7f2898108d90 2024-04-26T23:43:10.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.155+0000 7f288fe00640 1 -- 172.21.15.72:0/2441519532 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2888004000 con 0x7f2898108d90 2024-04-26T23:43:10.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.157+0000 7f288fe00640 1 --2- 172.21.15.72:0/2441519532 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f28780a3c30 0x7f28780a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:10.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.157+0000 7f288fe00640 1 -- 172.21.15.72:0/2441519532 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f28880d0c00 con 0x7f2898108d90 2024-04-26T23:43:10.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.157+0000 7f288de00640 1 --2- 172.21.15.72:0/2441519532 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f28780a3c30 0x7f28780a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:10.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.157+0000 7f288de00640 1 --2- 172.21.15.72:0/2441519532 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f28780a3c30 0x7f28780a60f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f28981a0400 tx=0x7f2880005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:10.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.159+0000 7f288fe00640 1 -- 172.21.15.72:0/2441519532 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f288809d340 con 0x7f2898108d90 2024-04-26T23:43:10.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.400+0000 7f289ca00640 1 -- 172.21.15.72:0/2441519532 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f289802dad0 con 0x7f2898108d90 2024-04-26T23:43:10.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.402+0000 7f288fe00640 1 -- 172.21.15.72:0/2441519532 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11803 (secure 0 0 0) 0x7f288809cc40 con 0x7f2898108d90 2024-04-26T23:43:10.405 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:43:10.405 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/2386456166","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2386456166},{"type":"v1","addr":"172.21.15.72:6835","nonce":2386456166}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1528342053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1528342053},{"type":"v1","addr":"172.21.15.136:6833","nonce":1528342053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6833/2427238559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":2427238559},{"type":"v1","addr":"172.21.15.76:6833","nonce":2427238559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24491,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/4050689894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":4050689894},{"type":"v1","addr":"172.21.15.72:6837","nonce":4050689894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3681925367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3681925367},{"type":"v1","addr":"172.21.15.136:6835","nonce":3681925367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4251109085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4251109085},{"type":"v1","addr":"172.21.15.76:6835","nonce":4251109085}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24514,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/3639308555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3639308555},{"type":"v1","addr":"172.21.15.136:6837","nonce":3639308555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6837/2506930945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2506930945},{"type":"v1","addr":"172.21.15.76:6837","nonce":2506930945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24535,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/3938677630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3938677630},{"type":"v1","addr":"172.21.15.136:6839","nonce":3938677630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24553,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6839/1403162102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6838","nonce":1403162102},{"type":"v1","addr":"172.21.15.72:6839","nonce":1403162102}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24571,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6841/1356345077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6840","nonce":1356345077},{"type":"v1","addr":"172.21.15.72:6841","nonce":1356345077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T23:43:02.928649+0000","modified":"2024-04-26T23:43:07.991051+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24590},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.76:6839/3405829058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":3405829058},{"type":"v1","addr":"172.21.15.76:6839","nonce":3405829058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24590,"qdb_cluster":[24590]},"id":1}]} 2024-04-26T23:43:10.405 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 17 2024-04-26T23:43:10.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.405+0000 7f289ca00640 1 -- 172.21.15.72:0/2441519532 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f28780a3c30 msgr2=0x7f28780a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:10.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.405+0000 7f289ca00640 1 --2- 172.21.15.72:0/2441519532 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f28780a3c30 0x7f28780a60f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f28981a0400 tx=0x7f2880005f90 comp rx=0 tx=0).stop 2024-04-26T23:43:10.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.405+0000 7f289ca00640 1 -- 172.21.15.72:0/2441519532 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2898108d90 msgr2=0x7f289819eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:10.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.405+0000 7f289ca00640 1 --2- 172.21.15.72:0/2441519532 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2898108d90 0x7f289819eef0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f288800e9c0 tx=0x7f288800ee80 comp rx=0 tx=0).stop 2024-04-26T23:43:10.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.405+0000 7f289ca00640 1 -- 172.21.15.72:0/2441519532 shutdown_connections 2024-04-26T23:43:10.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.405+0000 7f289ca00640 1 --2- 172.21.15.72:0/2441519532 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f28780a3c30 0x7f28780a60f0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:10.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.405+0000 7f289ca00640 1 --2- 172.21.15.72:0/2441519532 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f289810a940 0x7f28981a64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:10.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.405+0000 7f289ca00640 1 --2- 172.21.15.72:0/2441519532 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2898109f80 0x7f289819f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:10.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.405+0000 7f289ca00640 1 --2- 172.21.15.72:0/2441519532 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2898108d90 0x7f289819eef0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:10.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.405+0000 7f289ca00640 1 -- 172.21.15.72:0/2441519532 >> 172.21.15.72:0/2441519532 conn(0x7f2898077d30 msgr2=0x7f289810d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:10.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.405+0000 7f289ca00640 1 -- 172.21.15.72:0/2441519532 shutdown_connections 2024-04-26T23:43:10.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.405+0000 7f289ca00640 1 -- 172.21.15.72:0/2441519532 wait complete. 2024-04-26T23:43:10.427 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:43:10.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.620+0000 7f54b1000640 1 -- 172.21.15.72:0/2160587575 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f54ac104b90 msgr2=0x7f54ac104fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:10.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.620+0000 7f54b1000640 1 --2- 172.21.15.72:0/2160587575 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f54ac104b90 0x7f54ac104fb0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f549c009900 tx=0x7f549c02a3f0 comp rx=0 tx=0).stop 2024-04-26T23:43:10.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.621+0000 7f54b1000640 1 -- 172.21.15.72:0/2160587575 shutdown_connections 2024-04-26T23:43:10.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.621+0000 7f54b1000640 1 --2- 172.21.15.72:0/2160587575 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54ac106740 0x7f54ac10cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:10.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.621+0000 7f54b1000640 1 --2- 172.21.15.72:0/2160587575 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f54ac105d80 0x7f54ac106200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:10.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.621+0000 7f54b1000640 1 --2- 172.21.15.72:0/2160587575 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f54ac104b90 0x7f54ac104fb0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:10.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.621+0000 7f54b1000640 1 -- 172.21.15.72:0/2160587575 >> 172.21.15.72:0/2160587575 conn(0x7f54ac100120 msgr2=0x7f54ac102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:10.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.621+0000 7f54b1000640 1 -- 172.21.15.72:0/2160587575 shutdown_connections 2024-04-26T23:43:10.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.621+0000 7f54b1000640 1 -- 172.21.15.72:0/2160587575 wait complete. 2024-04-26T23:43:10.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.622+0000 7f54b1000640 1 Processor -- start 2024-04-26T23:43:10.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.622+0000 7f54b1000640 1 -- start start 2024-04-26T23:43:10.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.622+0000 7f54b1000640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54ac104b90 0x7f54ac19add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:10.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.622+0000 7f54b1000640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f54ac105d80 0x7f54ac19b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:10.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.622+0000 7f54b1000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f54ac106740 0x7f54ac1a2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:10.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.623+0000 7f54b1000640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f54ac110700 con 0x7f54ac104b90 2024-04-26T23:43:10.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.623+0000 7f54b1000640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f54ac110580 con 0x7f54ac106740 2024-04-26T23:43:10.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.623+0000 7f54b1000640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f54ac110880 con 0x7f54ac105d80 2024-04-26T23:43:10.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.623+0000 7f54a9e00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54ac104b90 0x7f54ac19add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:10.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.623+0000 7f54a9e00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54ac104b90 0x7f54ac19add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43318/0 (socket says 172.21.15.72:43318) 2024-04-26T23:43:10.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.623+0000 7f54a9e00640 1 -- 172.21.15.72:0/916550265 learned_addr learned my addr 172.21.15.72:0/916550265 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:43:10.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.623+0000 7f54a9400640 1 --2- 172.21.15.72:0/916550265 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f54ac105d80 0x7f54ac19b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:10.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.623+0000 7f54aa800640 1 --2- 172.21.15.72:0/916550265 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f54ac106740 0x7f54ac1a2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:10.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.623+0000 7f54a9e00640 1 -- 172.21.15.72:0/916550265 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f54ac105d80 msgr2=0x7f54ac19b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:10.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.623+0000 7f54a9e00640 1 --2- 172.21.15.72:0/916550265 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f54ac105d80 0x7f54ac19b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:10.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.623+0000 7f54a9e00640 1 -- 172.21.15.72:0/916550265 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f54ac106740 msgr2=0x7f54ac1a2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:10.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.623+0000 7f54a9e00640 1 --2- 172.21.15.72:0/916550265 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f54ac106740 0x7f54ac1a2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:10.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.623+0000 7f54a9e00640 1 -- 172.21.15.72:0/916550265 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54ac1a29f0 con 0x7f54ac104b90 2024-04-26T23:43:10.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.623+0000 7f54a9400640 1 --2- 172.21.15.72:0/916550265 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f54ac105d80 0x7f54ac19b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:43:10.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.623+0000 7f54a9e00640 1 --2- 172.21.15.72:0/916550265 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54ac104b90 0x7f54ac19add0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f549c031500 tx=0x7f549c003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:10.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.624+0000 7f54a2a00640 1 -- 172.21.15.72:0/916550265 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f549c03f070 con 0x7f54ac104b90 2024-04-26T23:43:10.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.624+0000 7f54b1000640 1 -- 172.21.15.72:0/916550265 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54ac1a2c80 con 0x7f54ac104b90 2024-04-26T23:43:10.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.624+0000 7f54a2a00640 1 -- 172.21.15.72:0/916550265 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f549c031c50 con 0x7f54ac104b90 2024-04-26T23:43:10.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.624+0000 7f54a2a00640 1 -- 172.21.15.72:0/916550265 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f549c035030 con 0x7f54ac104b90 2024-04-26T23:43:10.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.624+0000 7f54b1000640 1 -- 172.21.15.72:0/916550265 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54ac1a3170 con 0x7f54ac104b90 2024-04-26T23:43:10.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.625+0000 7f54b1000640 1 -- 172.21.15.72:0/916550265 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54ac10bf30 con 0x7f54ac104b90 2024-04-26T23:43:10.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.627+0000 7f54a2a00640 1 -- 172.21.15.72:0/916550265 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f549c042070 con 0x7f54ac104b90 2024-04-26T23:43:10.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.629+0000 7f54a2a00640 1 --2- 172.21.15.72:0/916550265 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f54900a3bb0 0x7f54900a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:10.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.629+0000 7f54a2a00640 1 -- 172.21.15.72:0/916550265 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f549c0f1560 con 0x7f54ac104b90 2024-04-26T23:43:10.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.629+0000 7f54a9400640 1 --2- 172.21.15.72:0/916550265 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f54900a3bb0 0x7f54900a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:10.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.630+0000 7f54a9400640 1 --2- 172.21.15.72:0/916550265 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f54900a3bb0 0x7f54900a6070 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f54ac19c2e0 tx=0x7f5498005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:10.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.631+0000 7f54a2a00640 1 -- 172.21.15.72:0/916550265 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f549c0bdca0 con 0x7f54ac104b90 2024-04-26T23:43:10.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:10 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/3723235722' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:43:10.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:10 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/2441519532' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:43:10.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.833+0000 7f54b1000640 1 -- 172.21.15.72:0/916550265 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f54ac02dd20 con 0x7f54ac104b90 2024-04-26T23:43:10.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.837+0000 7f54a2a00640 1 -- 172.21.15.72:0/916550265 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24081 (secure 0 0 0) 0x7f549c0bd5a0 con 0x7f54ac104b90 2024-04-26T23:43:10.840 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:43:10.841 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"fsid":"3c8453ae-0425-11ef-bc93-c7b262605968","created":"2024-04-26T23:33:48.088300+0000","modified":"2024-04-26T23:43:07.990780+0000","last_up_change":"2024-04-26T23:42:04.482891+0000","last_in_change":"2024-04-26T23:41:44.784415+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:37:44.428727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:43:00.431829+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:43:01.872977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:43:05.512050+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e20c9ebf-56f6-47c1-a11b-e1e615af042b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6803","nonce":1639298875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6805","nonce":1639298875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6809","nonce":1639298875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6807","nonce":1639298875}]},"public_addr":"172.21.15.72:6803/1639298875","cluster_addr":"172.21.15.72:6805/1639298875","heartbeat_back_addr":"172.21.15.72:6809/1639298875","heartbeat_front_addr":"172.21.15.72:6807/1639298875","state":["exists","up"]},{"osd":1,"uuid":"e6cac469-780c-4d66-b560-b686072371fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6801","nonce":4185342193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6803","nonce":4185342193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6807","nonce":4185342193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6805","nonce":4185342193}]},"public_addr":"172.21.15.76:6801/4185342193","cluster_addr":"172.21.15.76:6803/4185342193","heartbeat_back_addr":"172.21.15.76:6807/4185342193","heartbeat_front_addr":"172.21.15.76:6805/4185342193","state":["exists","up"]},{"osd":2,"uuid":"838a49fc-18de-446c-8336-67b3e05da231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6801","nonce":3033860799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6803","nonce":3033860799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6807","nonce":3033860799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6805","nonce":3033860799}]},"public_addr":"172.21.15.136:6801/3033860799","cluster_addr":"172.21.15.136:6803/3033860799","heartbeat_back_addr":"172.21.15.136:6807/3033860799","heartbeat_front_addr":"172.21.15.136:6805/3033860799","state":["exists","up"]},{"osd":3,"uuid":"e6da89be-5ba7-48a4-9306-f75498660e0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6811","nonce":2071107815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6813","nonce":2071107815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6817","nonce":2071107815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6815","nonce":2071107815}]},"public_addr":"172.21.15.72:6811/2071107815","cluster_addr":"172.21.15.72:6813/2071107815","heartbeat_back_addr":"172.21.15.72:6817/2071107815","heartbeat_front_addr":"172.21.15.72:6815/2071107815","state":["exists","up"]},{"osd":4,"uuid":"0b17c491-d798-488f-892d-521ef2fbc51e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6809","nonce":147449238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6811","nonce":147449238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6815","nonce":147449238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6813","nonce":147449238}]},"public_addr":"172.21.15.76:6809/147449238","cluster_addr":"172.21.15.76:6811/147449238","heartbeat_back_addr":"172.21.15.76:6815/147449238","heartbeat_front_addr":"172.21.15.76:6813/147449238","state":["exists","up"]},{"osd":5,"uuid":"c7e887b3-ab38-4608-9bfd-1ed49956e8af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6809","nonce":2320149212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6811","nonce":2320149212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6815","nonce":2320149212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6813","nonce":2320149212}]},"public_addr":"172.21.15.136:6809/2320149212","cluster_addr":"172.21.15.136:6811/2320149212","heartbeat_back_addr":"172.21.15.136:6815/2320149212","heartbeat_front_addr":"172.21.15.136:6813/2320149212","state":["exists","up"]},{"osd":6,"uuid":"f0bd604b-0f57-4a38-9295-eb200264f1ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6819","nonce":389499207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6821","nonce":389499207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6825","nonce":389499207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6823","nonce":389499207}]},"public_addr":"172.21.15.72:6819/389499207","cluster_addr":"172.21.15.72:6821/389499207","heartbeat_back_addr":"172.21.15.72:6825/389499207","heartbeat_front_addr":"172.21.15.72:6823/389499207","state":["exists","up"]},{"osd":7,"uuid":"1879d4e4-038e-4cd6-a3ec-3a940bcc3797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6817","nonce":375888575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6819","nonce":375888575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6823","nonce":375888575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6821","nonce":375888575}]},"public_addr":"172.21.15.76:6817/375888575","cluster_addr":"172.21.15.76:6819/375888575","heartbeat_back_addr":"172.21.15.76:6823/375888575","heartbeat_front_addr":"172.21.15.76:6821/375888575","state":["exists","up"]},{"osd":8,"uuid":"6f404062-97e7-4ccb-a582-76dc1ffae548","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6817","nonce":3446774052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6819","nonce":3446774052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6823","nonce":3446774052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6821","nonce":3446774052}]},"public_addr":"172.21.15.136:6817/3446774052","cluster_addr":"172.21.15.136:6819/3446774052","heartbeat_back_addr":"172.21.15.136:6823/3446774052","heartbeat_front_addr":"172.21.15.136:6821/3446774052","state":["exists","up"]},{"osd":9,"uuid":"2989ca68-3164-4667-9bcf-230dcbb7b8f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6827","nonce":3423789152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6829","nonce":3423789152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6832","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6833","nonce":3423789152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6831","nonce":3423789152}]},"public_addr":"172.21.15.72:6827/3423789152","cluster_addr":"172.21.15.72:6829/3423789152","heartbeat_back_addr":"172.21.15.72:6833/3423789152","heartbeat_front_addr":"172.21.15.72:6831/3423789152","state":["exists","up"]},{"osd":10,"uuid":"9ec6d46a-0913-4208-97b5-167188ee8cb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6825","nonce":1120991574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6827","nonce":1120991574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6831","nonce":1120991574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6829","nonce":1120991574}]},"public_addr":"172.21.15.76:6825/1120991574","cluster_addr":"172.21.15.76:6827/1120991574","heartbeat_back_addr":"172.21.15.76:6831/1120991574","heartbeat_front_addr":"172.21.15.76:6829/1120991574","state":["exists","up"]},{"osd":11,"uuid":"bdd0774f-6e58-4918-b4cc-afd4f1118d3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6825","nonce":1135995193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6827","nonce":1135995193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6831","nonce":1135995193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6829","nonce":1135995193}]},"public_addr":"172.21.15.136:6825/1135995193","cluster_addr":"172.21.15.136:6827/1135995193","heartbeat_back_addr":"172.21.15.136:6831/1135995193","heartbeat_front_addr":"172.21.15.136:6829/1135995193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:36:52.128788+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:37:17.523654+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:37:41.634677+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:38:15.060797+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:38:41.234111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:39:06.783560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:39:40.884386+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:40:08.340465+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:40:34.344858+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:41:08.525309+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:41:36.782624+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:42:02.756888+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:0/3625071330":"2024-04-27T23:34:58.266218+0000","172.21.15.72:6801/2903979134":"2024-04-27T23:34:58.266218+0000","172.21.15.72:6800/2903979134":"2024-04-27T23:34:58.266218+0000","172.21.15.72:0/1519142576":"2024-04-27T23:34:58.266218+0000","172.21.15.72:0/3669626460":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/945623297":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/3479150073":"2024-04-27T23:34:24.233319+0000","172.21.15.72:6800/4175896290":"2024-04-27T23:34:24.233319+0000","172.21.15.72:6801/4175896290":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/3783175012":"2024-04-27T23:34:58.266218+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:43:10.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.840+0000 7f54b1000640 1 -- 172.21.15.72:0/916550265 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f54900a3bb0 msgr2=0x7f54900a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:10.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.840+0000 7f54b1000640 1 --2- 172.21.15.72:0/916550265 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f54900a3bb0 0x7f54900a6070 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f54ac19c2e0 tx=0x7f5498005f90 comp rx=0 tx=0).stop 2024-04-26T23:43:10.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.840+0000 7f54b1000640 1 -- 172.21.15.72:0/916550265 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54ac104b90 msgr2=0x7f54ac19add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:10.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.840+0000 7f54b1000640 1 --2- 172.21.15.72:0/916550265 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54ac104b90 0x7f54ac19add0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f549c031500 tx=0x7f549c003c10 comp rx=0 tx=0).stop 2024-04-26T23:43:10.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.840+0000 7f54b1000640 1 -- 172.21.15.72:0/916550265 shutdown_connections 2024-04-26T23:43:10.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.840+0000 7f54b1000640 1 --2- 172.21.15.72:0/916550265 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f54900a3bb0 0x7f54900a6070 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:10.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.840+0000 7f54b1000640 1 --2- 172.21.15.72:0/916550265 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f54ac106740 0x7f54ac1a2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:10.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.840+0000 7f54b1000640 1 --2- 172.21.15.72:0/916550265 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f54ac105d80 0x7f54ac19b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:10.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.840+0000 7f54b1000640 1 --2- 172.21.15.72:0/916550265 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54ac104b90 0x7f54ac19add0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:10.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.840+0000 7f54b1000640 1 -- 172.21.15.72:0/916550265 >> 172.21.15.72:0/916550265 conn(0x7f54ac100120 msgr2=0x7f54ac109100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:10.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.840+0000 7f54b1000640 1 -- 172.21.15.72:0/916550265 shutdown_connections 2024-04-26T23:43:10.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:10.840+0000 7f54b1000640 1 -- 172.21.15.72:0/916550265 wait complete. 2024-04-26T23:43:10.862 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-26T23:43:02.928649+0000', 'modified': '2024-04-26T23:43:07.991051+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24590}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24590': {'gid': 24590, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.76:6839/3405829058', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6838', 'nonce': 3405829058}, {'type': 'v1', 'addr': '172.21.15.76:6839', 'nonce': 3405829058}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24590, 'qdb_cluster': [24590]} 2024-04-26T23:43:10.863 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-26T23:43:10.863 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-26T23:43:10.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:10 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/3723235722' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:43:10.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:10 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/2441519532' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:43:11.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:10 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/3723235722' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:43:11.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:10 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/2441519532' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:43:11.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.052+0000 7f9906e00640 1 -- 172.21.15.72:0/69732696 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9900078530 msgr2=0x7f9900078990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:11.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.052+0000 7f9906e00640 1 --2- 172.21.15.72:0/69732696 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9900078530 0x7f9900078990 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f98ec009860 tx=0x7f98ec02a340 comp rx=0 tx=0).stop 2024-04-26T23:43:11.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.052+0000 7f9906e00640 1 -- 172.21.15.72:0/69732696 shutdown_connections 2024-04-26T23:43:11.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.052+0000 7f9906e00640 1 --2- 172.21.15.72:0/69732696 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9900078ed0 0x7f9900079770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:11.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.052+0000 7f9906e00640 1 --2- 172.21.15.72:0/69732696 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9900078530 0x7f9900078990 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:11.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.052+0000 7f9906e00640 1 --2- 172.21.15.72:0/69732696 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99000772f0 0x7f9900077710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:11.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.052+0000 7f9906e00640 1 -- 172.21.15.72:0/69732696 >> 172.21.15.72:0/69732696 conn(0x7f99000ffec0 msgr2=0x7f9900102320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:11.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.052+0000 7f9906e00640 1 -- 172.21.15.72:0/69732696 shutdown_connections 2024-04-26T23:43:11.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.052+0000 7f9906e00640 1 -- 172.21.15.72:0/69732696 wait complete. 2024-04-26T23:43:11.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.053+0000 7f9906e00640 1 Processor -- start 2024-04-26T23:43:11.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.053+0000 7f9906e00640 1 -- start start 2024-04-26T23:43:11.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.053+0000 7f9906e00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99000772f0 0x7f990019ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:11.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.053+0000 7f9906e00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9900078530 0x7f990019f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:11.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.053+0000 7f9906e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9900078ed0 0x7f99001a6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:11.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.053+0000 7f9906e00640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f99001148b0 con 0x7f99000772f0 2024-04-26T23:43:11.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.053+0000 7f9906e00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f9900114730 con 0x7f9900078ed0 2024-04-26T23:43:11.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.053+0000 7f9906e00640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f9900114a30 con 0x7f9900078530 2024-04-26T23:43:11.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.054+0000 7f9905a00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99000772f0 0x7f990019ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:11.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.054+0000 7f9905a00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99000772f0 0x7f990019ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43334/0 (socket says 172.21.15.72:43334) 2024-04-26T23:43:11.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.054+0000 7f9905a00640 1 -- 172.21.15.72:0/1492450399 learned_addr learned my addr 172.21.15.72:0/1492450399 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:43:11.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.054+0000 7f9905000640 1 --2- 172.21.15.72:0/1492450399 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9900078530 0x7f990019f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:11.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.054+0000 7f9906400640 1 --2- 172.21.15.72:0/1492450399 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9900078ed0 0x7f99001a6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:11.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.054+0000 7f9905a00640 1 -- 172.21.15.72:0/1492450399 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9900078530 msgr2=0x7f990019f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:11.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.054+0000 7f9905a00640 1 --2- 172.21.15.72:0/1492450399 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9900078530 0x7f990019f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:11.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.054+0000 7f9905a00640 1 -- 172.21.15.72:0/1492450399 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9900078ed0 msgr2=0x7f99001a6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:11.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.054+0000 7f9905a00640 1 --2- 172.21.15.72:0/1492450399 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9900078ed0 0x7f99001a6530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:11.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.054+0000 7f9905a00640 1 -- 172.21.15.72:0/1492450399 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99001a6ba0 con 0x7f99000772f0 2024-04-26T23:43:11.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.054+0000 7f9906400640 1 --2- 172.21.15.72:0/1492450399 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9900078ed0 0x7f99001a6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:43:11.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.054+0000 7f9905a00640 1 --2- 172.21.15.72:0/1492450399 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99000772f0 0x7f990019ef80 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f98f000e9c0 tx=0x7f98f000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:11.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.055+0000 7f98fe000640 1 -- 172.21.15.72:0/1492450399 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f000cde0 con 0x7f99000772f0 2024-04-26T23:43:11.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.055+0000 7f98fe000640 1 -- 172.21.15.72:0/1492450399 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98f00035f0 con 0x7f99000772f0 2024-04-26T23:43:11.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.055+0000 7f9906e00640 1 -- 172.21.15.72:0/1492450399 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99001a6e90 con 0x7f99000772f0 2024-04-26T23:43:11.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.055+0000 7f98fe000640 1 -- 172.21.15.72:0/1492450399 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f0010690 con 0x7f99000772f0 2024-04-26T23:43:11.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.055+0000 7f9906e00640 1 -- 172.21.15.72:0/1492450399 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99001a72f0 con 0x7f99000772f0 2024-04-26T23:43:11.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.058+0000 7f98fe000640 1 -- 172.21.15.72:0/1492450399 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f98f0004000 con 0x7f99000772f0 2024-04-26T23:43:11.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.058+0000 7f990b167740 1 Processor -- start 2024-04-26T23:43:11.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.058+0000 7f990b167740 1 -- start start 2024-04-26T23:43:11.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.058+0000 7f990b167740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ea92e702c0 0x55ea92e706e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:11.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.058+0000 7f990b167740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ea92e70cb0 0x55ea92e757b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:11.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.058+0000 7f990b167740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ea92e75e20 0x55ea92e782b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:11.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.058+0000 7f990b167740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55ea92dd4600 con 0x55ea92e75e20 2024-04-26T23:43:11.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.058+0000 7f990b167740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55ea92d71100 con 0x55ea92e70cb0 2024-04-26T23:43:11.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.058+0000 7f990b167740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55ea92dd05c0 con 0x55ea92e702c0 2024-04-26T23:43:11.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.058+0000 7f9906400640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ea92e702c0 0x55ea92e706e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:11.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.058+0000 7f9905000640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ea92e75e20 0x55ea92e782b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:11.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.058+0000 7f9906400640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ea92e702c0 0x55ea92e706e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.72:35840/0 (socket says 172.21.15.72:35840) 2024-04-26T23:43:11.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.058+0000 7f9906400640 1 -- 172.21.15.72:0/247936321 learned_addr learned my addr 172.21.15.72:0/247936321 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:43:11.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.059+0000 7f9905a00640 1 --2- 172.21.15.72:0/247936321 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ea92e70cb0 0x55ea92e757b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:11.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.059+0000 7f9906400640 1 -- 172.21.15.72:0/247936321 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ea92e70cb0 msgr2=0x55ea92e757b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:43:11.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.059+0000 7f9906400640 1 --2- 172.21.15.72:0/247936321 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ea92e70cb0 0x55ea92e757b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:11.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.059+0000 7f9906400640 1 -- 172.21.15.72:0/247936321 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ea92e75e20 msgr2=0x55ea92e782b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:43:11.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.059+0000 7f9906400640 1 --2- 172.21.15.72:0/247936321 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ea92e75e20 0x55ea92e782b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:11.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.059+0000 7f9906400640 1 -- 172.21.15.72:0/247936321 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea92ae62d0 con 0x55ea92e702c0 2024-04-26T23:43:11.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.059+0000 7f9905000640 1 --2- 172.21.15.72:0/247936321 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ea92e75e20 0x55ea92e782b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:43:11.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.059+0000 7f9906400640 1 --2- 172.21.15.72:0/247936321 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ea92e702c0 0x55ea92e706e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f98f8009fd0 tx=0x7f98f80108e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4dc15a656b276832 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:11.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.059+0000 7f98fe000640 1 --2- 172.21.15.72:0/1492450399 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f98e40a3bb0 0x7f98e40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:11.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.059+0000 7f98f6a00640 1 -- 172.21.15.72:0/247936321 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f8011760 con 0x55ea92e702c0 2024-04-26T23:43:11.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.059+0000 7f98f6a00640 1 -- 172.21.15.72:0/247936321 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98f8011db0 con 0x55ea92e702c0 2024-04-26T23:43:11.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.059+0000 7f98f6a00640 1 -- 172.21.15.72:0/247936321 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f800e3c0 con 0x55ea92e702c0 2024-04-26T23:43:11.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.060+0000 7f98fe000640 1 -- 172.21.15.72:0/1492450399 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f98f00d0a40 con 0x7f99000772f0 2024-04-26T23:43:11.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.060+0000 7f9905a00640 1 --2- 172.21.15.72:0/1492450399 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f98e40a3bb0 0x7f98e40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:11.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.060+0000 7f990b167740 1 -- 172.21.15.72:0/247936321 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ea92e702c0 msgr2=0x55ea92e706e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:11.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.060+0000 7f990b167740 1 --2- 172.21.15.72:0/247936321 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ea92e702c0 0x55ea92e706e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f98f8009fd0 tx=0x7f98f80108e0 comp rx=0 tx=0).stop 2024-04-26T23:43:11.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.060+0000 7f9905a00640 1 --2- 172.21.15.72:0/1492450399 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f98e40a3bb0 0x7f98e40a6070 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f98f0099450 tx=0x7f98f00d6530 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:11.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.060+0000 7f990b167740 1 -- 172.21.15.72:0/247936321 shutdown_connections 2024-04-26T23:43:11.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.060+0000 7f990b167740 1 --2- 172.21.15.72:0/247936321 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ea92e70cb0 0x55ea92e757b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:11.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.060+0000 7f990b167740 1 --2- 172.21.15.72:0/247936321 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ea92e75e20 0x55ea92e782b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:11.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.060+0000 7f990b167740 1 --2- 172.21.15.72:0/247936321 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ea92e702c0 0x55ea92e706e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:11.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.060+0000 7f990b167740 1 -- 172.21.15.72:0/247936321 >> 172.21.15.72:0/247936321 conn(0x55ea92e04d10 msgr2=0x55ea92e6bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:11.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.060+0000 7f990b167740 1 -- 172.21.15.72:0/247936321 shutdown_connections 2024-04-26T23:43:11.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.060+0000 7f990b167740 1 -- 172.21.15.72:0/247936321 wait complete. 2024-04-26T23:43:11.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.061+0000 7f990b167740 1 -- start start 2024-04-26T23:43:11.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.061+0000 7f990b167740 1 Processor -- start 2024-04-26T23:43:11.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.061+0000 7f990b167740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ea92e702c0 0x55ea92e7a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:11.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.061+0000 7f990b167740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ea92e70cb0 0x55ea92e7ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:11.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.061+0000 7f990b167740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ea92e75e20 0x55ea92e81d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:11.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.061+0000 7f990b167740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55ea92e78dd0 con 0x55ea92e70cb0 2024-04-26T23:43:11.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.061+0000 7f990b167740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55ea92d5bf70 con 0x55ea92e702c0 2024-04-26T23:43:11.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.061+0000 7f990b167740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55ea92e78f50 con 0x55ea92e75e20 2024-04-26T23:43:11.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.061+0000 7f990b167740 10 client.0 _ll_register_callbacks cb 0x55ea92dca480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:43:11.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.062+0000 7f9906400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ea92e702c0 0x55ea92e7a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:11.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.062+0000 7f9905000640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ea92e70cb0 0x55ea92e7ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:11.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.062+0000 7f9906400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ea92e702c0 0x55ea92e7a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:34804/0 (socket says 172.21.15.72:34804) 2024-04-26T23:43:11.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.062+0000 7f9906400640 1 -- 172.21.15.72:0/385310652 learned_addr learned my addr 172.21.15.72:0/385310652 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:43:11.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.062+0000 7f9906400640 1 --2- 172.21.15.72:0/385310652 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ea92e75e20 0x55ea92e81d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:11.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.062+0000 7f9905000640 1 -- 172.21.15.72:0/385310652 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ea92e75e20 msgr2=0x55ea92e81d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:43:11.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.062+0000 7f9905000640 1 --2- 172.21.15.72:0/385310652 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ea92e75e20 0x55ea92e81d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:11.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.062+0000 7f9905000640 1 -- 172.21.15.72:0/385310652 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ea92e702c0 msgr2=0x55ea92e7a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:43:11.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.062+0000 7f9905000640 1 --2- 172.21.15.72:0/385310652 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ea92e702c0 0x55ea92e7a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:11.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.062+0000 7f9905000640 1 -- 172.21.15.72:0/385310652 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea92bb06b0 con 0x55ea92e70cb0 2024-04-26T23:43:11.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.062+0000 7f9905000640 1 --2- 172.21.15.72:0/385310652 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ea92e70cb0 0x55ea92e7ace0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f98ec009830 tx=0x7f98ec030780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=595553ccfc8d827d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:11.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.062+0000 7f98eb400640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-04-26T23:43:11.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.063+0000 7f98eb400640 1 -- 172.21.15.72:0/385310652 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ec041070 con 0x55ea92e70cb0 2024-04-26T23:43:11.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.063+0000 7f98eb400640 1 -- 172.21.15.72:0/385310652 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f98ec004240 con 0x55ea92e70cb0 2024-04-26T23:43:11.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.063+0000 7f990b167740 1 -- 172.21.15.72:0/385310652 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98f8011db0 con 0x55ea92e70cb0 2024-04-26T23:43:11.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.063+0000 7f98eb400640 1 -- 172.21.15.72:0/385310652 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f98ec04d050 con 0x55ea92e70cb0 2024-04-26T23:43:11.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.063+0000 7f990b167740 10 client.14592 fetch_fsmap learned FSMap version 17 2024-04-26T23:43:11.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.063+0000 7f990b167740 1 -- 172.21.15.72:0/385310652 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x55ea92e84d70 con 0x55ea92e70cb0 2024-04-26T23:43:11.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.063+0000 7f98eb400640 1 -- 172.21.15.72:0/385310652 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98f8011db0 con 0x55ea92e70cb0 2024-04-26T23:43:11.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.063+0000 7f98eb400640 1 -- 172.21.15.72:0/385310652 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7f98ec03d450 con 0x55ea92e70cb0 2024-04-26T23:43:11.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.063+0000 7f990b167740 10 client.14592 fetch_fsmap finished waiting for FSMap version 17 2024-04-26T23:43:11.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.063+0000 7f990b167740 10 client.14592 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T23:43:11.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.063+0000 7f990b167740 20 client.14592 populate_metadata read hostname 'smithi072' 2024-04-26T23:43:11.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.064+0000 7f990b167740 1 --2- 172.21.15.72:0/385310652 >> [v2:172.21.15.76:6838/3405829058,v1:172.21.15.76:6839/3405829058] conn(0x55ea92e873a0 0x55ea92e88810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:11.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.064+0000 7f990b167740 4 client.14592 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:43:11.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.064+0000 7f990b167740 1 -- 172.21.15.72:0/385310652 --> [v2:172.21.15.76:6838/3405829058,v1:172.21.15.76:6839/3405829058] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ea92de2210 con 0x55ea92e873a0 2024-04-26T23:43:11.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.064+0000 7f9906400640 1 --2- 172.21.15.72:0/385310652 >> [v2:172.21.15.76:6838/3405829058,v1:172.21.15.76:6839/3405829058] conn(0x55ea92e873a0 0x55ea92e88810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:11.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.064+0000 7f9906400640 1 --2- 172.21.15.72:0/385310652 >> [v2:172.21.15.76:6838/3405829058,v1:172.21.15.76:6839/3405829058] conn(0x55ea92e873a0 0x55ea92e88810 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2343cc1ae30b36a1 server_cookie=4cd7bbff1b912a21 in_seq=0 out_seq=0 2024-04-26T23:43:11.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.064+0000 7f98eb400640 10 client.14592 ms_handle_connect on v2:172.21.15.76:6838/3405829058 2024-04-26T23:43:11.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.065+0000 7f98eb400640 1 -- 172.21.15.72:0/385310652 <== mds.0 v2:172.21.15.76:6838/3405829058 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ea92de2210 con 0x55ea92e873a0 2024-04-26T23:43:11.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.065+0000 7f98eb400640 10 client.14592 handle_command_reply: tid=0 2024-04-26T23:43:11.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.065+0000 7f990b167740 1 client.14592 shutdown 2024-04-26T23:43:11.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.065+0000 7f990b167740 1 -- 172.21.15.72:0/385310652 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ea92e70cb0 msgr2=0x55ea92e7ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:11.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.065+0000 7f990b167740 1 --2- 172.21.15.72:0/385310652 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ea92e70cb0 0x55ea92e7ace0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f98ec009830 tx=0x7f98ec030780 comp rx=0 tx=0).stop 2024-04-26T23:43:11.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.065+0000 7f990b167740 1 -- 172.21.15.72:0/385310652 shutdown_connections 2024-04-26T23:43:11.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.065+0000 7f990b167740 1 --2- 172.21.15.72:0/385310652 >> [v2:172.21.15.76:6838/3405829058,v1:172.21.15.76:6839/3405829058] conn(0x55ea92e873a0 0x55ea92e88810 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:11.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.065+0000 7f990b167740 1 --2- 172.21.15.72:0/385310652 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ea92e75e20 0x55ea92e81d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:11.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.065+0000 7f990b167740 1 --2- 172.21.15.72:0/385310652 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ea92e70cb0 0x55ea92e7ace0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:11.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.065+0000 7f990b167740 1 --2- 172.21.15.72:0/385310652 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ea92e702c0 0x55ea92e7a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:11.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.065+0000 7f98eb400640 0 client.14592 ms_handle_reset on v2:172.21.15.76:6838/3405829058 2024-04-26T23:43:11.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.065+0000 7f990b167740 1 -- 172.21.15.72:0/385310652 >> 172.21.15.72:0/385310652 conn(0x55ea92e04d10 msgr2=0x55ea92e6af60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:11.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.066+0000 7f990b167740 1 -- 172.21.15.72:0/385310652 shutdown_connections 2024-04-26T23:43:11.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.066+0000 7f990b167740 1 -- 172.21.15.72:0/385310652 wait complete. 2024-04-26T23:43:11.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.066+0000 7f990b167740 20 client.14592 trim_cache size 0 max 16384 2024-04-26T23:43:11.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.085+0000 7f990b167740 1 Processor -- start 2024-04-26T23:43:11.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.085+0000 7f990b167740 1 -- start start 2024-04-26T23:43:11.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.085+0000 7f990b167740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ea92e702c0 0x55ea92e66df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:11.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.085+0000 7f990b167740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ea92e70cb0 0x55ea92e81ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:11.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.085+0000 7f990b167740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ea92e75e20 0x55ea92e72110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:11.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.085+0000 7f990b167740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55ea92e86b50 con 0x55ea92e702c0 2024-04-26T23:43:11.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.085+0000 7f990b167740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55ea92e87220 con 0x55ea92e70cb0 2024-04-26T23:43:11.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.085+0000 7f990b167740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55ea92e878f0 con 0x55ea92e75e20 2024-04-26T23:43:11.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.085+0000 7f9905000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ea92e70cb0 0x55ea92e81ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:11.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.085+0000 7f9906400640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ea92e702c0 0x55ea92e66df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:11.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.085+0000 7f9905000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ea92e70cb0 0x55ea92e81ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:34820/0 (socket says 172.21.15.72:34820) 2024-04-26T23:43:11.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.085+0000 7f9905000640 1 -- 172.21.15.72:0/1704211032 learned_addr learned my addr 172.21.15.72:0/1704211032 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:43:11.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.086+0000 7f9905000640 1 -- 172.21.15.72:0/1704211032 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ea92e75e20 msgr2=0x55ea92e72110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:43:11.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.086+0000 7f9905000640 1 --2- 172.21.15.72:0/1704211032 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ea92e75e20 0x55ea92e72110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:11.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.086+0000 7f9905000640 1 -- 172.21.15.72:0/1704211032 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ea92e702c0 msgr2=0x55ea92e66df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:43:11.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.086+0000 7f9905000640 1 --2- 172.21.15.72:0/1704211032 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ea92e702c0 0x55ea92e66df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:11.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.086+0000 7f9905000640 1 -- 172.21.15.72:0/1704211032 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea92e67680 con 0x55ea92e70cb0 2024-04-26T23:43:11.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.086+0000 7f9906400640 1 --2- 172.21.15.72:0/1704211032 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ea92e702c0 0x55ea92e66df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:43:11.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.086+0000 7f9905000640 1 --2- 172.21.15.72:0/1704211032 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ea92e70cb0 0x55ea92e81ce0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f98ec03da10 tx=0x7f98ec0096b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1153d7c8e9d3e52 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:11.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.086+0000 7f98f6000640 1 -- 172.21.15.72:0/1704211032 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ec041070 con 0x55ea92e70cb0 2024-04-26T23:43:11.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.086+0000 7f98f6000640 1 -- 172.21.15.72:0/1704211032 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98ec03da90 con 0x55ea92e70cb0 2024-04-26T23:43:11.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.086+0000 7f98f6000640 1 -- 172.21.15.72:0/1704211032 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ec037040 con 0x55ea92e70cb0 2024-04-26T23:43:11.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.087+0000 7f990b167740 1 -- 172.21.15.72:0/1704211032 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ea92e70cb0 msgr2=0x55ea92e81ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:11.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.087+0000 7f990b167740 1 --2- 172.21.15.72:0/1704211032 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ea92e70cb0 0x55ea92e81ce0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f98ec03da10 tx=0x7f98ec0096b0 comp rx=0 tx=0).stop 2024-04-26T23:43:11.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.087+0000 7f990b167740 1 -- 172.21.15.72:0/1704211032 shutdown_connections 2024-04-26T23:43:11.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.087+0000 7f990b167740 1 --2- 172.21.15.72:0/1704211032 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ea92e70cb0 0x55ea92e81ce0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:11.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.087+0000 7f990b167740 1 --2- 172.21.15.72:0/1704211032 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ea92e75e20 0x55ea92e72110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:11.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.087+0000 7f990b167740 1 --2- 172.21.15.72:0/1704211032 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ea92e702c0 0x55ea92e66df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:11.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.087+0000 7f990b167740 1 -- 172.21.15.72:0/1704211032 >> 172.21.15.72:0/1704211032 conn(0x55ea92e04d10 msgr2=0x55ea92e6b7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:11.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.087+0000 7f990b167740 1 -- 172.21.15.72:0/1704211032 shutdown_connections 2024-04-26T23:43:11.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.087+0000 7f990b167740 1 -- 172.21.15.72:0/1704211032 wait complete. 2024-04-26T23:43:11.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.087+0000 7f990b167740 1 -- start start 2024-04-26T23:43:11.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.088+0000 7f990b167740 1 Processor -- start 2024-04-26T23:43:11.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.088+0000 7f990b167740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ea92e702c0 0x55ea92e7d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:11.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.088+0000 7f990b167740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ea92e70cb0 0x55ea92e7d950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:11.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.088+0000 7f990b167740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ea92e75e20 0x55ea92e80890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:11.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.088+0000 7f990b167740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55ea92e88690 con 0x55ea92e702c0 2024-04-26T23:43:11.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.088+0000 7f990b167740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55ea92e87fc0 con 0x55ea92e75e20 2024-04-26T23:43:11.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.088+0000 7f990b167740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55ea92e88d60 con 0x55ea92e70cb0 2024-04-26T23:43:11.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.088+0000 7f990b167740 10 client.0 _ll_register_callbacks cb 0x7f98f0005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:43:11.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.088+0000 7f9906400640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ea92e702c0 0x55ea92e7d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:11.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.088+0000 7f9906400640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ea92e702c0 0x55ea92e7d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43368/0 (socket says 172.21.15.72:43368) 2024-04-26T23:43:11.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.088+0000 7f9906400640 1 -- 172.21.15.72:0/4053762373 learned_addr learned my addr 172.21.15.72:0/4053762373 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:43:11.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.088+0000 7f9906400640 1 --2- 172.21.15.72:0/4053762373 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ea92e75e20 0x55ea92e80890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:11.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.088+0000 7f9905000640 1 --2- 172.21.15.72:0/4053762373 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ea92e70cb0 0x55ea92e7d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:11.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.088+0000 7f9906400640 1 -- 172.21.15.72:0/4053762373 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ea92e70cb0 msgr2=0x55ea92e7d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:43:11.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.088+0000 7f9906400640 1 --2- 172.21.15.72:0/4053762373 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ea92e70cb0 0x55ea92e7d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:11.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.088+0000 7f9906400640 1 -- 172.21.15.72:0/4053762373 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ea92e75e20 msgr2=0x55ea92e80890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:43:11.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.088+0000 7f9906400640 1 --2- 172.21.15.72:0/4053762373 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ea92e75e20 0x55ea92e80890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:11.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.088+0000 7f9906400640 1 -- 172.21.15.72:0/4053762373 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea92e80e80 con 0x55ea92e702c0 2024-04-26T23:43:11.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.088+0000 7f9905000640 1 --2- 172.21.15.72:0/4053762373 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ea92e70cb0 0x55ea92e7d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:43:11.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.089+0000 7f9906400640 1 --2- 172.21.15.72:0/4053762373 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ea92e702c0 0x55ea92e7d410 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f98f8011010 tx=0x7f98f8011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b24989392a8f939 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:11.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.089+0000 7f98eb400640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-04-26T23:43:11.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.089+0000 7f98eb400640 1 -- 172.21.15.72:0/4053762373 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f801c880 con 0x55ea92e702c0 2024-04-26T23:43:11.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.089+0000 7f98eb400640 1 -- 172.21.15.72:0/4053762373 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f98f801ca20 con 0x55ea92e702c0 2024-04-26T23:43:11.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.089+0000 7f990b167740 1 -- 172.21.15.72:0/4053762373 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98ec03da90 con 0x55ea92e702c0 2024-04-26T23:43:11.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.089+0000 7f98eb400640 1 -- 172.21.15.72:0/4053762373 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f98f8011560 con 0x55ea92e702c0 2024-04-26T23:43:11.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.089+0000 7f990b167740 10 client.14598 fetch_fsmap learned FSMap version 17 2024-04-26T23:43:11.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.089+0000 7f990b167740 1 -- 172.21.15.72:0/4053762373 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x55ea92eed350 con 0x55ea92e702c0 2024-04-26T23:43:11.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.090+0000 7f98eb400640 1 -- 172.21.15.72:0/4053762373 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98ec03da90 con 0x55ea92e702c0 2024-04-26T23:43:11.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.090+0000 7f98eb400640 1 -- 172.21.15.72:0/4053762373 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7f98f8029040 con 0x55ea92e702c0 2024-04-26T23:43:11.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.090+0000 7f990b167740 10 client.14598 fetch_fsmap finished waiting for FSMap version 17 2024-04-26T23:43:11.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.090+0000 7f990b167740 10 client.14598 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T23:43:11.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.090+0000 7f990b167740 20 client.14598 populate_metadata read hostname 'smithi072' 2024-04-26T23:43:11.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.090+0000 7f990b167740 1 --2- 172.21.15.72:0/4053762373 >> [v2:172.21.15.76:6838/3405829058,v1:172.21.15.76:6839/3405829058] conn(0x55ea92ef1c20 0x55ea92eee730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:11.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.090+0000 7f990b167740 4 client.14598 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T23:43:11.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.090+0000 7f990b167740 1 -- 172.21.15.72:0/4053762373 --> [v2:172.21.15.76:6838/3405829058,v1:172.21.15.76:6839/3405829058] -- command(tid 0: {"prefix": "status"}) -- 0x55ea92ddc080 con 0x55ea92ef1c20 2024-04-26T23:43:11.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.090+0000 7f9905000640 1 --2- 172.21.15.72:0/4053762373 >> [v2:172.21.15.76:6838/3405829058,v1:172.21.15.76:6839/3405829058] conn(0x55ea92ef1c20 0x55ea92eee730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:11.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.091+0000 7f98eb400640 10 client.14598 ms_handle_connect on v2:172.21.15.76:6838/3405829058 2024-04-26T23:43:11.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.091+0000 7f9905000640 1 --2- 172.21.15.72:0/4053762373 >> [v2:172.21.15.76:6838/3405829058,v1:172.21.15.76:6839/3405829058] conn(0x55ea92ef1c20 0x55ea92eee730 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4cfd018a38ed6575 server_cookie=2613a4220eab822f in_seq=0 out_seq=0 2024-04-26T23:43:11.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.091+0000 7f98eb400640 1 -- 172.21.15.72:0/4053762373 <== mds.0 v2:172.21.15.76:6838/3405829058 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55ea92ddc080 con 0x55ea92ef1c20 2024-04-26T23:43:11.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.091+0000 7f98eb400640 10 client.14598 handle_command_reply: tid=0 2024-04-26T23:43:11.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.091+0000 7f990b167740 1 client.14598 shutdown 2024-04-26T23:43:11.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.091+0000 7f990b167740 1 -- 172.21.15.72:0/4053762373 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ea92e702c0 msgr2=0x55ea92e7d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:11.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.091+0000 7f990b167740 1 --2- 172.21.15.72:0/4053762373 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ea92e702c0 0x55ea92e7d410 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f98f8011010 tx=0x7f98f8011f60 comp rx=0 tx=0).stop 2024-04-26T23:43:11.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.091+0000 7f990b167740 1 -- 172.21.15.72:0/4053762373 shutdown_connections 2024-04-26T23:43:11.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.091+0000 7f990b167740 1 --2- 172.21.15.72:0/4053762373 >> [v2:172.21.15.76:6838/3405829058,v1:172.21.15.76:6839/3405829058] conn(0x55ea92ef1c20 0x55ea92eee730 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:11.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.091+0000 7f990b167740 1 --2- 172.21.15.72:0/4053762373 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55ea92e75e20 0x55ea92e80890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:11.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.091+0000 7f98eb400640 0 client.14598 ms_handle_reset on v2:172.21.15.76:6838/3405829058 2024-04-26T23:43:11.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.091+0000 7f990b167740 1 --2- 172.21.15.72:0/4053762373 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ea92e70cb0 0x55ea92e7d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:11.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.091+0000 7f990b167740 1 --2- 172.21.15.72:0/4053762373 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55ea92e702c0 0x55ea92e7d410 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:11.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.091+0000 7f990b167740 1 -- 172.21.15.72:0/4053762373 >> 172.21.15.72:0/4053762373 conn(0x55ea92e04d10 msgr2=0x55ea92e6b7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:11.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.091+0000 7f990b167740 1 -- 172.21.15.72:0/4053762373 shutdown_connections 2024-04-26T23:43:11.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.091+0000 7f990b167740 1 -- 172.21.15.72:0/4053762373 wait complete. 2024-04-26T23:43:11.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.091+0000 7f990b167740 20 client.14598 trim_cache size 0 max 16384 2024-04-26T23:43:11.100 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-04-26T23:43:11.100 INFO:teuthology.orchestra.run.smithi072.stdout: "cluster_fsid": "3c8453ae-0425-11ef-bc93-c7b262605968", 2024-04-26T23:43:11.100 INFO:teuthology.orchestra.run.smithi072.stdout: "whoami": 0, 2024-04-26T23:43:11.100 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 24590, 2024-04-26T23:43:11.100 INFO:teuthology.orchestra.run.smithi072.stdout: "want_state": "up:active", 2024-04-26T23:43:11.100 INFO:teuthology.orchestra.run.smithi072.stdout: "state": "up:active", 2024-04-26T23:43:11.100 INFO:teuthology.orchestra.run.smithi072.stdout: "fs_name": "cephfs", 2024-04-26T23:43:11.100 INFO:teuthology.orchestra.run.smithi072.stdout: "rank_uptime": 7.6303852709999997, 2024-04-26T23:43:11.100 INFO:teuthology.orchestra.run.smithi072.stdout: "mdsmap_epoch": 17, 2024-04-26T23:43:11.101 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_epoch": 89, 2024-04-26T23:43:11.101 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_epoch_barrier": 89, 2024-04-26T23:43:11.101 INFO:teuthology.orchestra.run.smithi072.stdout: "uptime": 14.129775262000001 2024-04-26T23:43:11.101 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-04-26T23:43:11.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.092+0000 7f9906e00640 1 -- 172.21.15.72:0/1492450399 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f98e40a3bb0 msgr2=0x7f98e40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:11.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.092+0000 7f9906e00640 1 --2- 172.21.15.72:0/1492450399 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f98e40a3bb0 0x7f98e40a6070 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f98f0099450 tx=0x7f98f00d6530 comp rx=0 tx=0).stop 2024-04-26T23:43:11.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.092+0000 7f9906e00640 1 -- 172.21.15.72:0/1492450399 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99000772f0 msgr2=0x7f990019ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:11.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.092+0000 7f9906e00640 1 --2- 172.21.15.72:0/1492450399 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99000772f0 0x7f990019ef80 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f98f000e9c0 tx=0x7f98f000ee80 comp rx=0 tx=0).stop 2024-04-26T23:43:11.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.092+0000 7f9906e00640 1 -- 172.21.15.72:0/1492450399 shutdown_connections 2024-04-26T23:43:11.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.092+0000 7f9906e00640 1 --2- 172.21.15.72:0/1492450399 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f98e40a3bb0 0x7f98e40a6070 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:11.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.092+0000 7f9906e00640 1 --2- 172.21.15.72:0/1492450399 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9900078ed0 0x7f99001a6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:11.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.092+0000 7f9906e00640 1 --2- 172.21.15.72:0/1492450399 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9900078530 0x7f990019f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:11.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.092+0000 7f9906e00640 1 --2- 172.21.15.72:0/1492450399 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99000772f0 0x7f990019ef80 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:11.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.092+0000 7f9906e00640 1 -- 172.21.15.72:0/1492450399 >> 172.21.15.72:0/1492450399 conn(0x7f99000ffec0 msgr2=0x7f9900103240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:11.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.092+0000 7f9906e00640 1 -- 172.21.15.72:0/1492450399 shutdown_connections 2024-04-26T23:43:11.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:11.092+0000 7f9906e00640 1 -- 172.21.15.72:0/1492450399 wait complete. 2024-04-26T23:43:11.111 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-26T23:43:11.111 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-26T23:43:11.111 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-26T23:43:11.520 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:43:11.852 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:11 smithi072 ceph-mon[28184]: pgmap v325: 129 pgs: 2 creating+peering, 119 active+clean, 8 unknown; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 416 B/s wr, 5 op/s 2024-04-26T23:43:11.852 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:11 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/916550265' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:43:11.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:11 smithi076 ceph-mon[32774]: pgmap v325: 129 pgs: 2 creating+peering, 119 active+clean, 8 unknown; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 416 B/s wr, 5 op/s 2024-04-26T23:43:11.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:11 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/916550265' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:43:12.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:11 smithi136 ceph-mon[33050]: pgmap v325: 129 pgs: 2 creating+peering, 119 active+clean, 8 unknown; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 416 B/s wr, 5 op/s 2024-04-26T23:43:12.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:11 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/916550265' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:43:13.712 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:13 smithi072 ceph-mon[28184]: pgmap v326: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:43:13.852 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:13 smithi136 ceph-mon[33050]: pgmap v326: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:43:13.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:13.877+0000 7f0037e00700 1 -- 172.21.15.72:0/3713332963 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0030105640 msgr2=0x7f0030105ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:13.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:13.877+0000 7f0037e00700 1 --2- 172.21.15.72:0/3713332963 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0030105640 0x7f0030105ac0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f001c009990 tx=0x7f001c009ca0 comp rx=0 tx=0).stop 2024-04-26T23:43:13.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:13.877+0000 7f0037e00700 1 -- 172.21.15.72:0/3713332963 shutdown_connections 2024-04-26T23:43:13.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:13.877+0000 7f0037e00700 1 --2- 172.21.15.72:0/3713332963 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0030106000 0x7f0030112510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:13.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:13.877+0000 7f0037e00700 1 --2- 172.21.15.72:0/3713332963 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f003010b640 0x7f003010ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:13.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:13.877+0000 7f0037e00700 1 --2- 172.21.15.72:0/3713332963 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0030105640 0x7f0030105ac0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:13.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:13.877+0000 7f0037e00700 1 -- 172.21.15.72:0/3713332963 >> 172.21.15.72:0/3713332963 conn(0x7f0030101130 msgr2=0x7f0030103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:13.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:13.878+0000 7f0037e00700 1 -- 172.21.15.72:0/3713332963 shutdown_connections 2024-04-26T23:43:13.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:13.878+0000 7f0037e00700 1 -- 172.21.15.72:0/3713332963 wait complete. 2024-04-26T23:43:13.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:13.879+0000 7f0037e00700 1 Processor -- start 2024-04-26T23:43:13.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:13.879+0000 7f0037e00700 1 -- start start 2024-04-26T23:43:13.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:13.879+0000 7f0037e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0030105640 0x7f00301a1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:13.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:13.879+0000 7f0037e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0030106000 0x7f00301a1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:13.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:13.879+0000 7f0037e00700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f003010b640 0x7f003019bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:13.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:13.880+0000 7f0037e00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f0030115880 con 0x7f0030106000 2024-04-26T23:43:13.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:13.880+0000 7f0037e00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f0030115700 con 0x7f0030105640 2024-04-26T23:43:13.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:13.880+0000 7f0037e00700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f0030115a00 con 0x7f003010b640 2024-04-26T23:43:13.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:13.880+0000 7f0034a00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0030106000 0x7f00301a1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:13.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:13.880+0000 7f0034a00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0030106000 0x7f00301a1f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43384/0 (socket says 172.21.15.72:43384) 2024-04-26T23:43:13.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:13.880+0000 7f0034a00700 1 -- 172.21.15.72:0/3416587569 learned_addr learned my addr 172.21.15.72:0/3416587569 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:43:13.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:13.880+0000 7f0035400700 1 --2- 172.21.15.72:0/3416587569 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0030105640 0x7f00301a1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:13.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:13.880+0000 7f0035e00700 1 --2- 172.21.15.72:0/3416587569 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f003010b640 0x7f003019bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:13.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:13.880+0000 7f0034a00700 1 -- 172.21.15.72:0/3416587569 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f003010b640 msgr2=0x7f003019bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:13.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:13.880+0000 7f0034a00700 1 --2- 172.21.15.72:0/3416587569 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f003010b640 0x7f003019bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:13.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:13.880+0000 7f0034a00700 1 -- 172.21.15.72:0/3416587569 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0030105640 msgr2=0x7f00301a1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:13.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:13.880+0000 7f0034a00700 1 --2- 172.21.15.72:0/3416587569 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0030105640 0x7f00301a1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:13.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:13.880+0000 7f0034a00700 1 -- 172.21.15.72:0/3416587569 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f003019c440 con 0x7f0030106000 2024-04-26T23:43:13.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:13.881+0000 7f0034a00700 1 --2- 172.21.15.72:0/3416587569 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0030106000 0x7f00301a1f80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f001c000c00 tx=0x7f001c009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:13.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:13.881+0000 7f002e000700 1 -- 172.21.15.72:0/3416587569 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f001c027070 con 0x7f0030106000 2024-04-26T23:43:13.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:13.881+0000 7f002e000700 1 -- 172.21.15.72:0/3416587569 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f001c022470 con 0x7f0030106000 2024-04-26T23:43:13.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:13.881+0000 7f0037e00700 1 -- 172.21.15.72:0/3416587569 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f003019c6d0 con 0x7f0030106000 2024-04-26T23:43:13.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:13.881+0000 7f002e000700 1 -- 172.21.15.72:0/3416587569 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f001c017650 con 0x7f0030106000 2024-04-26T23:43:13.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:13.881+0000 7f0037e00700 1 -- 172.21.15.72:0/3416587569 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00301a87f0 con 0x7f0030106000 2024-04-26T23:43:13.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:13.883+0000 7f0037e00700 1 -- 172.21.15.72:0/3416587569 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0030106d80 con 0x7f0030106000 2024-04-26T23:43:13.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:13.884+0000 7f002e000700 1 -- 172.21.15.72:0/3416587569 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f001c02a070 con 0x7f0030106000 2024-04-26T23:43:13.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:13.886+0000 7f002e000700 1 --2- 172.21.15.72:0/3416587569 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f00180a3c40 0x7f00180a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:13.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:13.886+0000 7f002e000700 1 -- 172.21.15.72:0/3416587569 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f001c0d13b0 con 0x7f0030106000 2024-04-26T23:43:13.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:13.886+0000 7f0035400700 1 --2- 172.21.15.72:0/3416587569 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f00180a3c40 0x7f00180a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:13.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:13.887+0000 7f0035400700 1 --2- 172.21.15.72:0/3416587569 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f00180a3c40 0x7f00180a6100 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0024009be0 tx=0x7f0024009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:13.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:13.890+0000 7f002e000700 1 -- 172.21.15.72:0/3416587569 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f001c013070 con 0x7f0030106000 2024-04-26T23:43:13.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:13 smithi076 ceph-mon[32774]: pgmap v326: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:43:14.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:14.166+0000 7f0037e00700 1 -- 172.21.15.72:0/3416587569 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f003002e490 con 0x7f0030106000 2024-04-26T23:43:14.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:14.167+0000 7f002e000700 1 -- 172.21.15.72:0/3416587569 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7f001c09db80 con 0x7f0030106000 2024-04-26T23:43:14.171 INFO:teuthology.orchestra.run.smithi072.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-26T23:43:14.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:14.170+0000 7f0037e00700 1 -- 172.21.15.72:0/3416587569 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f00180a3c40 msgr2=0x7f00180a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:14.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:14.170+0000 7f0037e00700 1 --2- 172.21.15.72:0/3416587569 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f00180a3c40 0x7f00180a6100 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0024009be0 tx=0x7f0024009260 comp rx=0 tx=0).stop 2024-04-26T23:43:14.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:14.170+0000 7f0037e00700 1 -- 172.21.15.72:0/3416587569 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0030106000 msgr2=0x7f00301a1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:14.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:14.170+0000 7f0037e00700 1 --2- 172.21.15.72:0/3416587569 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0030106000 0x7f00301a1f80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f001c000c00 tx=0x7f001c009580 comp rx=0 tx=0).stop 2024-04-26T23:43:14.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:14.170+0000 7f0037e00700 1 -- 172.21.15.72:0/3416587569 shutdown_connections 2024-04-26T23:43:14.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:14.170+0000 7f0037e00700 1 --2- 172.21.15.72:0/3416587569 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f00180a3c40 0x7f00180a6100 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:14.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:14.170+0000 7f0037e00700 1 --2- 172.21.15.72:0/3416587569 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f003010b640 0x7f003019bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:14.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:14.170+0000 7f0037e00700 1 --2- 172.21.15.72:0/3416587569 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0030105640 0x7f00301a1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:14.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:14.170+0000 7f0037e00700 1 --2- 172.21.15.72:0/3416587569 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0030106000 0x7f00301a1f80 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:14.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:14.170+0000 7f0037e00700 1 -- 172.21.15.72:0/3416587569 >> 172.21.15.72:0/3416587569 conn(0x7f0030101130 msgr2=0x7f0030101750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:14.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:14.171+0000 7f0037e00700 1 -- 172.21.15.72:0/3416587569 shutdown_connections 2024-04-26T23:43:14.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:14.171+0000 7f0037e00700 1 -- 172.21.15.72:0/3416587569 wait complete. 2024-04-26T23:43:14.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:14 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/3416587569' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-26T23:43:14.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:14 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/3416587569' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-26T23:43:15.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:14 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/3416587569' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-26T23:43:15.114 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-26T23:43:15.115 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-26T23:43:15.115 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi072.front.sepia.ceph.com 2024-04-26T23:43:15.115 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-04-26T23:43:15.115 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-26T23:43:15.115 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-26T23:43:15.115 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-26T23:43:15.115 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-26T23:43:15.115 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-04-26T23:43:15.115 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:43:15.116 DEBUG:teuthology.orchestra.run.smithi072:> ip addr 2024-04-26T23:43:15.173 INFO:teuthology.orchestra.run.smithi072.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-26T23:43:15.174 INFO:teuthology.orchestra.run.smithi072.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-26T23:43:15.174 INFO:teuthology.orchestra.run.smithi072.stdout: inet 127.0.0.1/8 scope host lo 2024-04-26T23:43:15.174 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:43:15.174 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 ::1/128 scope host 2024-04-26T23:43:15.174 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:43:15.174 INFO:teuthology.orchestra.run.smithi072.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T23:43:15.174 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:d8:da:f4 brd ff:ff:ff:ff:ff:ff 2024-04-26T23:43:15.174 INFO:teuthology.orchestra.run.smithi072.stdout: altname enp7s0f0 2024-04-26T23:43:15.174 INFO:teuthology.orchestra.run.smithi072.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T23:43:15.174 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:d8:da:f5 brd ff:ff:ff:ff:ff:ff 2024-04-26T23:43:15.174 INFO:teuthology.orchestra.run.smithi072.stdout: altname enp7s0f1 2024-04-26T23:43:15.174 INFO:teuthology.orchestra.run.smithi072.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T23:43:15.175 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:8f:b7:64 brd ff:ff:ff:ff:ff:ff 2024-04-26T23:43:15.175 INFO:teuthology.orchestra.run.smithi072.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-26T23:43:15.175 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:8f:b7:65 brd ff:ff:ff:ff:ff:ff 2024-04-26T23:43:15.175 INFO:teuthology.orchestra.run.smithi072.stdout: inet 172.21.15.72/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-26T23:43:15.175 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft 41997sec preferred_lft 41997sec 2024-04-26T23:43:15.175 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::ec4:7aff:fe8f:b765/64 scope link 2024-04-26T23:43:15.175 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:43:15.176 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-26T23:43:15.176 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T23:43:15.176 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-04-26T23:43:15.176 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link add name ceph-brx type bridge 2024-04-26T23:43:15.177 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip addr flush dev ceph-brx 2024-04-26T23:43:15.177 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link set ceph-brx up 2024-04-26T23:43:15.177 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-26T23:43:15.177 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-04-26T23:43:15.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:15 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T23:43:15.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:43:15.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:43:15.777 DEBUG:teuthology.orchestra.run.smithi072:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-26T23:43:15.841 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:15 smithi072 ceph-mon[28184]: pgmap v327: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:43:15.847 INFO:teuthology.orchestra.run.smithi072.stdout:1 2024-04-26T23:43:15.851 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:43:15.851 DEBUG:teuthology.orchestra.run.smithi072:> route 2024-04-26T23:43:15.918 INFO:teuthology.orchestra.run.smithi072.stdout:Kernel IP routing table 2024-04-26T23:43:15.918 INFO:teuthology.orchestra.run.smithi072.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-26T23:43:15.918 INFO:teuthology.orchestra.run.smithi072.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-26T23:43:15.918 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-26T23:43:15.918 INFO:teuthology.orchestra.run.smithi072.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-26T23:43:15.919 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T23:43:15.919 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-04-26T23:43:15.919 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-26T23:43:15.919 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-26T23:43:15.919 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-26T23:43:15.919 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-04-26T23:43:15.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:15 smithi076 ceph-mon[32774]: pgmap v327: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:43:16.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:16 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T23:43:16.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:15 smithi136 ceph-mon[33050]: pgmap v327: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:43:16.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:43:16.938 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:43:16.938 DEBUG:teuthology.orchestra.run.smithi072:> ip netns list 2024-04-26T23:43:16.991 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:43:16.992 DEBUG:teuthology.orchestra.run.smithi072:> ip netns list-id 2024-04-26T23:43:17.043 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T23:43:17.043 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-04-26T23:43:17.043 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-26T23:43:17.043 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-04-26T23:43:17.043 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-04-26T23:43:17.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T23:43:17.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:43:17.227 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-04-26T23:43:17.228 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T23:43:17.228 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-04-26T23:43:17.228 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-04-26T23:43:17.228 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-26T23:43:17.228 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-04-26T23:43:17.228 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-04-26T23:43:17.228 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-04-26T23:43:17.228 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-04-26T23:43:17.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T23:43:17.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:43:17.531 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T23:43:17.531 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-04-26T23:43:17.531 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link set brx.0 up 2024-04-26T23:43:17.531 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link set dev brx.0 master ceph-brx 2024-04-26T23:43:17.531 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-04-26T23:43:17.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T23:43:17.696 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:17 smithi072 ceph-mon[28184]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:43:17.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:43:17.730 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-04-26T23:43:17.730 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T23:43:17.730 DEBUG:teuthology.orchestra.run.smithi072:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-04-26T23:43:17.780 INFO:teuthology.orchestra.run.smithi072.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-04-26T23:43:17.780 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T23:43:17.780 DEBUG:teuthology.orchestra.run.smithi072:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-04-26T23:43:17.827 DEBUG:teuthology.orchestra.run.smithi072:> sudo modprobe fuse 2024-04-26T23:43:17.899 DEBUG:teuthology.orchestra.run.smithi072:> cat /proc/self/mounts | awk '{print $2}' 2024-04-26T23:43:17.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:17 smithi076 ceph-mon[32774]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:43:17.950 INFO:teuthology.orchestra.run.smithi072.stdout:/proc 2024-04-26T23:43:17.950 INFO:teuthology.orchestra.run.smithi072.stdout:/sys 2024-04-26T23:43:17.950 INFO:teuthology.orchestra.run.smithi072.stdout:/dev 2024-04-26T23:43:17.950 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/security 2024-04-26T23:43:17.951 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/shm 2024-04-26T23:43:17.951 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/pts 2024-04-26T23:43:17.951 INFO:teuthology.orchestra.run.smithi072.stdout:/run 2024-04-26T23:43:17.951 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/cgroup 2024-04-26T23:43:17.951 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/pstore 2024-04-26T23:43:17.951 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/bpf 2024-04-26T23:43:17.951 INFO:teuthology.orchestra.run.smithi072.stdout:/ 2024-04-26T23:43:17.951 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/selinux 2024-04-26T23:43:17.951 INFO:teuthology.orchestra.run.smithi072.stdout:/proc/sys/fs/binfmt_misc 2024-04-26T23:43:17.951 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/mqueue 2024-04-26T23:43:17.951 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/hugepages 2024-04-26T23:43:17.951 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/debug 2024-04-26T23:43:17.951 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/tracing 2024-04-26T23:43:17.951 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/fuse/connections 2024-04-26T23:43:17.951 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-sysctl.service 2024-04-26T23:43:17.952 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-26T23:43:17.952 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/config 2024-04-26T23:43:17.952 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-26T23:43:17.952 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/nfs/rpc_pipefs 2024-04-26T23:43:17.952 INFO:teuthology.orchestra.run.smithi072.stdout:/run/user/1000 2024-04-26T23:43:17.952 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph 2024-04-26T23:43:17.952 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay 2024-04-26T23:43:17.952 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay/fe41a7d15ff9cc46d141715c65b446ee16f2fe1f429f38ea6a79092d5e212d70/merged 2024-04-26T23:43:17.952 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay/efc450ab0a9e4c2a71fd374dcdffbebcb363ceec31c2f1368ada4425f1bdba74/merged 2024-04-26T23:43:17.952 INFO:teuthology.orchestra.run.smithi072.stdout:/run/user/0 2024-04-26T23:43:17.952 INFO:teuthology.orchestra.run.smithi072.stdout:/proc/sys/fs/binfmt_misc 2024-04-26T23:43:17.952 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay/e204e52b7aed3fff44bf4ba3e1cacd102be6b6a86f8ab8b0f0d650351b64b667/merged 2024-04-26T23:43:17.952 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay/51017b70270964c447612f00abffb0a0ff4d0d566e51fa34e15c26a37c5fa0f9/merged 2024-04-26T23:43:17.952 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay/43f702904ac6c2da22a054268129f7cad2a21bfdb15d00395207b5545e0d2040/merged 2024-04-26T23:43:17.952 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay/7f76d3e36baced43076af2707dfe4f296da9663d64104e3052531abc377e7a52/merged 2024-04-26T23:43:17.953 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay/c6366f41dc7707be08b66ada12ac0f32aa91d67a15905f023760286155799f6e/merged 2024-04-26T23:43:17.953 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay/b8d7975cef5e51107b034630676846a0fa5ea3e2941fa2be3bcfbadba480adf6/merged 2024-04-26T23:43:17.953 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay/1c4a01fbb903e1853cb7871fe399d1256dba17749672dabe03d3ffb1009994ef/merged 2024-04-26T23:43:17.953 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay/444ab55e6fb08ec5896021f7b9b8b9bfe182a7ce20d5124afdeae8fc1e3781b3/merged 2024-04-26T23:43:17.953 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns 2024-04-26T23:43:17.953 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-26T23:43:17.953 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-26T23:43:17.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:43:17.954 DEBUG:teuthology.orchestra.run.smithi072:> ls /sys/fs/fuse/connections 2024-04-26T23:43:18.002 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-26T23:43:18.002 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-04-26T23:43:18.006 DEBUG:teuthology.orchestra.run.smithi072:> sudo modprobe fuse 2024-04-26T23:43:18.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:17 smithi136 ceph-mon[33050]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:43:18.078 DEBUG:teuthology.orchestra.run.smithi072:> cat /proc/self/mounts | awk '{print $2}' 2024-04-26T23:43:18.121 INFO:teuthology.orchestra.run.smithi072.stdout:/proc 2024-04-26T23:43:18.121 INFO:teuthology.orchestra.run.smithi072.stdout:/sys 2024-04-26T23:43:18.121 INFO:teuthology.orchestra.run.smithi072.stdout:/dev 2024-04-26T23:43:18.121 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/security 2024-04-26T23:43:18.121 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/shm 2024-04-26T23:43:18.121 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/pts 2024-04-26T23:43:18.121 INFO:teuthology.orchestra.run.smithi072.stdout:/run 2024-04-26T23:43:18.121 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/cgroup 2024-04-26T23:43:18.121 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/pstore 2024-04-26T23:43:18.121 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/bpf 2024-04-26T23:43:18.122 INFO:teuthology.orchestra.run.smithi072.stdout:/ 2024-04-26T23:43:18.122 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/selinux 2024-04-26T23:43:18.122 INFO:teuthology.orchestra.run.smithi072.stdout:/proc/sys/fs/binfmt_misc 2024-04-26T23:43:18.122 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/mqueue 2024-04-26T23:43:18.122 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/hugepages 2024-04-26T23:43:18.122 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/debug 2024-04-26T23:43:18.122 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/tracing 2024-04-26T23:43:18.122 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/fuse/connections 2024-04-26T23:43:18.122 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-sysctl.service 2024-04-26T23:43:18.122 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-26T23:43:18.122 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/config 2024-04-26T23:43:18.123 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-26T23:43:18.123 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/nfs/rpc_pipefs 2024-04-26T23:43:18.123 INFO:teuthology.orchestra.run.smithi072.stdout:/run/user/1000 2024-04-26T23:43:18.123 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph 2024-04-26T23:43:18.123 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay 2024-04-26T23:43:18.123 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay/fe41a7d15ff9cc46d141715c65b446ee16f2fe1f429f38ea6a79092d5e212d70/merged 2024-04-26T23:43:18.123 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay/efc450ab0a9e4c2a71fd374dcdffbebcb363ceec31c2f1368ada4425f1bdba74/merged 2024-04-26T23:43:18.123 INFO:teuthology.orchestra.run.smithi072.stdout:/run/user/0 2024-04-26T23:43:18.123 INFO:teuthology.orchestra.run.smithi072.stdout:/proc/sys/fs/binfmt_misc 2024-04-26T23:43:18.123 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay/e204e52b7aed3fff44bf4ba3e1cacd102be6b6a86f8ab8b0f0d650351b64b667/merged 2024-04-26T23:43:18.123 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay/51017b70270964c447612f00abffb0a0ff4d0d566e51fa34e15c26a37c5fa0f9/merged 2024-04-26T23:43:18.123 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay/43f702904ac6c2da22a054268129f7cad2a21bfdb15d00395207b5545e0d2040/merged 2024-04-26T23:43:18.124 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay/7f76d3e36baced43076af2707dfe4f296da9663d64104e3052531abc377e7a52/merged 2024-04-26T23:43:18.124 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay/c6366f41dc7707be08b66ada12ac0f32aa91d67a15905f023760286155799f6e/merged 2024-04-26T23:43:18.124 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay/b8d7975cef5e51107b034630676846a0fa5ea3e2941fa2be3bcfbadba480adf6/merged 2024-04-26T23:43:18.124 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay/1c4a01fbb903e1853cb7871fe399d1256dba17749672dabe03d3ffb1009994ef/merged 2024-04-26T23:43:18.124 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay/444ab55e6fb08ec5896021f7b9b8b9bfe182a7ce20d5124afdeae8fc1e3781b3/merged 2024-04-26T23:43:18.124 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns 2024-04-26T23:43:18.124 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-26T23:43:18.124 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-26T23:43:18.125 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:43:18.125 DEBUG:teuthology.orchestra.run.smithi072:> ls /sys/fs/fuse/connections 2024-04-26T23:43:18.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi072.stderr:2024-04-26T23:43:18.207+0000 7fdef7e11480 -1 init, newargv = 0x556e1ab914f0 newargc=15 2024-04-26T23:43:18.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi072.stderr:ceph-fuse[70221]: starting ceph client 2024-04-26T23:43:18.222 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi072.stderr:ceph-fuse[70221]: starting fuse 2024-04-26T23:43:19.313 DEBUG:teuthology.orchestra.run.smithi072:> sudo modprobe fuse 2024-04-26T23:43:19.384 DEBUG:teuthology.orchestra.run.smithi072:> cat /proc/self/mounts | awk '{print $2}' 2024-04-26T23:43:19.428 INFO:teuthology.orchestra.run.smithi072.stdout:/proc 2024-04-26T23:43:19.428 INFO:teuthology.orchestra.run.smithi072.stdout:/sys 2024-04-26T23:43:19.428 INFO:teuthology.orchestra.run.smithi072.stdout:/dev 2024-04-26T23:43:19.428 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/security 2024-04-26T23:43:19.429 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/shm 2024-04-26T23:43:19.429 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/pts 2024-04-26T23:43:19.429 INFO:teuthology.orchestra.run.smithi072.stdout:/run 2024-04-26T23:43:19.429 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/cgroup 2024-04-26T23:43:19.429 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/pstore 2024-04-26T23:43:19.429 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/bpf 2024-04-26T23:43:19.429 INFO:teuthology.orchestra.run.smithi072.stdout:/ 2024-04-26T23:43:19.429 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/selinux 2024-04-26T23:43:19.429 INFO:teuthology.orchestra.run.smithi072.stdout:/proc/sys/fs/binfmt_misc 2024-04-26T23:43:19.429 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/mqueue 2024-04-26T23:43:19.429 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/hugepages 2024-04-26T23:43:19.430 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/debug 2024-04-26T23:43:19.430 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/tracing 2024-04-26T23:43:19.430 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/fuse/connections 2024-04-26T23:43:19.430 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-sysctl.service 2024-04-26T23:43:19.430 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-26T23:43:19.430 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/config 2024-04-26T23:43:19.430 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-26T23:43:19.430 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/nfs/rpc_pipefs 2024-04-26T23:43:19.430 INFO:teuthology.orchestra.run.smithi072.stdout:/run/user/1000 2024-04-26T23:43:19.430 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph 2024-04-26T23:43:19.430 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay 2024-04-26T23:43:19.430 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay/fe41a7d15ff9cc46d141715c65b446ee16f2fe1f429f38ea6a79092d5e212d70/merged 2024-04-26T23:43:19.431 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay/efc450ab0a9e4c2a71fd374dcdffbebcb363ceec31c2f1368ada4425f1bdba74/merged 2024-04-26T23:43:19.431 INFO:teuthology.orchestra.run.smithi072.stdout:/run/user/0 2024-04-26T23:43:19.431 INFO:teuthology.orchestra.run.smithi072.stdout:/proc/sys/fs/binfmt_misc 2024-04-26T23:43:19.431 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay/e204e52b7aed3fff44bf4ba3e1cacd102be6b6a86f8ab8b0f0d650351b64b667/merged 2024-04-26T23:43:19.431 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay/51017b70270964c447612f00abffb0a0ff4d0d566e51fa34e15c26a37c5fa0f9/merged 2024-04-26T23:43:19.431 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay/43f702904ac6c2da22a054268129f7cad2a21bfdb15d00395207b5545e0d2040/merged 2024-04-26T23:43:19.431 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay/7f76d3e36baced43076af2707dfe4f296da9663d64104e3052531abc377e7a52/merged 2024-04-26T23:43:19.431 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay/c6366f41dc7707be08b66ada12ac0f32aa91d67a15905f023760286155799f6e/merged 2024-04-26T23:43:19.431 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay/b8d7975cef5e51107b034630676846a0fa5ea3e2941fa2be3bcfbadba480adf6/merged 2024-04-26T23:43:19.431 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay/1c4a01fbb903e1853cb7871fe399d1256dba17749672dabe03d3ffb1009994ef/merged 2024-04-26T23:43:19.431 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/containers/storage/overlay/444ab55e6fb08ec5896021f7b9b8b9bfe182a7ce20d5124afdeae8fc1e3781b3/merged 2024-04-26T23:43:19.431 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns 2024-04-26T23:43:19.431 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-26T23:43:19.432 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-26T23:43:19.432 INFO:teuthology.orchestra.run.smithi072.stdout:/home/ubuntu/cephtest/mnt.admin 2024-04-26T23:43:19.432 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:43:19.433 DEBUG:teuthology.orchestra.run.smithi072:> ls /sys/fs/fuse/connections 2024-04-26T23:43:19.522 INFO:teuthology.orchestra.run.smithi072.stdout:114 2024-04-26T23:43:19.522 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [114] 2024-04-26T23:43:19.522 DEBUG:teuthology.orchestra.run.smithi072:> sudo stdin-killer -- python3 -c ' 2024-04-26T23:43:19.523 DEBUG:teuthology.orchestra.run.smithi072:> import glob 2024-04-26T23:43:19.523 DEBUG:teuthology.orchestra.run.smithi072:> import re 2024-04-26T23:43:19.523 DEBUG:teuthology.orchestra.run.smithi072:> import os 2024-04-26T23:43:19.523 DEBUG:teuthology.orchestra.run.smithi072:> import subprocess 2024-04-26T23:43:19.523 DEBUG:teuthology.orchestra.run.smithi072:> 2024-04-26T23:43:19.523 DEBUG:teuthology.orchestra.run.smithi072:> def _find_admin_socket(client_name): 2024-04-26T23:43:19.523 DEBUG:teuthology.orchestra.run.smithi072:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-04-26T23:43:19.523 DEBUG:teuthology.orchestra.run.smithi072:> files = glob.glob(asok_path) 2024-04-26T23:43:19.523 DEBUG:teuthology.orchestra.run.smithi072:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-04-26T23:43:19.523 DEBUG:teuthology.orchestra.run.smithi072:> 2024-04-26T23:43:19.523 DEBUG:teuthology.orchestra.run.smithi072:> # Given a non-glob path, it better be there 2024-04-26T23:43:19.523 DEBUG:teuthology.orchestra.run.smithi072:> if "*" not in asok_path: 2024-04-26T23:43:19.523 DEBUG:teuthology.orchestra.run.smithi072:> assert(len(files) == 1) 2024-04-26T23:43:19.523 DEBUG:teuthology.orchestra.run.smithi072:> return files[0] 2024-04-26T23:43:19.523 DEBUG:teuthology.orchestra.run.smithi072:> 2024-04-26T23:43:19.523 DEBUG:teuthology.orchestra.run.smithi072:> for f in files: 2024-04-26T23:43:19.523 DEBUG:teuthology.orchestra.run.smithi072:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-26T23:43:19.524 DEBUG:teuthology.orchestra.run.smithi072:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-26T23:43:19.524 DEBUG:teuthology.orchestra.run.smithi072:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-26T23:43:19.524 DEBUG:teuthology.orchestra.run.smithi072:> contents = proc_f.read() 2024-04-26T23:43:19.524 DEBUG:teuthology.orchestra.run.smithi072:> if mountpoint in contents: 2024-04-26T23:43:19.524 DEBUG:teuthology.orchestra.run.smithi072:> return f 2024-04-26T23:43:19.524 DEBUG:teuthology.orchestra.run.smithi072:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-26T23:43:19.524 DEBUG:teuthology.orchestra.run.smithi072:> 2024-04-26T23:43:19.524 DEBUG:teuthology.orchestra.run.smithi072:> print(_find_admin_socket("client.admin")) 2024-04-26T23:43:19.524 DEBUG:teuthology.orchestra.run.smithi072:> ' 2024-04-26T23:43:19.670 INFO:teuthology.orchestra.run.smithi072.stdout:/var/run/ceph/ceph-client.admin.70221.asok 2024-04-26T23:43:19.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:43:19.683 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:19 smithi072 ceph-mon[28184]: pgmap v329: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:43:19.685 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.70221.asok 2024-04-26T23:43:19.686 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:43:19.686 DEBUG:teuthology.orchestra.run.smithi072:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.70221.asok status 2024-04-26T23:43:19.906 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-04-26T23:43:19.907 INFO:teuthology.orchestra.run.smithi072.stdout: "metadata": { 2024-04-26T23:43:19.907 INFO:teuthology.orchestra.run.smithi072.stdout: "ceph_sha1": "bc612ffb9c1d5bc27536bc917a705262a36d4387", 2024-04-26T23:43:19.907 INFO:teuthology.orchestra.run.smithi072.stdout: "ceph_version": "ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev)", 2024-04-26T23:43:19.907 INFO:teuthology.orchestra.run.smithi072.stdout: "entity_id": "admin", 2024-04-26T23:43:19.907 INFO:teuthology.orchestra.run.smithi072.stdout: "hostname": "smithi072", 2024-04-26T23:43:19.907 INFO:teuthology.orchestra.run.smithi072.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-04-26T23:43:19.907 INFO:teuthology.orchestra.run.smithi072.stdout: "pid": "70221", 2024-04-26T23:43:19.907 INFO:teuthology.orchestra.run.smithi072.stdout: "root": "/" 2024-04-26T23:43:19.907 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-26T23:43:19.907 INFO:teuthology.orchestra.run.smithi072.stdout: "dentry_count": 0, 2024-04-26T23:43:19.908 INFO:teuthology.orchestra.run.smithi072.stdout: "dentry_pinned_count": 0, 2024-04-26T23:43:19.908 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 14610, 2024-04-26T23:43:19.908 INFO:teuthology.orchestra.run.smithi072.stdout: "inst": { 2024-04-26T23:43:19.908 INFO:teuthology.orchestra.run.smithi072.stdout: "name": { 2024-04-26T23:43:19.908 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "client", 2024-04-26T23:43:19.908 INFO:teuthology.orchestra.run.smithi072.stdout: "num": 14610 2024-04-26T23:43:19.908 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-26T23:43:19.908 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": { 2024-04-26T23:43:19.908 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "v1", 2024-04-26T23:43:19.908 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": "172.21.15.72:0", 2024-04-26T23:43:19.908 INFO:teuthology.orchestra.run.smithi072.stdout: "nonce": 1625470800 2024-04-26T23:43:19.908 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-04-26T23:43:19.908 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-26T23:43:19.908 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": { 2024-04-26T23:43:19.909 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "v1", 2024-04-26T23:43:19.909 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": "172.21.15.72:0", 2024-04-26T23:43:19.909 INFO:teuthology.orchestra.run.smithi072.stdout: "nonce": 1625470800 2024-04-26T23:43:19.909 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-26T23:43:19.909 INFO:teuthology.orchestra.run.smithi072.stdout: "inst_str": "client.14610 172.21.15.72:0/1625470800", 2024-04-26T23:43:19.909 INFO:teuthology.orchestra.run.smithi072.stdout: "addr_str": "172.21.15.72:0/1625470800", 2024-04-26T23:43:19.909 INFO:teuthology.orchestra.run.smithi072.stdout: "inode_count": 1, 2024-04-26T23:43:19.909 INFO:teuthology.orchestra.run.smithi072.stdout: "mds_epoch": 17, 2024-04-26T23:43:19.909 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_epoch": 89, 2024-04-26T23:43:19.910 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_epoch_barrier": 0, 2024-04-26T23:43:19.910 INFO:teuthology.orchestra.run.smithi072.stdout: "blocklisted": false, 2024-04-26T23:43:19.910 INFO:teuthology.orchestra.run.smithi072.stdout: "fs_name": "cephfs" 2024-04-26T23:43:19.910 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-04-26T23:43:19.920 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:43:19.920 DEBUG:teuthology.orchestra.run.smithi072:> stat --file-system '--printf=%T 2024-04-26T23:43:19.920 DEBUG:teuthology.orchestra.run.smithi072:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-04-26T23:43:19.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:19 smithi076 ceph-mon[32774]: pgmap v329: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:43:19.968 INFO:teuthology.orchestra.run.smithi072.stdout:fuseblk 2024-04-26T23:43:19.968 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-04-26T23:43:19.968 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:43:19.968 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-04-26T23:43:20.040 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-04-26T23:43:20.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:19 smithi136 ceph-mon[33050]: pgmap v329: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T23:43:20.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T23:43:20.170 INFO:teuthology.orchestra.run.smithi072.stdout:# file: . 2024-04-26T23:43:20.170 INFO:teuthology.orchestra.run.smithi072.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-04-26T23:43:20.170 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:43:20.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:43:20.180 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-26T23:43:20.180 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:43:20.229 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:43:20.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.467+0000 7fbe92200640 1 -- 172.21.15.72:0/1883931760 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbe8c10a5e0 msgr2=0x7fbe8c10ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:20.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.467+0000 7fbe92200640 1 --2- 172.21.15.72:0/1883931760 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbe8c10a5e0 0x7fbe8c10ca80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fbe84009ca0 tx=0x7fbe8402a950 comp rx=0 tx=0).stop 2024-04-26T23:43:20.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.468+0000 7fbe92200640 1 -- 172.21.15.72:0/1883931760 shutdown_connections 2024-04-26T23:43:20.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.468+0000 7fbe92200640 1 --2- 172.21.15.72:0/1883931760 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbe8c10a5e0 0x7fbe8c10ca80 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:20.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.468+0000 7fbe92200640 1 --2- 172.21.15.72:0/1883931760 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbe8c075330 0x7fbe8c075790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:20.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.468+0000 7fbe92200640 1 --2- 172.21.15.72:0/1883931760 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe8c076910 0x7fbe8c074d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:20.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.468+0000 7fbe92200640 1 -- 172.21.15.72:0/1883931760 >> 172.21.15.72:0/1883931760 conn(0x7fbe8c0fdcc0 msgr2=0x7fbe8c100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:20.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.468+0000 7fbe92200640 1 -- 172.21.15.72:0/1883931760 shutdown_connections 2024-04-26T23:43:20.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.468+0000 7fbe92200640 1 -- 172.21.15.72:0/1883931760 wait complete. 2024-04-26T23:43:20.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.469+0000 7fbe92200640 1 Processor -- start 2024-04-26T23:43:20.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.469+0000 7fbe92200640 1 -- start start 2024-04-26T23:43:20.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.469+0000 7fbe92200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe8c075330 0x7fbe8c19ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:20.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.469+0000 7fbe92200640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbe8c076910 0x7fbe8c19b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:20.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.469+0000 7fbe92200640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbe8c10a5e0 0x7fbe8c1a20d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:20.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.469+0000 7fbe92200640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fbe8c110460 con 0x7fbe8c076910 2024-04-26T23:43:20.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.469+0000 7fbe92200640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fbe8c1102e0 con 0x7fbe8c075330 2024-04-26T23:43:20.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.469+0000 7fbe92200640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fbe8c1105e0 con 0x7fbe8c10a5e0 2024-04-26T23:43:20.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.470+0000 7fbe91800640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbe8c10a5e0 0x7fbe8c1a20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:20.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.470+0000 7fbe91800640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbe8c10a5e0 0x7fbe8c1a20d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.72:57526/0 (socket says 172.21.15.72:57526) 2024-04-26T23:43:20.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.470+0000 7fbe91800640 1 -- 172.21.15.72:0/110901178 learned_addr learned my addr 172.21.15.72:0/110901178 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:43:20.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.470+0000 7fbe90e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe8c075330 0x7fbe8c19ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:20.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.470+0000 7fbe83e00640 1 --2- 172.21.15.72:0/110901178 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbe8c076910 0x7fbe8c19b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:20.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.470+0000 7fbe91800640 1 -- 172.21.15.72:0/110901178 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe8c075330 msgr2=0x7fbe8c19ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:20.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.470+0000 7fbe91800640 1 --2- 172.21.15.72:0/110901178 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe8c075330 0x7fbe8c19ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:20.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.470+0000 7fbe91800640 1 -- 172.21.15.72:0/110901178 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbe8c076910 msgr2=0x7fbe8c19b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:20.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.470+0000 7fbe91800640 1 --2- 172.21.15.72:0/110901178 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbe8c076910 0x7fbe8c19b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:20.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.470+0000 7fbe91800640 1 -- 172.21.15.72:0/110901178 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe8c1a2740 con 0x7fbe8c10a5e0 2024-04-26T23:43:20.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.470+0000 7fbe83e00640 1 --2- 172.21.15.72:0/110901178 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbe8c076910 0x7fbe8c19b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:43:20.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.471+0000 7fbe91800640 1 --2- 172.21.15.72:0/110901178 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbe8c10a5e0 0x7fbe8c1a20d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fbe84006010 tx=0x7fbe8402fc20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:20.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.471+0000 7fbe8a000640 1 -- 172.21.15.72:0/110901178 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe84034040 con 0x7fbe8c10a5e0 2024-04-26T23:43:20.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.471+0000 7fbe92200640 1 -- 172.21.15.72:0/110901178 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe8c1a29d0 con 0x7fbe8c10a5e0 2024-04-26T23:43:20.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.471+0000 7fbe92200640 1 -- 172.21.15.72:0/110901178 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe8c1a2f10 con 0x7fbe8c10a5e0 2024-04-26T23:43:20.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.471+0000 7fbe8a000640 1 -- 172.21.15.72:0/110901178 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe840308f0 con 0x7fbe8c10a5e0 2024-04-26T23:43:20.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.471+0000 7fbe8a000640 1 -- 172.21.15.72:0/110901178 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe84039450 con 0x7fbe8c10a5e0 2024-04-26T23:43:20.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.472+0000 7fbe92200640 1 -- 172.21.15.72:0/110901178 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe8c1093a0 con 0x7fbe8c10a5e0 2024-04-26T23:43:20.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.474+0000 7fbe8a000640 1 -- 172.21.15.72:0/110901178 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fbe84037020 con 0x7fbe8c10a5e0 2024-04-26T23:43:20.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.476+0000 7fbe8a000640 1 --2- 172.21.15.72:0/110901178 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fbe6c0a3bb0 0x7fbe6c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:20.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.476+0000 7fbe8a000640 1 -- 172.21.15.72:0/110901178 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fbe840f1a60 con 0x7fbe8c10a5e0 2024-04-26T23:43:20.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.476+0000 7fbe90e00640 1 --2- 172.21.15.72:0/110901178 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fbe6c0a3bb0 0x7fbe6c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:20.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.477+0000 7fbe90e00640 1 --2- 172.21.15.72:0/110901178 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fbe6c0a3bb0 0x7fbe6c0a6070 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fbe74009bd0 tx=0x7fbe74005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:20.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.478+0000 7fbe8a000640 1 -- 172.21.15.72:0/110901178 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fbe840be350 con 0x7fbe8c10a5e0 2024-04-26T23:43:20.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.682+0000 7fbe92200640 1 -- 172.21.15.72:0/110901178 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbe8c063530 con 0x7fbe8c10a5e0 2024-04-26T23:43:20.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.685+0000 7fbe8a000640 1 -- 172.21.15.72:0/110901178 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24081 (secure 0 0 0) 0x7fbe84030ac0 con 0x7fbe8c10a5e0 2024-04-26T23:43:20.689 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:43:20.689 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"fsid":"3c8453ae-0425-11ef-bc93-c7b262605968","created":"2024-04-26T23:33:48.088300+0000","modified":"2024-04-26T23:43:07.990780+0000","last_up_change":"2024-04-26T23:42:04.482891+0000","last_in_change":"2024-04-26T23:41:44.784415+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:37:44.428727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:43:00.431829+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:43:01.872977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:43:05.512050+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e20c9ebf-56f6-47c1-a11b-e1e615af042b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6803","nonce":1639298875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6805","nonce":1639298875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6809","nonce":1639298875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6807","nonce":1639298875}]},"public_addr":"172.21.15.72:6803/1639298875","cluster_addr":"172.21.15.72:6805/1639298875","heartbeat_back_addr":"172.21.15.72:6809/1639298875","heartbeat_front_addr":"172.21.15.72:6807/1639298875","state":["exists","up"]},{"osd":1,"uuid":"e6cac469-780c-4d66-b560-b686072371fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6801","nonce":4185342193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6803","nonce":4185342193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6807","nonce":4185342193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6805","nonce":4185342193}]},"public_addr":"172.21.15.76:6801/4185342193","cluster_addr":"172.21.15.76:6803/4185342193","heartbeat_back_addr":"172.21.15.76:6807/4185342193","heartbeat_front_addr":"172.21.15.76:6805/4185342193","state":["exists","up"]},{"osd":2,"uuid":"838a49fc-18de-446c-8336-67b3e05da231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6801","nonce":3033860799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6803","nonce":3033860799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6807","nonce":3033860799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6805","nonce":3033860799}]},"public_addr":"172.21.15.136:6801/3033860799","cluster_addr":"172.21.15.136:6803/3033860799","heartbeat_back_addr":"172.21.15.136:6807/3033860799","heartbeat_front_addr":"172.21.15.136:6805/3033860799","state":["exists","up"]},{"osd":3,"uuid":"e6da89be-5ba7-48a4-9306-f75498660e0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6811","nonce":2071107815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6813","nonce":2071107815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6817","nonce":2071107815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6815","nonce":2071107815}]},"public_addr":"172.21.15.72:6811/2071107815","cluster_addr":"172.21.15.72:6813/2071107815","heartbeat_back_addr":"172.21.15.72:6817/2071107815","heartbeat_front_addr":"172.21.15.72:6815/2071107815","state":["exists","up"]},{"osd":4,"uuid":"0b17c491-d798-488f-892d-521ef2fbc51e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6809","nonce":147449238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6811","nonce":147449238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6815","nonce":147449238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6813","nonce":147449238}]},"public_addr":"172.21.15.76:6809/147449238","cluster_addr":"172.21.15.76:6811/147449238","heartbeat_back_addr":"172.21.15.76:6815/147449238","heartbeat_front_addr":"172.21.15.76:6813/147449238","state":["exists","up"]},{"osd":5,"uuid":"c7e887b3-ab38-4608-9bfd-1ed49956e8af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6809","nonce":2320149212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6811","nonce":2320149212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6815","nonce":2320149212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6813","nonce":2320149212}]},"public_addr":"172.21.15.136:6809/2320149212","cluster_addr":"172.21.15.136:6811/2320149212","heartbeat_back_addr":"172.21.15.136:6815/2320149212","heartbeat_front_addr":"172.21.15.136:6813/2320149212","state":["exists","up"]},{"osd":6,"uuid":"f0bd604b-0f57-4a38-9295-eb200264f1ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6819","nonce":389499207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6821","nonce":389499207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6825","nonce":389499207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6823","nonce":389499207}]},"public_addr":"172.21.15.72:6819/389499207","cluster_addr":"172.21.15.72:6821/389499207","heartbeat_back_addr":"172.21.15.72:6825/389499207","heartbeat_front_addr":"172.21.15.72:6823/389499207","state":["exists","up"]},{"osd":7,"uuid":"1879d4e4-038e-4cd6-a3ec-3a940bcc3797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6817","nonce":375888575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6819","nonce":375888575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6823","nonce":375888575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6821","nonce":375888575}]},"public_addr":"172.21.15.76:6817/375888575","cluster_addr":"172.21.15.76:6819/375888575","heartbeat_back_addr":"172.21.15.76:6823/375888575","heartbeat_front_addr":"172.21.15.76:6821/375888575","state":["exists","up"]},{"osd":8,"uuid":"6f404062-97e7-4ccb-a582-76dc1ffae548","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6817","nonce":3446774052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6819","nonce":3446774052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6823","nonce":3446774052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6821","nonce":3446774052}]},"public_addr":"172.21.15.136:6817/3446774052","cluster_addr":"172.21.15.136:6819/3446774052","heartbeat_back_addr":"172.21.15.136:6823/3446774052","heartbeat_front_addr":"172.21.15.136:6821/3446774052","state":["exists","up"]},{"osd":9,"uuid":"2989ca68-3164-4667-9bcf-230dcbb7b8f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6827","nonce":3423789152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6829","nonce":3423789152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6832","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6833","nonce":3423789152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6831","nonce":3423789152}]},"public_addr":"172.21.15.72:6827/3423789152","cluster_addr":"172.21.15.72:6829/3423789152","heartbeat_back_addr":"172.21.15.72:6833/3423789152","heartbeat_front_addr":"172.21.15.72:6831/3423789152","state":["exists","up"]},{"osd":10,"uuid":"9ec6d46a-0913-4208-97b5-167188ee8cb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6825","nonce":1120991574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6827","nonce":1120991574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6831","nonce":1120991574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6829","nonce":1120991574}]},"public_addr":"172.21.15.76:6825/1120991574","cluster_addr":"172.21.15.76:6827/1120991574","heartbeat_back_addr":"172.21.15.76:6831/1120991574","heartbeat_front_addr":"172.21.15.76:6829/1120991574","state":["exists","up"]},{"osd":11,"uuid":"bdd0774f-6e58-4918-b4cc-afd4f1118d3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6825","nonce":1135995193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6827","nonce":1135995193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6831","nonce":1135995193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6829","nonce":1135995193}]},"public_addr":"172.21.15.136:6825/1135995193","cluster_addr":"172.21.15.136:6827/1135995193","heartbeat_back_addr":"172.21.15.136:6831/1135995193","heartbeat_front_addr":"172.21.15.136:6829/1135995193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:36:52.128788+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:37:17.523654+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:37:41.634677+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:38:15.060797+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:38:41.234111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:39:06.783560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:39:40.884386+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:40:08.340465+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:40:34.344858+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:41:08.525309+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:41:36.782624+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:42:02.756888+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:0/3625071330":"2024-04-27T23:34:58.266218+0000","172.21.15.72:6801/2903979134":"2024-04-27T23:34:58.266218+0000","172.21.15.72:6800/2903979134":"2024-04-27T23:34:58.266218+0000","172.21.15.72:0/1519142576":"2024-04-27T23:34:58.266218+0000","172.21.15.72:0/3669626460":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/945623297":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/3479150073":"2024-04-27T23:34:24.233319+0000","172.21.15.72:6800/4175896290":"2024-04-27T23:34:24.233319+0000","172.21.15.72:6801/4175896290":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/3783175012":"2024-04-27T23:34:58.266218+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:43:20.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.687+0000 7fbe92200640 1 -- 172.21.15.72:0/110901178 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fbe6c0a3bb0 msgr2=0x7fbe6c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:20.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.687+0000 7fbe92200640 1 --2- 172.21.15.72:0/110901178 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fbe6c0a3bb0 0x7fbe6c0a6070 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fbe74009bd0 tx=0x7fbe74005f90 comp rx=0 tx=0).stop 2024-04-26T23:43:20.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.688+0000 7fbe92200640 1 -- 172.21.15.72:0/110901178 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbe8c10a5e0 msgr2=0x7fbe8c1a20d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:20.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.688+0000 7fbe92200640 1 --2- 172.21.15.72:0/110901178 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbe8c10a5e0 0x7fbe8c1a20d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fbe84006010 tx=0x7fbe8402fc20 comp rx=0 tx=0).stop 2024-04-26T23:43:20.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.688+0000 7fbe92200640 1 -- 172.21.15.72:0/110901178 shutdown_connections 2024-04-26T23:43:20.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.688+0000 7fbe92200640 1 --2- 172.21.15.72:0/110901178 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fbe6c0a3bb0 0x7fbe6c0a6070 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:20.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.688+0000 7fbe92200640 1 --2- 172.21.15.72:0/110901178 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbe8c10a5e0 0x7fbe8c1a20d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:20.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.688+0000 7fbe92200640 1 --2- 172.21.15.72:0/110901178 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbe8c076910 0x7fbe8c19b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:20.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.688+0000 7fbe92200640 1 --2- 172.21.15.72:0/110901178 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbe8c075330 0x7fbe8c19ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:20.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.688+0000 7fbe92200640 1 -- 172.21.15.72:0/110901178 >> 172.21.15.72:0/110901178 conn(0x7fbe8c0fdcc0 msgr2=0x7fbe8c106570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:20.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.688+0000 7fbe92200640 1 -- 172.21.15.72:0/110901178 shutdown_connections 2024-04-26T23:43:20.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.688+0000 7fbe92200640 1 -- 172.21.15.72:0/110901178 wait complete. 2024-04-26T23:43:20.711 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:37:44.428727+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T23:43:00.431829+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T23:43:01.872977+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T23:43:05.512050+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-26T23:43:20.712 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:43:20.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.904+0000 7f8dcbe00640 1 -- 172.21.15.72:0/2770411052 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8dcc078530 msgr2=0x7f8dcc078990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:20.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.904+0000 7f8dcbe00640 1 --2- 172.21.15.72:0/2770411052 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8dcc078530 0x7f8dcc078990 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8db8009900 tx=0x7f8db802a3f0 comp rx=0 tx=0).stop 2024-04-26T23:43:20.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.904+0000 7f8dcbe00640 1 -- 172.21.15.72:0/2770411052 shutdown_connections 2024-04-26T23:43:20.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.904+0000 7f8dcbe00640 1 --2- 172.21.15.72:0/2770411052 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8dcc078ed0 0x7f8dcc079770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:20.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.904+0000 7f8dcbe00640 1 --2- 172.21.15.72:0/2770411052 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8dcc078530 0x7f8dcc078990 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:20.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.904+0000 7f8dcbe00640 1 --2- 172.21.15.72:0/2770411052 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8dcc0772f0 0x7f8dcc077710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:20.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.904+0000 7f8dcbe00640 1 -- 172.21.15.72:0/2770411052 >> 172.21.15.72:0/2770411052 conn(0x7f8dcc0fffe0 msgr2=0x7f8dcc102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:20.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.905+0000 7f8dcbe00640 1 -- 172.21.15.72:0/2770411052 shutdown_connections 2024-04-26T23:43:20.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.905+0000 7f8dcbe00640 1 -- 172.21.15.72:0/2770411052 wait complete. 2024-04-26T23:43:20.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.905+0000 7f8dcbe00640 1 Processor -- start 2024-04-26T23:43:20.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.906+0000 7f8dcbe00640 1 -- start start 2024-04-26T23:43:20.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.906+0000 7f8dcbe00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8dcc0772f0 0x7f8dcc19f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:20.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.906+0000 7f8dcbe00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8dcc078530 0x7f8dcc19f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:20.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.906+0000 7f8dcbe00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8dcc078ed0 0x7f8dcc1a6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:20.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.906+0000 7f8dcbe00640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f8dcc1149b0 con 0x7f8dcc0772f0 2024-04-26T23:43:20.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.906+0000 7f8dcbe00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8dcc114830 con 0x7f8dcc078ed0 2024-04-26T23:43:20.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.906+0000 7f8dcbe00640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f8dcc114b30 con 0x7f8dcc078530 2024-04-26T23:43:20.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.906+0000 7f8dcaa00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8dcc0772f0 0x7f8dcc19f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:20.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.906+0000 7f8dcaa00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8dcc0772f0 0x7f8dcc19f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56676/0 (socket says 172.21.15.72:56676) 2024-04-26T23:43:20.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.906+0000 7f8dcaa00640 1 -- 172.21.15.72:0/2630798730 learned_addr learned my addr 172.21.15.72:0/2630798730 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:43:20.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.906+0000 7f8dca000640 1 --2- 172.21.15.72:0/2630798730 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8dcc078530 0x7f8dcc19f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:20.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.906+0000 7f8dcb400640 1 --2- 172.21.15.72:0/2630798730 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8dcc078ed0 0x7f8dcc1a6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:20.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.907+0000 7f8dcaa00640 1 -- 172.21.15.72:0/2630798730 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8dcc078530 msgr2=0x7f8dcc19f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:20.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.907+0000 7f8dcaa00640 1 --2- 172.21.15.72:0/2630798730 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8dcc078530 0x7f8dcc19f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:20.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.907+0000 7f8dcaa00640 1 -- 172.21.15.72:0/2630798730 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8dcc078ed0 msgr2=0x7f8dcc1a6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:20.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.907+0000 7f8dcaa00640 1 --2- 172.21.15.72:0/2630798730 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8dcc078ed0 0x7f8dcc1a6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:20.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.907+0000 7f8dcaa00640 1 -- 172.21.15.72:0/2630798730 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dcc1a6cb0 con 0x7f8dcc0772f0 2024-04-26T23:43:20.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.907+0000 7f8dcaa00640 1 --2- 172.21.15.72:0/2630798730 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8dcc0772f0 0x7f8dcc19f090 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc00d950 tx=0x7f8dbc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:20.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.907+0000 7f8dc3400640 1 -- 172.21.15.72:0/2630798730 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dbc017030 con 0x7f8dcc0772f0 2024-04-26T23:43:20.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.907+0000 7f8dc3400640 1 -- 172.21.15.72:0/2630798730 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8dbc004020 con 0x7f8dcc0772f0 2024-04-26T23:43:20.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.907+0000 7f8dcbe00640 1 -- 172.21.15.72:0/2630798730 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8dcc1a6fa0 con 0x7f8dcc0772f0 2024-04-26T23:43:20.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.907+0000 7f8dc3400640 1 -- 172.21.15.72:0/2630798730 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dbc0047a0 con 0x7f8dcc0772f0 2024-04-26T23:43:20.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.907+0000 7f8dcbe00640 1 -- 172.21.15.72:0/2630798730 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8dcc1a7410 con 0x7f8dcc0772f0 2024-04-26T23:43:20.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.909+0000 7f8dcbe00640 1 -- 172.21.15.72:0/2630798730 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8dcc1113e0 con 0x7f8dcc0772f0 2024-04-26T23:43:20.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.911+0000 7f8dc3400640 1 -- 172.21.15.72:0/2630798730 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8dbc009a30 con 0x7f8dcc0772f0 2024-04-26T23:43:20.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.912+0000 7f8dc3400640 1 --2- 172.21.15.72:0/2630798730 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f8db00a3c30 0x7f8db00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:20.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.912+0000 7f8dc3400640 1 -- 172.21.15.72:0/2630798730 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f8dbc0d0660 con 0x7f8dcc0772f0 2024-04-26T23:43:20.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.912+0000 7f8dca000640 1 --2- 172.21.15.72:0/2630798730 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f8db00a3c30 0x7f8db00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:20.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.913+0000 7f8dca000640 1 --2- 172.21.15.72:0/2630798730 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f8db00a3c30 0x7f8db00a60f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8db8000c00 tx=0x7f8db80314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:20.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:20.916+0000 7f8dc3400640 1 -- 172.21.15.72:0/2630798730 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8dbc09ce80 con 0x7f8dcc0772f0 2024-04-26T23:43:21.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.120+0000 7f8dcbe00640 1 -- 172.21.15.72:0/2630798730 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8dcc02dd60 con 0x7f8dcc0772f0 2024-04-26T23:43:21.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.121+0000 7f8dc3400640 1 -- 172.21.15.72:0/2630798730 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7f8dbc09c780 con 0x7f8dcc0772f0 2024-04-26T23:43:21.125 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-04-26T23:43:21.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.123+0000 7f8dcbe00640 1 -- 172.21.15.72:0/2630798730 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f8db00a3c30 msgr2=0x7f8db00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:21.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.123+0000 7f8dcbe00640 1 --2- 172.21.15.72:0/2630798730 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f8db00a3c30 0x7f8db00a60f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8db8000c00 tx=0x7f8db80314a0 comp rx=0 tx=0).stop 2024-04-26T23:43:21.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.123+0000 7f8dcbe00640 1 -- 172.21.15.72:0/2630798730 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8dcc0772f0 msgr2=0x7f8dcc19f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:21.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.123+0000 7f8dcbe00640 1 --2- 172.21.15.72:0/2630798730 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8dcc0772f0 0x7f8dcc19f090 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc00d950 tx=0x7f8dbc00de10 comp rx=0 tx=0).stop 2024-04-26T23:43:21.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.124+0000 7f8dcbe00640 1 -- 172.21.15.72:0/2630798730 shutdown_connections 2024-04-26T23:43:21.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.124+0000 7f8dcbe00640 1 --2- 172.21.15.72:0/2630798730 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f8db00a3c30 0x7f8db00a60f0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:21.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.124+0000 7f8dcbe00640 1 --2- 172.21.15.72:0/2630798730 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8dcc078ed0 0x7f8dcc1a6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:21.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.124+0000 7f8dcbe00640 1 --2- 172.21.15.72:0/2630798730 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8dcc078530 0x7f8dcc19f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:21.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.124+0000 7f8dcbe00640 1 --2- 172.21.15.72:0/2630798730 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8dcc0772f0 0x7f8dcc19f090 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:21.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.124+0000 7f8dcbe00640 1 -- 172.21.15.72:0/2630798730 >> 172.21.15.72:0/2630798730 conn(0x7f8dcc0fffe0 msgr2=0x7f8dcc101ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:21.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.124+0000 7f8dcbe00640 1 -- 172.21.15.72:0/2630798730 shutdown_connections 2024-04-26T23:43:21.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.124+0000 7f8dcbe00640 1 -- 172.21.15.72:0/2630798730 wait complete. 2024-04-26T23:43:21.146 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T23:43:21.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.344+0000 7f076f600640 1 -- 172.21.15.72:0/1820441762 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0768105d80 msgr2=0x7f0768106200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:21.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.344+0000 7f076f600640 1 --2- 172.21.15.72:0/1820441762 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0768105d80 0x7f0768106200 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0754009900 tx=0x7f075402a3f0 comp rx=0 tx=0).stop 2024-04-26T23:43:21.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.344+0000 7f076f600640 1 -- 172.21.15.72:0/1820441762 shutdown_connections 2024-04-26T23:43:21.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.344+0000 7f076f600640 1 --2- 172.21.15.72:0/1820441762 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0768106740 0x7f076810cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:21.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.344+0000 7f076f600640 1 --2- 172.21.15.72:0/1820441762 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0768105d80 0x7f0768106200 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:21.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.344+0000 7f076f600640 1 --2- 172.21.15.72:0/1820441762 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0768104b90 0x7f0768104fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:21.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.344+0000 7f076f600640 1 -- 172.21.15.72:0/1820441762 >> 172.21.15.72:0/1820441762 conn(0x7f0768100120 msgr2=0x7f0768102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:21.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.345+0000 7f076f600640 1 -- 172.21.15.72:0/1820441762 shutdown_connections 2024-04-26T23:43:21.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.345+0000 7f076f600640 1 -- 172.21.15.72:0/1820441762 wait complete. 2024-04-26T23:43:21.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.346+0000 7f076f600640 1 Processor -- start 2024-04-26T23:43:21.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.346+0000 7f076f600640 1 -- start start 2024-04-26T23:43:21.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.346+0000 7f076f600640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0768104b90 0x7f076819ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:21.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.346+0000 7f076f600640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0768105d80 0x7f076819b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:21.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.346+0000 7f076f600640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0768106740 0x7f07681a2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:21.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.346+0000 7f076f600640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f07681106a0 con 0x7f0768106740 2024-04-26T23:43:21.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.346+0000 7f076f600640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f0768110520 con 0x7f0768105d80 2024-04-26T23:43:21.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.346+0000 7f076f600640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f0768110820 con 0x7f0768104b90 2024-04-26T23:43:21.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.346+0000 7f076cc00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0768104b90 0x7f076819ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:21.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.346+0000 7f0767e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0768105d80 0x7f076819b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:21.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.346+0000 7f076cc00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0768104b90 0x7f076819ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.72:57568/0 (socket says 172.21.15.72:57568) 2024-04-26T23:43:21.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.346+0000 7f076cc00640 1 -- 172.21.15.72:0/1673371149 learned_addr learned my addr 172.21.15.72:0/1673371149 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:43:21.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.346+0000 7f076d600640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0768106740 0x7f07681a2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:21.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.347+0000 7f0767e00640 1 -- 172.21.15.72:0/1673371149 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0768104b90 msgr2=0x7f076819ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:21.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.347+0000 7f0767e00640 1 --2- 172.21.15.72:0/1673371149 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0768104b90 0x7f076819ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:21.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.347+0000 7f0767e00640 1 -- 172.21.15.72:0/1673371149 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0768106740 msgr2=0x7f07681a2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:21.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.347+0000 7f0767e00640 1 --2- 172.21.15.72:0/1673371149 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0768106740 0x7f07681a2390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:21.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.347+0000 7f0767e00640 1 -- 172.21.15.72:0/1673371149 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07681a2a00 con 0x7f0768105d80 2024-04-26T23:43:21.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.347+0000 7f076cc00640 1 --2- 172.21.15.72:0/1673371149 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0768104b90 0x7f076819ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:43:21.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.347+0000 7f0767e00640 1 --2- 172.21.15.72:0/1673371149 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0768105d80 0x7f076819b320 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0754006010 tx=0x7f07540042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:21.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.347+0000 7f0765600640 1 -- 172.21.15.72:0/1673371149 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f075403f030 con 0x7f0768105d80 2024-04-26T23:43:21.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.348+0000 7f076f600640 1 -- 172.21.15.72:0/1673371149 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07681a2c90 con 0x7f0768105d80 2024-04-26T23:43:21.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.348+0000 7f0765600640 1 -- 172.21.15.72:0/1673371149 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0754031910 con 0x7f0768105d80 2024-04-26T23:43:21.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.348+0000 7f0765600640 1 -- 172.21.15.72:0/1673371149 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0754035070 con 0x7f0768105d80 2024-04-26T23:43:21.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.348+0000 7f076f600640 1 -- 172.21.15.72:0/1673371149 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07681a31d0 con 0x7f0768105d80 2024-04-26T23:43:21.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.349+0000 7f076f600640 1 -- 172.21.15.72:0/1673371149 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f076810bfc0 con 0x7f0768105d80 2024-04-26T23:43:21.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.351+0000 7f0765600640 1 -- 172.21.15.72:0/1673371149 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f0754042030 con 0x7f0768105d80 2024-04-26T23:43:21.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.352+0000 7f0765600640 1 --2- 172.21.15.72:0/1673371149 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f07500a3bb0 0x7f07500a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:21.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.352+0000 7f0765600640 1 -- 172.21.15.72:0/1673371149 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f07540f0df0 con 0x7f0768105d80 2024-04-26T23:43:21.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.352+0000 7f076cc00640 1 --2- 172.21.15.72:0/1673371149 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f07500a3bb0 0x7f07500a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:21.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.353+0000 7f076cc00640 1 --2- 172.21.15.72:0/1673371149 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f07500a3bb0 0x7f07500a6070 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f075c0095f0 tx=0x7f075c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:21.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.354+0000 7f0765600640 1 -- 172.21.15.72:0/1673371149 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f07540bcee0 con 0x7f0768105d80 2024-04-26T23:43:21.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.557+0000 7f076f600640 1 -- 172.21.15.72:0/1673371149 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f076802dad0 con 0x7f0768105d80 2024-04-26T23:43:21.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.557+0000 7f0765600640 1 -- 172.21.15.72:0/1673371149 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7f07540c11e0 con 0x7f0768105d80 2024-04-26T23:43:21.561 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-26T23:43:21.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.560+0000 7f076f600640 1 -- 172.21.15.72:0/1673371149 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f07500a3bb0 msgr2=0x7f07500a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:21.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.560+0000 7f076f600640 1 --2- 172.21.15.72:0/1673371149 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f07500a3bb0 0x7f07500a6070 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f075c0095f0 tx=0x7f075c005f90 comp rx=0 tx=0).stop 2024-04-26T23:43:21.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.560+0000 7f076f600640 1 -- 172.21.15.72:0/1673371149 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0768105d80 msgr2=0x7f076819b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:21.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.560+0000 7f076f600640 1 --2- 172.21.15.72:0/1673371149 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0768105d80 0x7f076819b320 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0754006010 tx=0x7f07540042f0 comp rx=0 tx=0).stop 2024-04-26T23:43:21.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.560+0000 7f076f600640 1 -- 172.21.15.72:0/1673371149 shutdown_connections 2024-04-26T23:43:21.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.560+0000 7f076f600640 1 --2- 172.21.15.72:0/1673371149 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f07500a3bb0 0x7f07500a6070 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:21.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.560+0000 7f076f600640 1 --2- 172.21.15.72:0/1673371149 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0768105d80 0x7f076819b320 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:21.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.560+0000 7f076f600640 1 --2- 172.21.15.72:0/1673371149 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0768106740 0x7f07681a2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:21.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.560+0000 7f076f600640 1 --2- 172.21.15.72:0/1673371149 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0768104b90 0x7f076819ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:21.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.560+0000 7f076f600640 1 -- 172.21.15.72:0/1673371149 >> 172.21.15.72:0/1673371149 conn(0x7f0768100120 msgr2=0x7f0768109190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:21.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.560+0000 7f076f600640 1 -- 172.21.15.72:0/1673371149 shutdown_connections 2024-04-26T23:43:21.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.560+0000 7f076f600640 1 -- 172.21.15.72:0/1673371149 wait complete. 2024-04-26T23:43:21.579 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T23:43:21.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.759+0000 7fb99ac00640 1 -- 172.21.15.72:0/3126503271 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb994103940 msgr2=0x7fb994103dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:21.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.759+0000 7fb99ac00640 1 --2- 172.21.15.72:0/3126503271 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb994103940 0x7fb994103dc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb984009900 tx=0x7fb98402a3f0 comp rx=0 tx=0).stop 2024-04-26T23:43:21.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.759+0000 7fb99ac00640 1 -- 172.21.15.72:0/3126503271 shutdown_connections 2024-04-26T23:43:21.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.759+0000 7fb99ac00640 1 --2- 172.21.15.72:0/3126503271 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb994104300 0x7fb99410ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:21.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.759+0000 7fb99ac00640 1 --2- 172.21.15.72:0/3126503271 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb994103940 0x7fb994103dc0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:21.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.759+0000 7fb99ac00640 1 --2- 172.21.15.72:0/3126503271 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb994102750 0x7fb994102b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:21.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.759+0000 7fb99ac00640 1 -- 172.21.15.72:0/3126503271 >> 172.21.15.72:0/3126503271 conn(0x7fb9940fdcc0 msgr2=0x7fb994100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:21.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.759+0000 7fb99ac00640 1 -- 172.21.15.72:0/3126503271 shutdown_connections 2024-04-26T23:43:21.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.759+0000 7fb99ac00640 1 -- 172.21.15.72:0/3126503271 wait complete. 2024-04-26T23:43:21.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.760+0000 7fb99ac00640 1 Processor -- start 2024-04-26T23:43:21.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.760+0000 7fb99ac00640 1 -- start start 2024-04-26T23:43:21.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.760+0000 7fb99ac00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb994102750 0x7fb99419ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:21.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.760+0000 7fb99ac00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb994103940 0x7fb99419b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:21.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.760+0000 7fb99ac00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb994104300 0x7fb9941a20e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:21.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.760+0000 7fb99ac00640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fb99410e280 con 0x7fb994102750 2024-04-26T23:43:21.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.760+0000 7fb99ac00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb99410e100 con 0x7fb994104300 2024-04-26T23:43:21.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.760+0000 7fb99ac00640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fb99410e400 con 0x7fb994103940 2024-04-26T23:43:21.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.761+0000 7fb999800640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb994102750 0x7fb99419ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:21.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.761+0000 7fb99a200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb994104300 0x7fb9941a20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:21.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.761+0000 7fb99a200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb994104300 0x7fb9941a20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:52366/0 (socket says 172.21.15.72:52366) 2024-04-26T23:43:21.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.761+0000 7fb99a200640 1 -- 172.21.15.72:0/345668243 learned_addr learned my addr 172.21.15.72:0/345668243 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:43:21.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.761+0000 7fb998e00640 1 --2- 172.21.15.72:0/345668243 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb994103940 0x7fb99419b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:21.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.761+0000 7fb99a200640 1 -- 172.21.15.72:0/345668243 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb994103940 msgr2=0x7fb99419b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:21.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.761+0000 7fb99a200640 1 --2- 172.21.15.72:0/345668243 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb994103940 0x7fb99419b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:21.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.761+0000 7fb99a200640 1 -- 172.21.15.72:0/345668243 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb994102750 msgr2=0x7fb99419ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:21.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.761+0000 7fb99a200640 1 --2- 172.21.15.72:0/345668243 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb994102750 0x7fb99419ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:21.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.761+0000 7fb99a200640 1 -- 172.21.15.72:0/345668243 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9941a2750 con 0x7fb994104300 2024-04-26T23:43:21.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.761+0000 7fb999800640 1 --2- 172.21.15.72:0/345668243 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb994102750 0x7fb99419ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:43:21.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.762+0000 7fb99a200640 1 --2- 172.21.15.72:0/345668243 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb994104300 0x7fb9941a20e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb98c00eb10 tx=0x7fb98c00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:21.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.762+0000 7fb992000640 1 -- 172.21.15.72:0/345668243 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb98c004b20 con 0x7fb994104300 2024-04-26T23:43:21.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.762+0000 7fb99ac00640 1 -- 172.21.15.72:0/345668243 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9941a2a40 con 0x7fb994104300 2024-04-26T23:43:21.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.762+0000 7fb992000640 1 -- 172.21.15.72:0/345668243 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb98c004cc0 con 0x7fb994104300 2024-04-26T23:43:21.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.762+0000 7fb992000640 1 -- 172.21.15.72:0/345668243 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb98c010690 con 0x7fb994104300 2024-04-26T23:43:21.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.762+0000 7fb99ac00640 1 -- 172.21.15.72:0/345668243 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9941a2f60 con 0x7fb994104300 2024-04-26T23:43:21.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.763+0000 7fb99ac00640 1 -- 172.21.15.72:0/345668243 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9941a3190 con 0x7fb994104300 2024-04-26T23:43:21.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.765+0000 7fb992000640 1 -- 172.21.15.72:0/345668243 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb98c005020 con 0x7fb994104300 2024-04-26T23:43:21.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.766+0000 7fb992000640 1 --2- 172.21.15.72:0/345668243 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fb97c0a3bb0 0x7fb97c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:21.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.767+0000 7fb992000640 1 -- 172.21.15.72:0/345668243 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fb98c0d0ba0 con 0x7fb994104300 2024-04-26T23:43:21.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.767+0000 7fb999800640 1 --2- 172.21.15.72:0/345668243 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fb97c0a3bb0 0x7fb97c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:21.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.767+0000 7fb999800640 1 --2- 172.21.15.72:0/345668243 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fb97c0a3bb0 0x7fb97c0a6070 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb9941037a0 tx=0x7fb988005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:21.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.769+0000 7fb992000640 1 -- 172.21.15.72:0/345668243 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb98c09d2e0 con 0x7fb994104300 2024-04-26T23:43:21.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:21 smithi076 ceph-mon[32774]: pgmap v330: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-04-26T23:43:21.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:21 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/110901178' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:43:21.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:21 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/2630798730' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T23:43:21.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:21 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/1673371149' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T23:43:21.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.970+0000 7fb99ac00640 1 -- 172.21.15.72:0/345668243 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb9941a36d0 con 0x7fb994104300 2024-04-26T23:43:21.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.971+0000 7fb992000640 1 -- 172.21.15.72:0/345668243 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7fb98c09cbe0 con 0x7fb994104300 2024-04-26T23:43:21.976 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-04-26T23:43:21.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.974+0000 7fb99ac00640 1 -- 172.21.15.72:0/345668243 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fb97c0a3bb0 msgr2=0x7fb97c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:21.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.974+0000 7fb99ac00640 1 --2- 172.21.15.72:0/345668243 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fb97c0a3bb0 0x7fb97c0a6070 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb9941037a0 tx=0x7fb988005f90 comp rx=0 tx=0).stop 2024-04-26T23:43:21.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.974+0000 7fb99ac00640 1 -- 172.21.15.72:0/345668243 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb994104300 msgr2=0x7fb9941a20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:21.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.974+0000 7fb99ac00640 1 --2- 172.21.15.72:0/345668243 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb994104300 0x7fb9941a20e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb98c00eb10 tx=0x7fb98c00c2d0 comp rx=0 tx=0).stop 2024-04-26T23:43:21.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.974+0000 7fb99ac00640 1 -- 172.21.15.72:0/345668243 shutdown_connections 2024-04-26T23:43:21.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.974+0000 7fb99ac00640 1 --2- 172.21.15.72:0/345668243 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fb97c0a3bb0 0x7fb97c0a6070 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:21.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.974+0000 7fb99ac00640 1 --2- 172.21.15.72:0/345668243 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb994104300 0x7fb9941a20e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:21.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.974+0000 7fb99ac00640 1 --2- 172.21.15.72:0/345668243 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb994103940 0x7fb99419b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:21.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.974+0000 7fb99ac00640 1 --2- 172.21.15.72:0/345668243 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb994102750 0x7fb99419ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:21.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.974+0000 7fb99ac00640 1 -- 172.21.15.72:0/345668243 >> 172.21.15.72:0/345668243 conn(0x7fb9940fdcc0 msgr2=0x7fb994074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:21.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.974+0000 7fb99ac00640 1 -- 172.21.15.72:0/345668243 shutdown_connections 2024-04-26T23:43:21.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:21.974+0000 7fb99ac00640 1 -- 172.21.15.72:0/345668243 wait complete. 2024-04-26T23:43:21.994 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:21 smithi072 ceph-mon[28184]: pgmap v330: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-04-26T23:43:21.994 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:21 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/110901178' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:43:21.995 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:21 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/2630798730' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T23:43:21.995 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:21 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/1673371149' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T23:43:21.997 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-26T23:43:22.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:21 smithi136 ceph-mon[33050]: pgmap v330: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-04-26T23:43:22.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:21 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/110901178' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:43:22.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:21 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/2630798730' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T23:43:22.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:21 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/1673371149' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T23:43:22.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.198+0000 7f0bbc200640 1 -- 172.21.15.72:0/413553173 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0bb4101240 msgr2=0x7f0bb4101660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:22.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.198+0000 7f0bbc200640 1 --2- 172.21.15.72:0/413553173 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0bb4101240 0x7f0bb4101660 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0ba8009900 tx=0x7f0ba802a3f0 comp rx=0 tx=0).stop 2024-04-26T23:43:22.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.199+0000 7f0bbc200640 1 -- 172.21.15.72:0/413553173 shutdown_connections 2024-04-26T23:43:22.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.199+0000 7f0bbc200640 1 --2- 172.21.15.72:0/413553173 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0bb4102540 0x7f0bb41029c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:22.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.199+0000 7f0bbc200640 1 --2- 172.21.15.72:0/413553173 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0bb4101ba0 0x7f0bb4102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:22.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.199+0000 7f0bbc200640 1 --2- 172.21.15.72:0/413553173 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0bb4101240 0x7f0bb4101660 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:22.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.199+0000 7f0bbc200640 1 -- 172.21.15.72:0/413553173 >> 172.21.15.72:0/413553173 conn(0x7f0bb40fcde0 msgr2=0x7f0bb40ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:22.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.199+0000 7f0bbc200640 1 -- 172.21.15.72:0/413553173 shutdown_connections 2024-04-26T23:43:22.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.199+0000 7f0bbc200640 1 -- 172.21.15.72:0/413553173 wait complete. 2024-04-26T23:43:22.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.200+0000 7f0bbc200640 1 Processor -- start 2024-04-26T23:43:22.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.200+0000 7f0bbc200640 1 -- start start 2024-04-26T23:43:22.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.200+0000 7f0bbc200640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0bb4101240 0x7f0bb419adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:22.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.200+0000 7f0bbc200640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0bb4101ba0 0x7f0bb419b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:22.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.200+0000 7f0bbc200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0bb4102540 0x7f0bb41a2360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:22.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.200+0000 7f0bbc200640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f0bb41106b0 con 0x7f0bb4101240 2024-04-26T23:43:22.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.200+0000 7f0bbc200640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f0bb4110530 con 0x7f0bb4102540 2024-04-26T23:43:22.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.200+0000 7f0bbc200640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f0bb4110830 con 0x7f0bb4101ba0 2024-04-26T23:43:22.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.200+0000 7f0bb9800640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0bb4101240 0x7f0bb419adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:22.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.200+0000 7f0bb9800640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0bb4101240 0x7f0bb419adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56744/0 (socket says 172.21.15.72:56744) 2024-04-26T23:43:22.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.200+0000 7f0bb9800640 1 -- 172.21.15.72:0/1941207493 learned_addr learned my addr 172.21.15.72:0/1941207493 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:43:22.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.201+0000 7f0bb8e00640 1 --2- 172.21.15.72:0/1941207493 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0bb4101ba0 0x7f0bb419b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:22.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.201+0000 7f0bba200640 1 --2- 172.21.15.72:0/1941207493 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0bb4102540 0x7f0bb41a2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:22.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.201+0000 7f0bb9800640 1 -- 172.21.15.72:0/1941207493 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0bb4101ba0 msgr2=0x7f0bb419b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:22.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.201+0000 7f0bb9800640 1 --2- 172.21.15.72:0/1941207493 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0bb4101ba0 0x7f0bb419b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:22.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.201+0000 7f0bb9800640 1 -- 172.21.15.72:0/1941207493 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0bb4102540 msgr2=0x7f0bb41a2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:22.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.201+0000 7f0bb9800640 1 --2- 172.21.15.72:0/1941207493 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0bb4102540 0x7f0bb41a2360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:22.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.201+0000 7f0bb9800640 1 -- 172.21.15.72:0/1941207493 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bb41a29d0 con 0x7f0bb4101240 2024-04-26T23:43:22.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.201+0000 7f0bba200640 1 --2- 172.21.15.72:0/1941207493 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0bb4102540 0x7f0bb41a2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:43:22.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.201+0000 7f0bb9800640 1 --2- 172.21.15.72:0/1941207493 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0bb4101240 0x7f0bb419adb0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0ba8005ab0 tx=0x7f0ba8003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:22.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.202+0000 7f0bb2000640 1 -- 172.21.15.72:0/1941207493 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ba803f070 con 0x7f0bb4101240 2024-04-26T23:43:22.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.202+0000 7f0bb2000640 1 -- 172.21.15.72:0/1941207493 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ba8031c50 con 0x7f0bb4101240 2024-04-26T23:43:22.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.202+0000 7f0bbc200640 1 -- 172.21.15.72:0/1941207493 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bb41a2c60 con 0x7f0bb4101240 2024-04-26T23:43:22.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.202+0000 7f0bb2000640 1 -- 172.21.15.72:0/1941207493 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ba8035030 con 0x7f0bb4101240 2024-04-26T23:43:22.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.202+0000 7f0bbc200640 1 -- 172.21.15.72:0/1941207493 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bb41a3150 con 0x7f0bb4101240 2024-04-26T23:43:22.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.203+0000 7f0bbc200640 1 -- 172.21.15.72:0/1941207493 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bb410c220 con 0x7f0bb4101240 2024-04-26T23:43:22.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.205+0000 7f0bb2000640 1 -- 172.21.15.72:0/1941207493 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f0ba8042070 con 0x7f0bb4101240 2024-04-26T23:43:22.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.206+0000 7f0bb2000640 1 --2- 172.21.15.72:0/1941207493 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f0b9c0a3bb0 0x7f0b9c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:22.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.207+0000 7f0bb2000640 1 -- 172.21.15.72:0/1941207493 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f0ba80f0dd0 con 0x7f0bb4101240 2024-04-26T23:43:22.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.207+0000 7f0bb8e00640 1 --2- 172.21.15.72:0/1941207493 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f0b9c0a3bb0 0x7f0b9c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:22.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.208+0000 7f0bb8e00640 1 --2- 172.21.15.72:0/1941207493 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f0b9c0a3bb0 0x7f0b9c0a6070 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0bb419c2c0 tx=0x7f0ba0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:22.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.210+0000 7f0bb2000640 1 -- 172.21.15.72:0/1941207493 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f0ba80bcec0 con 0x7f0bb4101240 2024-04-26T23:43:22.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.408+0000 7f0bbc200640 1 -- 172.21.15.72:0/1941207493 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f0bb4063560 con 0x7f0bb4101240 2024-04-26T23:43:22.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.409+0000 7f0bb2000640 1 -- 172.21.15.72:0/1941207493 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) ==== 103+0+11 (secure 0 0 0) 0x7f0ba80c11c0 con 0x7f0bb4101240 2024-04-26T23:43:22.413 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-26T23:43:22.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.411+0000 7f0bbc200640 1 -- 172.21.15.72:0/1941207493 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f0b9c0a3bb0 msgr2=0x7f0b9c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:22.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.411+0000 7f0bbc200640 1 --2- 172.21.15.72:0/1941207493 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f0b9c0a3bb0 0x7f0b9c0a6070 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0bb419c2c0 tx=0x7f0ba0005f90 comp rx=0 tx=0).stop 2024-04-26T23:43:22.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.411+0000 7f0bbc200640 1 -- 172.21.15.72:0/1941207493 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0bb4101240 msgr2=0x7f0bb419adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:22.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.411+0000 7f0bbc200640 1 --2- 172.21.15.72:0/1941207493 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0bb4101240 0x7f0bb419adb0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0ba8005ab0 tx=0x7f0ba8003b90 comp rx=0 tx=0).stop 2024-04-26T23:43:22.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.411+0000 7f0bbc200640 1 -- 172.21.15.72:0/1941207493 shutdown_connections 2024-04-26T23:43:22.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.411+0000 7f0bbc200640 1 --2- 172.21.15.72:0/1941207493 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f0b9c0a3bb0 0x7f0b9c0a6070 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:22.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.411+0000 7f0bbc200640 1 --2- 172.21.15.72:0/1941207493 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0bb4102540 0x7f0bb41a2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:22.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.411+0000 7f0bbc200640 1 --2- 172.21.15.72:0/1941207493 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0bb4101ba0 0x7f0bb419b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:22.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.411+0000 7f0bbc200640 1 --2- 172.21.15.72:0/1941207493 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0bb4101240 0x7f0bb419adb0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:22.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.411+0000 7f0bbc200640 1 -- 172.21.15.72:0/1941207493 >> 172.21.15.72:0/1941207493 conn(0x7f0bb40fcde0 msgr2=0x7f0bb40fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:22.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.412+0000 7f0bbc200640 1 -- 172.21.15.72:0/1941207493 shutdown_connections 2024-04-26T23:43:22.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.412+0000 7f0bbc200640 1 -- 172.21.15.72:0/1941207493 wait complete. 2024-04-26T23:43:22.434 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-26T23:43:22.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.627+0000 7fc312c00640 1 -- 172.21.15.72:0/1832534315 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc30c10c360 msgr2=0x7fc30c10c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:22.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.627+0000 7fc312c00640 1 --2- 172.21.15.72:0/1832534315 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc30c10c360 0x7fc30c10c7e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8009900 tx=0x7fc2f802a3f0 comp rx=0 tx=0).stop 2024-04-26T23:43:22.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.627+0000 7fc312c00640 1 -- 172.21.15.72:0/1832534315 shutdown_connections 2024-04-26T23:43:22.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.628+0000 7fc312c00640 1 --2- 172.21.15.72:0/1832534315 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc30c10cd20 0x7fc30c1135b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:22.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.628+0000 7fc312c00640 1 --2- 172.21.15.72:0/1832534315 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc30c10c360 0x7fc30c10c7e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:22.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.628+0000 7fc312c00640 1 --2- 172.21.15.72:0/1832534315 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc30c10b170 0x7fc30c10b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:22.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.628+0000 7fc312c00640 1 -- 172.21.15.72:0/1832534315 >> 172.21.15.72:0/1832534315 conn(0x7fc30c071900 msgr2=0x7fc30c073d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:22.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.628+0000 7fc312c00640 1 -- 172.21.15.72:0/1832534315 shutdown_connections 2024-04-26T23:43:22.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.628+0000 7fc312c00640 1 -- 172.21.15.72:0/1832534315 wait complete. 2024-04-26T23:43:22.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.629+0000 7fc312c00640 1 Processor -- start 2024-04-26T23:43:22.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.629+0000 7fc312c00640 1 -- start start 2024-04-26T23:43:22.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.629+0000 7fc312c00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc30c10b170 0x7fc30c06dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:22.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.629+0000 7fc312c00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc30c10c360 0x7fc30c06c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:22.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.629+0000 7fc312c00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc30c10cd20 0x7fc30c06c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:22.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.629+0000 7fc312c00640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fc30c116cb0 con 0x7fc30c10cd20 2024-04-26T23:43:22.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.629+0000 7fc312c00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc30c116b30 con 0x7fc30c10c360 2024-04-26T23:43:22.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.629+0000 7fc312c00640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fc30c116e30 con 0x7fc30c10b170 2024-04-26T23:43:22.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.630+0000 7fc30be00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc30c10b170 0x7fc30c06dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:22.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.630+0000 7fc310c00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc30c10cd20 0x7fc30c06c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:22.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.630+0000 7fc30b400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc30c10c360 0x7fc30c06c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:22.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.630+0000 7fc30be00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc30c10b170 0x7fc30c06dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.72:57626/0 (socket says 172.21.15.72:57626) 2024-04-26T23:43:22.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.630+0000 7fc30b400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc30c10c360 0x7fc30c06c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:52420/0 (socket says 172.21.15.72:52420) 2024-04-26T23:43:22.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.630+0000 7fc30be00640 1 -- 172.21.15.72:0/541624469 learned_addr learned my addr 172.21.15.72:0/541624469 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:43:22.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.630+0000 7fc30be00640 1 -- 172.21.15.72:0/541624469 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc30c10c360 msgr2=0x7fc30c06c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:22.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.630+0000 7fc30be00640 1 --2- 172.21.15.72:0/541624469 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc30c10c360 0x7fc30c06c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:22.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.630+0000 7fc30be00640 1 -- 172.21.15.72:0/541624469 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc30c10cd20 msgr2=0x7fc30c06c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:22.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.630+0000 7fc30be00640 1 --2- 172.21.15.72:0/541624469 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc30c10cd20 0x7fc30c06c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:22.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.630+0000 7fc30be00640 1 -- 172.21.15.72:0/541624469 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc30c06d050 con 0x7fc30c10b170 2024-04-26T23:43:22.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.630+0000 7fc30be00640 1 --2- 172.21.15.72:0/541624469 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc30c10b170 0x7fc30c06dc40 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc00e9c0 tx=0x7fc2fc00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:22.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.631+0000 7fc308c00640 1 -- 172.21.15.72:0/541624469 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2fc00cde0 con 0x7fc30c10b170 2024-04-26T23:43:22.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.631+0000 7fc308c00640 1 -- 172.21.15.72:0/541624469 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2fc0108e0 con 0x7fc30c10b170 2024-04-26T23:43:22.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.631+0000 7fc308c00640 1 -- 172.21.15.72:0/541624469 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2fc018960 con 0x7fc30c10b170 2024-04-26T23:43:22.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.631+0000 7fc312c00640 1 -- 172.21.15.72:0/541624469 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc30c1ab4c0 con 0x7fc30c10b170 2024-04-26T23:43:22.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.631+0000 7fc312c00640 1 -- 172.21.15.72:0/541624469 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc30c1ab990 con 0x7fc30c10b170 2024-04-26T23:43:22.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.632+0000 7fc312c00640 1 -- 172.21.15.72:0/541624469 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc30c112510 con 0x7fc30c10b170 2024-04-26T23:43:22.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.634+0000 7fc308c00640 1 -- 172.21.15.72:0/541624469 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fc2fc010a90 con 0x7fc30c10b170 2024-04-26T23:43:22.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.635+0000 7fc308c00640 1 --2- 172.21.15.72:0/541624469 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fc2f00a3bb0 0x7fc2f00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:22.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.635+0000 7fc308c00640 1 -- 172.21.15.72:0/541624469 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fc2fc0d0b00 con 0x7fc30c10b170 2024-04-26T23:43:22.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.635+0000 7fc30b400640 1 --2- 172.21.15.72:0/541624469 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fc2f00a3bb0 0x7fc2f00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:22.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.636+0000 7fc30b400640 1 --2- 172.21.15.72:0/541624469 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fc2f00a3bb0 0x7fc2f00a6070 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8000c00 tx=0x7fc2f80314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:22.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.639+0000 7fc308c00640 1 -- 172.21.15.72:0/541624469 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc2fc054060 con 0x7fc30c10b170 2024-04-26T23:43:22.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:22 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/345668243' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T23:43:22.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:22 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/1941207493' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T23:43:22.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.855+0000 7fc312c00640 1 -- 172.21.15.72:0/541624469 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fc30c02dad0 con 0x7fc30c10b170 2024-04-26T23:43:22.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.856+0000 7fc308c00640 1 -- 172.21.15.72:0/541624469 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+591 (secure 0 0 0) 0x7fc2fc09d2a0 con 0x7fc30c10b170 2024-04-26T23:43:22.860 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:0/3625071330 2024-04-27T23:34:58.266218+0000 2024-04-26T23:43:22.860 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:6801/2903979134 2024-04-27T23:34:58.266218+0000 2024-04-26T23:43:22.860 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:6800/2903979134 2024-04-27T23:34:58.266218+0000 2024-04-26T23:43:22.860 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:0/1519142576 2024-04-27T23:34:58.266218+0000 2024-04-26T23:43:22.861 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:0/3669626460 2024-04-27T23:34:24.233319+0000 2024-04-26T23:43:22.861 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:0/945623297 2024-04-27T23:34:24.233319+0000 2024-04-26T23:43:22.861 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:0/3479150073 2024-04-27T23:34:24.233319+0000 2024-04-26T23:43:22.861 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:6800/4175896290 2024-04-27T23:34:24.233319+0000 2024-04-26T23:43:22.861 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:6801/4175896290 2024-04-27T23:34:24.233319+0000 2024-04-26T23:43:22.861 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:0/3783175012 2024-04-27T23:34:58.266218+0000 2024-04-26T23:43:22.861 INFO:teuthology.orchestra.run.smithi072.stderr:listed 10 entries 2024-04-26T23:43:22.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.859+0000 7fc312c00640 1 -- 172.21.15.72:0/541624469 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fc2f00a3bb0 msgr2=0x7fc2f00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:22.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.859+0000 7fc312c00640 1 --2- 172.21.15.72:0/541624469 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fc2f00a3bb0 0x7fc2f00a6070 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8000c00 tx=0x7fc2f80314a0 comp rx=0 tx=0).stop 2024-04-26T23:43:22.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.859+0000 7fc312c00640 1 -- 172.21.15.72:0/541624469 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc30c10b170 msgr2=0x7fc30c06dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:22.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.859+0000 7fc312c00640 1 --2- 172.21.15.72:0/541624469 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc30c10b170 0x7fc30c06dc40 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc00e9c0 tx=0x7fc2fc00ee80 comp rx=0 tx=0).stop 2024-04-26T23:43:22.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.859+0000 7fc312c00640 1 -- 172.21.15.72:0/541624469 shutdown_connections 2024-04-26T23:43:22.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.859+0000 7fc312c00640 1 --2- 172.21.15.72:0/541624469 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fc2f00a3bb0 0x7fc2f00a6070 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:22.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.859+0000 7fc312c00640 1 --2- 172.21.15.72:0/541624469 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc30c10c360 0x7fc30c06c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:22.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.859+0000 7fc312c00640 1 --2- 172.21.15.72:0/541624469 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc30c10cd20 0x7fc30c06c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:22.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.859+0000 7fc312c00640 1 --2- 172.21.15.72:0/541624469 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc30c10b170 0x7fc30c06dc40 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:22.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.859+0000 7fc312c00640 1 -- 172.21.15.72:0/541624469 >> 172.21.15.72:0/541624469 conn(0x7fc30c071900 msgr2=0x7fc30c10f6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:22.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.859+0000 7fc312c00640 1 -- 172.21.15.72:0/541624469 shutdown_connections 2024-04-26T23:43:22.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:22.859+0000 7fc312c00640 1 -- 172.21.15.72:0/541624469 wait complete. 2024-04-26T23:43:22.881 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-26T23:43:22.881 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:43:22.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:22 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/345668243' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T23:43:22.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:22 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/1941207493' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T23:43:22.931 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:43:23.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:22 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/345668243' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T23:43:23.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:22 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/1941207493' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T23:43:23.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.176+0000 7f042ae00640 1 -- 172.21.15.72:0/1351488277 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0424103b60 msgr2=0x7f0424103fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:23.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.176+0000 7f042ae00640 1 --2- 172.21.15.72:0/1351488277 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0424103b60 0x7f0424103fe0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0410009900 tx=0x7f041002a3f0 comp rx=0 tx=0).stop 2024-04-26T23:43:23.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.176+0000 7f042ae00640 1 -- 172.21.15.72:0/1351488277 shutdown_connections 2024-04-26T23:43:23.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.176+0000 7f042ae00640 1 --2- 172.21.15.72:0/1351488277 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0424104520 0x7f042410adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:23.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.176+0000 7f042ae00640 1 --2- 172.21.15.72:0/1351488277 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0424103b60 0x7f0424103fe0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:23.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.176+0000 7f042ae00640 1 --2- 172.21.15.72:0/1351488277 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0424102970 0x7f0424102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:23.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.176+0000 7f042ae00640 1 -- 172.21.15.72:0/1351488277 >> 172.21.15.72:0/1351488277 conn(0x7f04240fdee0 msgr2=0x7f0424100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:23.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.176+0000 7f042ae00640 1 -- 172.21.15.72:0/1351488277 shutdown_connections 2024-04-26T23:43:23.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.176+0000 7f042ae00640 1 -- 172.21.15.72:0/1351488277 wait complete. 2024-04-26T23:43:23.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.177+0000 7f042ae00640 1 Processor -- start 2024-04-26T23:43:23.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.177+0000 7f042ae00640 1 -- start start 2024-04-26T23:43:23.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.178+0000 7f042ae00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0424102970 0x7f042419adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:23.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.178+0000 7f042ae00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0424103b60 0x7f042419b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:23.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.178+0000 7f042ae00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0424104520 0x7f04241a2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:23.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.178+0000 7f042ae00640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f042410e4a0 con 0x7f0424103b60 2024-04-26T23:43:23.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.178+0000 7f042ae00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f042410e320 con 0x7f0424102970 2024-04-26T23:43:23.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.178+0000 7f042ae00640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f042410e620 con 0x7f0424104520 2024-04-26T23:43:23.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.178+0000 7f0423400640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0424103b60 0x7f042419b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:23.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.178+0000 7f0423400640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0424103b60 0x7f042419b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56780/0 (socket says 172.21.15.72:56780) 2024-04-26T23:43:23.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.178+0000 7f0423400640 1 -- 172.21.15.72:0/3016248482 learned_addr learned my addr 172.21.15.72:0/3016248482 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:43:23.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.178+0000 7f0423e00640 1 --2- 172.21.15.72:0/3016248482 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0424102970 0x7f042419adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:23.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.178+0000 7f0428e00640 1 --2- 172.21.15.72:0/3016248482 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0424104520 0x7f04241a2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:23.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.179+0000 7f0428e00640 1 -- 172.21.15.72:0/3016248482 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0424102970 msgr2=0x7f042419adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:23.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.179+0000 7f0428e00640 1 --2- 172.21.15.72:0/3016248482 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0424102970 0x7f042419adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:23.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.179+0000 7f0428e00640 1 -- 172.21.15.72:0/3016248482 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0424103b60 msgr2=0x7f042419b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:23.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.179+0000 7f0428e00640 1 --2- 172.21.15.72:0/3016248482 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0424103b60 0x7f042419b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:23.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.179+0000 7f0428e00640 1 -- 172.21.15.72:0/3016248482 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04241a29c0 con 0x7f0424104520 2024-04-26T23:43:23.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.179+0000 7f0428e00640 1 --2- 172.21.15.72:0/3016248482 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0424104520 0x7f04241a2320 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f041c009fd0 tx=0x7f041c00ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:23.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.179+0000 7f0420c00640 1 -- 172.21.15.72:0/3016248482 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f041c00ce20 con 0x7f0424104520 2024-04-26T23:43:23.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.179+0000 7f042ae00640 1 -- 172.21.15.72:0/3016248482 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04241a2cb0 con 0x7f0424104520 2024-04-26T23:43:23.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.179+0000 7f0420c00640 1 -- 172.21.15.72:0/3016248482 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f041c005020 con 0x7f0424104520 2024-04-26T23:43:23.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.179+0000 7f0420c00640 1 -- 172.21.15.72:0/3016248482 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f041c010690 con 0x7f0424104520 2024-04-26T23:43:23.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.179+0000 7f042ae00640 1 -- 172.21.15.72:0/3016248482 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04241a31a0 con 0x7f0424104520 2024-04-26T23:43:23.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.180+0000 7f042ae00640 1 -- 172.21.15.72:0/3016248482 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04241a3430 con 0x7f0424104520 2024-04-26T23:43:23.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.183+0000 7f0420c00640 1 -- 172.21.15.72:0/3016248482 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f041c0054d0 con 0x7f0424104520 2024-04-26T23:43:23.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.184+0000 7f0420c00640 1 --2- 172.21.15.72:0/3016248482 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f04080a3bb0 0x7f04080a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:23.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.185+0000 7f0420c00640 1 -- 172.21.15.72:0/3016248482 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f041c0d0d70 con 0x7f0424104520 2024-04-26T23:43:23.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.185+0000 7f0423e00640 1 --2- 172.21.15.72:0/3016248482 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f04080a3bb0 0x7f04080a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:23.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.185+0000 7f0423e00640 1 --2- 172.21.15.72:0/3016248482 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f04080a3bb0 0x7f04080a6070 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f042419c280 tx=0x7f0414005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:23.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.186+0000 7f0420c00640 1 -- 172.21.15.72:0/3016248482 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f041c09d4b0 con 0x7f0424104520 2024-04-26T23:43:23.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.392+0000 7f042ae00640 1 -- 172.21.15.72:0/3016248482 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f04241a3770 con 0x7f0424104520 2024-04-26T23:43:23.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.394+0000 7f0420c00640 1 -- 172.21.15.72:0/3016248482 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24081 (secure 0 0 0) 0x7f041c09cdb0 con 0x7f0424104520 2024-04-26T23:43:23.399 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:43:23.399 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"fsid":"3c8453ae-0425-11ef-bc93-c7b262605968","created":"2024-04-26T23:33:48.088300+0000","modified":"2024-04-26T23:43:07.990780+0000","last_up_change":"2024-04-26T23:42:04.482891+0000","last_in_change":"2024-04-26T23:41:44.784415+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:37:44.428727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:43:00.431829+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:43:01.872977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:43:05.512050+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e20c9ebf-56f6-47c1-a11b-e1e615af042b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6803","nonce":1639298875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6805","nonce":1639298875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6809","nonce":1639298875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6807","nonce":1639298875}]},"public_addr":"172.21.15.72:6803/1639298875","cluster_addr":"172.21.15.72:6805/1639298875","heartbeat_back_addr":"172.21.15.72:6809/1639298875","heartbeat_front_addr":"172.21.15.72:6807/1639298875","state":["exists","up"]},{"osd":1,"uuid":"e6cac469-780c-4d66-b560-b686072371fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6801","nonce":4185342193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6803","nonce":4185342193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6807","nonce":4185342193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6805","nonce":4185342193}]},"public_addr":"172.21.15.76:6801/4185342193","cluster_addr":"172.21.15.76:6803/4185342193","heartbeat_back_addr":"172.21.15.76:6807/4185342193","heartbeat_front_addr":"172.21.15.76:6805/4185342193","state":["exists","up"]},{"osd":2,"uuid":"838a49fc-18de-446c-8336-67b3e05da231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6801","nonce":3033860799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6803","nonce":3033860799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6807","nonce":3033860799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6805","nonce":3033860799}]},"public_addr":"172.21.15.136:6801/3033860799","cluster_addr":"172.21.15.136:6803/3033860799","heartbeat_back_addr":"172.21.15.136:6807/3033860799","heartbeat_front_addr":"172.21.15.136:6805/3033860799","state":["exists","up"]},{"osd":3,"uuid":"e6da89be-5ba7-48a4-9306-f75498660e0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6811","nonce":2071107815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6813","nonce":2071107815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6817","nonce":2071107815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6815","nonce":2071107815}]},"public_addr":"172.21.15.72:6811/2071107815","cluster_addr":"172.21.15.72:6813/2071107815","heartbeat_back_addr":"172.21.15.72:6817/2071107815","heartbeat_front_addr":"172.21.15.72:6815/2071107815","state":["exists","up"]},{"osd":4,"uuid":"0b17c491-d798-488f-892d-521ef2fbc51e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6809","nonce":147449238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6811","nonce":147449238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6815","nonce":147449238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6813","nonce":147449238}]},"public_addr":"172.21.15.76:6809/147449238","cluster_addr":"172.21.15.76:6811/147449238","heartbeat_back_addr":"172.21.15.76:6815/147449238","heartbeat_front_addr":"172.21.15.76:6813/147449238","state":["exists","up"]},{"osd":5,"uuid":"c7e887b3-ab38-4608-9bfd-1ed49956e8af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6809","nonce":2320149212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6811","nonce":2320149212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6815","nonce":2320149212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6813","nonce":2320149212}]},"public_addr":"172.21.15.136:6809/2320149212","cluster_addr":"172.21.15.136:6811/2320149212","heartbeat_back_addr":"172.21.15.136:6815/2320149212","heartbeat_front_addr":"172.21.15.136:6813/2320149212","state":["exists","up"]},{"osd":6,"uuid":"f0bd604b-0f57-4a38-9295-eb200264f1ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6819","nonce":389499207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6821","nonce":389499207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6825","nonce":389499207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6823","nonce":389499207}]},"public_addr":"172.21.15.72:6819/389499207","cluster_addr":"172.21.15.72:6821/389499207","heartbeat_back_addr":"172.21.15.72:6825/389499207","heartbeat_front_addr":"172.21.15.72:6823/389499207","state":["exists","up"]},{"osd":7,"uuid":"1879d4e4-038e-4cd6-a3ec-3a940bcc3797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6817","nonce":375888575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6819","nonce":375888575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6823","nonce":375888575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6821","nonce":375888575}]},"public_addr":"172.21.15.76:6817/375888575","cluster_addr":"172.21.15.76:6819/375888575","heartbeat_back_addr":"172.21.15.76:6823/375888575","heartbeat_front_addr":"172.21.15.76:6821/375888575","state":["exists","up"]},{"osd":8,"uuid":"6f404062-97e7-4ccb-a582-76dc1ffae548","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6817","nonce":3446774052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6819","nonce":3446774052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6823","nonce":3446774052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6821","nonce":3446774052}]},"public_addr":"172.21.15.136:6817/3446774052","cluster_addr":"172.21.15.136:6819/3446774052","heartbeat_back_addr":"172.21.15.136:6823/3446774052","heartbeat_front_addr":"172.21.15.136:6821/3446774052","state":["exists","up"]},{"osd":9,"uuid":"2989ca68-3164-4667-9bcf-230dcbb7b8f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6827","nonce":3423789152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6829","nonce":3423789152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6832","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6833","nonce":3423789152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6831","nonce":3423789152}]},"public_addr":"172.21.15.72:6827/3423789152","cluster_addr":"172.21.15.72:6829/3423789152","heartbeat_back_addr":"172.21.15.72:6833/3423789152","heartbeat_front_addr":"172.21.15.72:6831/3423789152","state":["exists","up"]},{"osd":10,"uuid":"9ec6d46a-0913-4208-97b5-167188ee8cb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6825","nonce":1120991574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6827","nonce":1120991574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6831","nonce":1120991574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6829","nonce":1120991574}]},"public_addr":"172.21.15.76:6825/1120991574","cluster_addr":"172.21.15.76:6827/1120991574","heartbeat_back_addr":"172.21.15.76:6831/1120991574","heartbeat_front_addr":"172.21.15.76:6829/1120991574","state":["exists","up"]},{"osd":11,"uuid":"bdd0774f-6e58-4918-b4cc-afd4f1118d3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6825","nonce":1135995193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6827","nonce":1135995193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6831","nonce":1135995193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6829","nonce":1135995193}]},"public_addr":"172.21.15.136:6825/1135995193","cluster_addr":"172.21.15.136:6827/1135995193","heartbeat_back_addr":"172.21.15.136:6831/1135995193","heartbeat_front_addr":"172.21.15.136:6829/1135995193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:36:52.128788+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:37:17.523654+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:37:41.634677+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:38:15.060797+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:38:41.234111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:39:06.783560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:39:40.884386+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:40:08.340465+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:40:34.344858+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:41:08.525309+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:41:36.782624+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:42:02.756888+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:0/3625071330":"2024-04-27T23:34:58.266218+0000","172.21.15.72:6801/2903979134":"2024-04-27T23:34:58.266218+0000","172.21.15.72:6800/2903979134":"2024-04-27T23:34:58.266218+0000","172.21.15.72:0/1519142576":"2024-04-27T23:34:58.266218+0000","172.21.15.72:0/3669626460":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/945623297":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/3479150073":"2024-04-27T23:34:24.233319+0000","172.21.15.72:6800/4175896290":"2024-04-27T23:34:24.233319+0000","172.21.15.72:6801/4175896290":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/3783175012":"2024-04-27T23:34:58.266218+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:43:23.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.397+0000 7f042ae00640 1 -- 172.21.15.72:0/3016248482 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f04080a3bb0 msgr2=0x7f04080a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:23.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.397+0000 7f042ae00640 1 --2- 172.21.15.72:0/3016248482 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f04080a3bb0 0x7f04080a6070 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f042419c280 tx=0x7f0414005f90 comp rx=0 tx=0).stop 2024-04-26T23:43:23.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.397+0000 7f042ae00640 1 -- 172.21.15.72:0/3016248482 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0424104520 msgr2=0x7f04241a2320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:23.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.397+0000 7f042ae00640 1 --2- 172.21.15.72:0/3016248482 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0424104520 0x7f04241a2320 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f041c009fd0 tx=0x7f041c00ef10 comp rx=0 tx=0).stop 2024-04-26T23:43:23.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.397+0000 7f042ae00640 1 -- 172.21.15.72:0/3016248482 shutdown_connections 2024-04-26T23:43:23.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.397+0000 7f042ae00640 1 --2- 172.21.15.72:0/3016248482 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f04080a3bb0 0x7f04080a6070 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:23.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.397+0000 7f042ae00640 1 --2- 172.21.15.72:0/3016248482 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0424104520 0x7f04241a2320 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:23.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.397+0000 7f042ae00640 1 --2- 172.21.15.72:0/3016248482 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0424103b60 0x7f042419b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:23.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.397+0000 7f042ae00640 1 --2- 172.21.15.72:0/3016248482 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0424102970 0x7f042419adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:23.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.397+0000 7f042ae00640 1 -- 172.21.15.72:0/3016248482 >> 172.21.15.72:0/3016248482 conn(0x7f04240fdee0 msgr2=0x7f0424074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:23.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.397+0000 7f042ae00640 1 -- 172.21.15.72:0/3016248482 shutdown_connections 2024-04-26T23:43:23.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.397+0000 7f042ae00640 1 -- 172.21.15.72:0/3016248482 wait complete. 2024-04-26T23:43:23.421 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:37:44.428727+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T23:43:00.431829+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T23:43:01.872977+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T23:43:05.512050+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-26T23:43:23.421 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:43:23.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.624+0000 7fc670800640 1 -- 172.21.15.72:0/3772445611 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc668104b90 msgr2=0x7fc668104fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:23.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.624+0000 7fc670800640 1 --2- 172.21.15.72:0/3772445611 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc668104b90 0x7fc668104fb0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc65c009900 tx=0x7fc65c02a410 comp rx=0 tx=0).stop 2024-04-26T23:43:23.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.624+0000 7fc670800640 1 -- 172.21.15.72:0/3772445611 shutdown_connections 2024-04-26T23:43:23.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.624+0000 7fc670800640 1 --2- 172.21.15.72:0/3772445611 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc668105d80 0x7fc668106200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:23.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.624+0000 7fc670800640 1 --2- 172.21.15.72:0/3772445611 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc668106740 0x7fc66810cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:23.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.624+0000 7fc670800640 1 --2- 172.21.15.72:0/3772445611 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc668104b90 0x7fc668104fb0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:23.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.624+0000 7fc670800640 1 -- 172.21.15.72:0/3772445611 >> 172.21.15.72:0/3772445611 conn(0x7fc668100120 msgr2=0x7fc668102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:23.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.624+0000 7fc670800640 1 -- 172.21.15.72:0/3772445611 shutdown_connections 2024-04-26T23:43:23.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.624+0000 7fc670800640 1 -- 172.21.15.72:0/3772445611 wait complete. 2024-04-26T23:43:23.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.626+0000 7fc670800640 1 Processor -- start 2024-04-26T23:43:23.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.626+0000 7fc670800640 1 -- start start 2024-04-26T23:43:23.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.626+0000 7fc670800640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc668104b90 0x7fc66819ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:23.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.626+0000 7fc670800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc668105d80 0x7fc66819b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:23.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.626+0000 7fc670800640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc668106740 0x7fc6681a2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:23.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.626+0000 7fc670800640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fc668110690 con 0x7fc668104b90 2024-04-26T23:43:23.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.626+0000 7fc670800640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc668110510 con 0x7fc668105d80 2024-04-26T23:43:23.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.626+0000 7fc670800640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fc668110810 con 0x7fc668106740 2024-04-26T23:43:23.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.626+0000 7fc66d400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc668105d80 0x7fc66819b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:23.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.626+0000 7fc66d400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc668105d80 0x7fc66819b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:52436/0 (socket says 172.21.15.72:52436) 2024-04-26T23:43:23.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.626+0000 7fc66d400640 1 -- 172.21.15.72:0/3976459702 learned_addr learned my addr 172.21.15.72:0/3976459702 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:43:23.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.626+0000 7fc66de00640 1 --2- 172.21.15.72:0/3976459702 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc668104b90 0x7fc66819ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:23.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.627+0000 7fc66e800640 1 --2- 172.21.15.72:0/3976459702 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc668106740 0x7fc6681a2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:23.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.627+0000 7fc66d400640 1 -- 172.21.15.72:0/3976459702 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc668106740 msgr2=0x7fc6681a2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:23.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.627+0000 7fc66d400640 1 --2- 172.21.15.72:0/3976459702 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc668106740 0x7fc6681a2390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:23.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.627+0000 7fc66d400640 1 -- 172.21.15.72:0/3976459702 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc668104b90 msgr2=0x7fc66819ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:23.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.627+0000 7fc66d400640 1 --2- 172.21.15.72:0/3976459702 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc668104b90 0x7fc66819ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:23.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.627+0000 7fc66d400640 1 -- 172.21.15.72:0/3976459702 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6681a2a00 con 0x7fc668105d80 2024-04-26T23:43:23.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.627+0000 7fc66d400640 1 --2- 172.21.15.72:0/3976459702 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc668105d80 0x7fc66819b320 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc65800ea60 tx=0x7fc65800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:23.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.627+0000 7fc662a00640 1 -- 172.21.15.72:0/3976459702 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc65800cde0 con 0x7fc668105d80 2024-04-26T23:43:23.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.628+0000 7fc662a00640 1 -- 172.21.15.72:0/3976459702 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6580035f0 con 0x7fc668105d80 2024-04-26T23:43:23.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.628+0000 7fc662a00640 1 -- 172.21.15.72:0/3976459702 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc658010690 con 0x7fc668105d80 2024-04-26T23:43:23.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.628+0000 7fc670800640 1 -- 172.21.15.72:0/3976459702 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6681a2cf0 con 0x7fc668105d80 2024-04-26T23:43:23.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.628+0000 7fc670800640 1 -- 172.21.15.72:0/3976459702 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6681a3230 con 0x7fc668105d80 2024-04-26T23:43:23.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.629+0000 7fc670800640 1 -- 172.21.15.72:0/3976459702 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc66810c010 con 0x7fc668105d80 2024-04-26T23:43:23.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.631+0000 7fc662a00640 1 -- 172.21.15.72:0/3976459702 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fc6580037a0 con 0x7fc668105d80 2024-04-26T23:43:23.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.632+0000 7fc662a00640 1 --2- 172.21.15.72:0/3976459702 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fc6500a3c30 0x7fc6500a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:23.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.632+0000 7fc662a00640 1 -- 172.21.15.72:0/3976459702 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fc6580d08e0 con 0x7fc668105d80 2024-04-26T23:43:23.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.632+0000 7fc66de00640 1 --2- 172.21.15.72:0/3976459702 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fc6500a3c30 0x7fc6500a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:23.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.633+0000 7fc66de00640 1 --2- 172.21.15.72:0/3976459702 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fc6500a3c30 0x7fc6500a60f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc65c009230 tx=0x7fc65c0314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:23.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.636+0000 7fc662a00640 1 -- 172.21.15.72:0/3976459702 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc65809d020 con 0x7fc668105d80 2024-04-26T23:43:23.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:23 smithi072 ceph-mon[28184]: pgmap v331: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-04-26T23:43:23.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:23 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/541624469' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T23:43:23.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:23 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/3016248482' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:43:23.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.839+0000 7fc670800640 1 -- 172.21.15.72:0/3976459702 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc6680635d0 con 0x7fc668105d80 2024-04-26T23:43:23.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.840+0000 7fc662a00640 1 -- 172.21.15.72:0/3976459702 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7fc65809c920 con 0x7fc668105d80 2024-04-26T23:43:23.844 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-04-26T23:43:23.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.843+0000 7fc670800640 1 -- 172.21.15.72:0/3976459702 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fc6500a3c30 msgr2=0x7fc6500a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:23.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.843+0000 7fc670800640 1 --2- 172.21.15.72:0/3976459702 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fc6500a3c30 0x7fc6500a60f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc65c009230 tx=0x7fc65c0314a0 comp rx=0 tx=0).stop 2024-04-26T23:43:23.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.843+0000 7fc670800640 1 -- 172.21.15.72:0/3976459702 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc668105d80 msgr2=0x7fc66819b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:23.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.843+0000 7fc670800640 1 --2- 172.21.15.72:0/3976459702 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc668105d80 0x7fc66819b320 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc65800ea60 tx=0x7fc65800ef20 comp rx=0 tx=0).stop 2024-04-26T23:43:23.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.843+0000 7fc670800640 1 -- 172.21.15.72:0/3976459702 shutdown_connections 2024-04-26T23:43:23.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.843+0000 7fc670800640 1 --2- 172.21.15.72:0/3976459702 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fc6500a3c30 0x7fc6500a60f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:23.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.843+0000 7fc670800640 1 --2- 172.21.15.72:0/3976459702 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc668105d80 0x7fc66819b320 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:23.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.843+0000 7fc670800640 1 --2- 172.21.15.72:0/3976459702 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc668106740 0x7fc6681a2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:23.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.843+0000 7fc670800640 1 --2- 172.21.15.72:0/3976459702 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc668104b90 0x7fc66819ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:23.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.843+0000 7fc670800640 1 -- 172.21.15.72:0/3976459702 >> 172.21.15.72:0/3976459702 conn(0x7fc668100120 msgr2=0x7fc6681091e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:23.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.843+0000 7fc670800640 1 -- 172.21.15.72:0/3976459702 shutdown_connections 2024-04-26T23:43:23.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:23.843+0000 7fc670800640 1 -- 172.21.15.72:0/3976459702 wait complete. 2024-04-26T23:43:23.866 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T23:43:23.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:23 smithi076 ceph-mon[32774]: pgmap v331: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-04-26T23:43:23.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:23 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/541624469' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T23:43:23.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:23 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/3016248482' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:43:24.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:23 smithi136 ceph-mon[33050]: pgmap v331: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-04-26T23:43:24.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:23 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/541624469' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T23:43:24.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:23 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/3016248482' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:43:24.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.064+0000 7f1f01200640 1 -- 172.21.15.72:0/4286688095 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1efc10a180 msgr2=0x7f1efc10a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:24.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.064+0000 7f1f01200640 1 --2- 172.21.15.72:0/4286688095 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1efc10a180 0x7f1efc10a600 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1ee4009900 tx=0x7f1ee402a400 comp rx=0 tx=0).stop 2024-04-26T23:43:24.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.064+0000 7f1f01200640 1 -- 172.21.15.72:0/4286688095 shutdown_connections 2024-04-26T23:43:24.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.064+0000 7f1f01200640 1 --2- 172.21.15.72:0/4286688095 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1efc10ab40 0x7f1efc1113d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:24.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.064+0000 7f1f01200640 1 --2- 172.21.15.72:0/4286688095 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1efc10a180 0x7f1efc10a600 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:24.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.064+0000 7f1f01200640 1 --2- 172.21.15.72:0/4286688095 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1efc108f90 0x7f1efc1093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:24.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.064+0000 7f1f01200640 1 -- 172.21.15.72:0/4286688095 >> 172.21.15.72:0/4286688095 conn(0x7f1efc077f30 msgr2=0x7f1efc07a370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:24.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.064+0000 7f1f01200640 1 -- 172.21.15.72:0/4286688095 shutdown_connections 2024-04-26T23:43:24.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.064+0000 7f1f01200640 1 -- 172.21.15.72:0/4286688095 wait complete. 2024-04-26T23:43:24.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.065+0000 7f1f01200640 1 Processor -- start 2024-04-26T23:43:24.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.065+0000 7f1f01200640 1 -- start start 2024-04-26T23:43:24.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.065+0000 7f1f01200640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1efc108f90 0x7f1efc19f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:24.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.065+0000 7f1f01200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1efc10a180 0x7f1efc19f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:24.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.065+0000 7f1f01200640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1efc10ab40 0x7f1efc1a6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:24.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.065+0000 7f1f01200640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f1efc114b20 con 0x7f1efc108f90 2024-04-26T23:43:24.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.065+0000 7f1f01200640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f1efc1149a0 con 0x7f1efc10a180 2024-04-26T23:43:24.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.065+0000 7f1f01200640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f1efc114ca0 con 0x7f1efc10ab40 2024-04-26T23:43:24.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.066+0000 7f1ef9e00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1efc108f90 0x7f1efc19f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:24.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.066+0000 7f1ef9e00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1efc108f90 0x7f1efc19f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56806/0 (socket says 172.21.15.72:56806) 2024-04-26T23:43:24.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.066+0000 7f1ef9e00640 1 -- 172.21.15.72:0/1347712555 learned_addr learned my addr 172.21.15.72:0/1347712555 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:43:24.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.066+0000 7f1efa800640 1 --2- 172.21.15.72:0/1347712555 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1efc10ab40 0x7f1efc1a6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:24.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.066+0000 7f1ef9400640 1 --2- 172.21.15.72:0/1347712555 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1efc10a180 0x7f1efc19f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:24.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.066+0000 7f1ef9e00640 1 -- 172.21.15.72:0/1347712555 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1efc10ab40 msgr2=0x7f1efc1a6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:24.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.066+0000 7f1ef9e00640 1 --2- 172.21.15.72:0/1347712555 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1efc10ab40 0x7f1efc1a6750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:24.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.066+0000 7f1ef9e00640 1 -- 172.21.15.72:0/1347712555 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1efc10a180 msgr2=0x7f1efc19f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:24.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.066+0000 7f1ef9e00640 1 --2- 172.21.15.72:0/1347712555 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1efc10a180 0x7f1efc19f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:24.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.066+0000 7f1ef9e00640 1 -- 172.21.15.72:0/1347712555 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1efc1a6dc0 con 0x7f1efc108f90 2024-04-26T23:43:24.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.066+0000 7f1ef9e00640 1 --2- 172.21.15.72:0/1347712555 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1efc108f90 0x7f1efc19f1a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1eec00e9c0 tx=0x7f1eec00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:24.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.066+0000 7f1ef2a00640 1 -- 172.21.15.72:0/1347712555 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eec00cd80 con 0x7f1efc108f90 2024-04-26T23:43:24.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.067+0000 7f1ef2a00640 1 -- 172.21.15.72:0/1347712555 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1eec0035f0 con 0x7f1efc108f90 2024-04-26T23:43:24.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.067+0000 7f1ef2a00640 1 -- 172.21.15.72:0/1347712555 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eec010690 con 0x7f1efc108f90 2024-04-26T23:43:24.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.067+0000 7f1f01200640 1 -- 172.21.15.72:0/1347712555 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1efc1a70b0 con 0x7f1efc108f90 2024-04-26T23:43:24.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.067+0000 7f1f01200640 1 -- 172.21.15.72:0/1347712555 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1efc1a7620 con 0x7f1efc108f90 2024-04-26T23:43:24.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.068+0000 7f1f01200640 1 -- 172.21.15.72:0/1347712555 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1efc110330 con 0x7f1efc108f90 2024-04-26T23:43:24.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.070+0000 7f1ef2a00640 1 -- 172.21.15.72:0/1347712555 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f1eec0037a0 con 0x7f1efc108f90 2024-04-26T23:43:24.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.071+0000 7f1ef2a00640 1 --2- 172.21.15.72:0/1347712555 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f1ee00a3c30 0x7f1ee00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:24.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.071+0000 7f1ef2a00640 1 -- 172.21.15.72:0/1347712555 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f1eec0d0a90 con 0x7f1efc108f90 2024-04-26T23:43:24.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.072+0000 7f1ef9400640 1 --2- 172.21.15.72:0/1347712555 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f1ee00a3c30 0x7f1ee00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:24.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.072+0000 7f1ef9400640 1 --2- 172.21.15.72:0/1347712555 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f1ee00a3c30 0x7f1ee00a60f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1ee4031500 tx=0x7f1ee4005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:24.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.074+0000 7f1ef2a00640 1 -- 172.21.15.72:0/1347712555 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1eec09d1d0 con 0x7f1efc108f90 2024-04-26T23:43:24.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.275+0000 7f1f01200640 1 -- 172.21.15.72:0/1347712555 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1efc063530 con 0x7f1efc108f90 2024-04-26T23:43:24.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.276+0000 7f1ef2a00640 1 -- 172.21.15.72:0/1347712555 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7f1eec09cad0 con 0x7f1efc108f90 2024-04-26T23:43:24.280 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-26T23:43:24.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.278+0000 7f1f01200640 1 -- 172.21.15.72:0/1347712555 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f1ee00a3c30 msgr2=0x7f1ee00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:24.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.278+0000 7f1f01200640 1 --2- 172.21.15.72:0/1347712555 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f1ee00a3c30 0x7f1ee00a60f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1ee4031500 tx=0x7f1ee4005a20 comp rx=0 tx=0).stop 2024-04-26T23:43:24.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.278+0000 7f1f01200640 1 -- 172.21.15.72:0/1347712555 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1efc108f90 msgr2=0x7f1efc19f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:24.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.279+0000 7f1f01200640 1 --2- 172.21.15.72:0/1347712555 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1efc108f90 0x7f1efc19f1a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1eec00e9c0 tx=0x7f1eec00ee80 comp rx=0 tx=0).stop 2024-04-26T23:43:24.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.279+0000 7f1f01200640 1 -- 172.21.15.72:0/1347712555 shutdown_connections 2024-04-26T23:43:24.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.279+0000 7f1f01200640 1 --2- 172.21.15.72:0/1347712555 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f1ee00a3c30 0x7f1ee00a60f0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:24.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.279+0000 7f1f01200640 1 --2- 172.21.15.72:0/1347712555 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1efc10a180 0x7f1efc19f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:24.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.279+0000 7f1f01200640 1 --2- 172.21.15.72:0/1347712555 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1efc10ab40 0x7f1efc1a6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:24.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.279+0000 7f1f01200640 1 --2- 172.21.15.72:0/1347712555 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1efc108f90 0x7f1efc19f1a0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:24.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.279+0000 7f1f01200640 1 -- 172.21.15.72:0/1347712555 >> 172.21.15.72:0/1347712555 conn(0x7f1efc077f30 msgr2=0x7f1efc10d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:24.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.279+0000 7f1f01200640 1 -- 172.21.15.72:0/1347712555 shutdown_connections 2024-04-26T23:43:24.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.279+0000 7f1f01200640 1 -- 172.21.15.72:0/1347712555 wait complete. 2024-04-26T23:43:24.303 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T23:43:24.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.504+0000 7f6d28a00640 1 -- 172.21.15.72:0/2626065160 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6d24078530 msgr2=0x7f6d24078990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:24.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.504+0000 7f6d28a00640 1 --2- 172.21.15.72:0/2626065160 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6d24078530 0x7f6d24078990 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c009900 tx=0x7f6d0c02a410 comp rx=0 tx=0).stop 2024-04-26T23:43:24.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.505+0000 7f6d28a00640 1 -- 172.21.15.72:0/2626065160 shutdown_connections 2024-04-26T23:43:24.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.505+0000 7f6d28a00640 1 --2- 172.21.15.72:0/2626065160 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6d24078530 0x7f6d24078990 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:24.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.505+0000 7f6d28a00640 1 --2- 172.21.15.72:0/2626065160 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d24078ed0 0x7f6d24079770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:24.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.505+0000 7f6d28a00640 1 --2- 172.21.15.72:0/2626065160 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6d240772f0 0x7f6d24077710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:24.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.505+0000 7f6d28a00640 1 -- 172.21.15.72:0/2626065160 >> 172.21.15.72:0/2626065160 conn(0x7f6d24100120 msgr2=0x7f6d24102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:24.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.505+0000 7f6d28a00640 1 -- 172.21.15.72:0/2626065160 shutdown_connections 2024-04-26T23:43:24.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.505+0000 7f6d28a00640 1 -- 172.21.15.72:0/2626065160 wait complete. 2024-04-26T23:43:24.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.506+0000 7f6d28a00640 1 Processor -- start 2024-04-26T23:43:24.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.506+0000 7f6d28a00640 1 -- start start 2024-04-26T23:43:24.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.506+0000 7f6d28a00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6d240772f0 0x7f6d2419f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:24.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.506+0000 7f6d28a00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d24078530 0x7f6d2419f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:24.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.506+0000 7f6d28a00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6d24078ed0 0x7f6d241a6710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:24.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.506+0000 7f6d28a00640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f6d24114ad0 con 0x7f6d24078530 2024-04-26T23:43:24.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.506+0000 7f6d28a00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f6d24114950 con 0x7f6d24078ed0 2024-04-26T23:43:24.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.506+0000 7f6d28a00640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f6d24114c50 con 0x7f6d240772f0 2024-04-26T23:43:24.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.507+0000 7f6d1be00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d24078530 0x7f6d2419f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:24.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.507+0000 7f6d22800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6d24078ed0 0x7f6d241a6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:24.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.507+0000 7f6d1be00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d24078530 0x7f6d2419f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56820/0 (socket says 172.21.15.72:56820) 2024-04-26T23:43:24.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.507+0000 7f6d1be00640 1 -- 172.21.15.72:0/2276623695 learned_addr learned my addr 172.21.15.72:0/2276623695 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:43:24.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.507+0000 7f6d21e00640 1 --2- 172.21.15.72:0/2276623695 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6d240772f0 0x7f6d2419f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:24.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.507+0000 7f6d22800640 1 -- 172.21.15.72:0/2276623695 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6d240772f0 msgr2=0x7f6d2419f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:24.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.507+0000 7f6d22800640 1 --2- 172.21.15.72:0/2276623695 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6d240772f0 0x7f6d2419f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:24.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.507+0000 7f6d22800640 1 -- 172.21.15.72:0/2276623695 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d24078530 msgr2=0x7f6d2419f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:24.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.507+0000 7f6d22800640 1 --2- 172.21.15.72:0/2276623695 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d24078530 0x7f6d2419f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:24.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.507+0000 7f6d22800640 1 -- 172.21.15.72:0/2276623695 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d241a6d80 con 0x7f6d24078ed0 2024-04-26T23:43:24.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.507+0000 7f6d22800640 1 --2- 172.21.15.72:0/2276623695 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6d24078ed0 0x7f6d241a6710 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c009fd0 tx=0x7f6d1c00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:24.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.508+0000 7f6d1aa00640 1 -- 172.21.15.72:0/2276623695 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d1c00ce20 con 0x7f6d24078ed0 2024-04-26T23:43:24.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.508+0000 7f6d1aa00640 1 -- 172.21.15.72:0/2276623695 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6d1c005020 con 0x7f6d24078ed0 2024-04-26T23:43:24.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.508+0000 7f6d28a00640 1 -- 172.21.15.72:0/2276623695 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d241a7070 con 0x7f6d24078ed0 2024-04-26T23:43:24.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.508+0000 7f6d1aa00640 1 -- 172.21.15.72:0/2276623695 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d1c010690 con 0x7f6d24078ed0 2024-04-26T23:43:24.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.508+0000 7f6d28a00640 1 -- 172.21.15.72:0/2276623695 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d241a7590 con 0x7f6d24078ed0 2024-04-26T23:43:24.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.509+0000 7f6d28a00640 1 -- 172.21.15.72:0/2276623695 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d24111680 con 0x7f6d24078ed0 2024-04-26T23:43:24.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.511+0000 7f6d1aa00640 1 -- 172.21.15.72:0/2276623695 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6d1c0054d0 con 0x7f6d24078ed0 2024-04-26T23:43:24.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.512+0000 7f6d1aa00640 1 --2- 172.21.15.72:0/2276623695 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6d080a3c30 0x7f6d080a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:24.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.512+0000 7f6d1aa00640 1 -- 172.21.15.72:0/2276623695 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f6d1c0d0d10 con 0x7f6d24078ed0 2024-04-26T23:43:24.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.513+0000 7f6d21e00640 1 --2- 172.21.15.72:0/2276623695 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6d080a3c30 0x7f6d080a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:24.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.513+0000 7f6d21e00640 1 --2- 172.21.15.72:0/2276623695 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6d080a3c30 0x7f6d080a60f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6d14009bd0 tx=0x7f6d14005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:24.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.517+0000 7f6d1aa00640 1 -- 172.21.15.72:0/2276623695 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6d1c0d6020 con 0x7f6d24078ed0 2024-04-26T23:43:24.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.721+0000 7f6d28a00640 1 -- 172.21.15.72:0/2276623695 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6d24063530 con 0x7f6d24078ed0 2024-04-26T23:43:24.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.721+0000 7f6d1aa00640 1 -- 172.21.15.72:0/2276623695 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7f6d1c09d450 con 0x7f6d24078ed0 2024-04-26T23:43:24.726 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-04-26T23:43:24.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.724+0000 7f6d28a00640 1 -- 172.21.15.72:0/2276623695 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6d080a3c30 msgr2=0x7f6d080a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:24.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.724+0000 7f6d28a00640 1 --2- 172.21.15.72:0/2276623695 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6d080a3c30 0x7f6d080a60f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6d14009bd0 tx=0x7f6d14005f90 comp rx=0 tx=0).stop 2024-04-26T23:43:24.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.724+0000 7f6d28a00640 1 -- 172.21.15.72:0/2276623695 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6d24078ed0 msgr2=0x7f6d241a6710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:24.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.724+0000 7f6d28a00640 1 --2- 172.21.15.72:0/2276623695 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6d24078ed0 0x7f6d241a6710 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c009fd0 tx=0x7f6d1c00ef10 comp rx=0 tx=0).stop 2024-04-26T23:43:24.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.724+0000 7f6d28a00640 1 -- 172.21.15.72:0/2276623695 shutdown_connections 2024-04-26T23:43:24.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.724+0000 7f6d28a00640 1 --2- 172.21.15.72:0/2276623695 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6d080a3c30 0x7f6d080a60f0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:24.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.724+0000 7f6d28a00640 1 --2- 172.21.15.72:0/2276623695 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6d24078ed0 0x7f6d241a6710 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:24.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.724+0000 7f6d28a00640 1 --2- 172.21.15.72:0/2276623695 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d24078530 0x7f6d2419f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:24.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.724+0000 7f6d28a00640 1 --2- 172.21.15.72:0/2276623695 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6d240772f0 0x7f6d2419f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:24.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.724+0000 7f6d28a00640 1 -- 172.21.15.72:0/2276623695 >> 172.21.15.72:0/2276623695 conn(0x7f6d24100120 msgr2=0x7f6d24101860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:24.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.724+0000 7f6d28a00640 1 -- 172.21.15.72:0/2276623695 shutdown_connections 2024-04-26T23:43:24.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.724+0000 7f6d28a00640 1 -- 172.21.15.72:0/2276623695 wait complete. 2024-04-26T23:43:24.747 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-26T23:43:24.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:24 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/3976459702' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T23:43:24.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:24 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/1347712555' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T23:43:24.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.952+0000 7f2393200640 1 -- 172.21.15.72:0/2463165944 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f238c10a980 msgr2=0x7f238c1112a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:24.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.952+0000 7f2393200640 1 --2- 172.21.15.72:0/2463165944 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f238c10a980 0x7f238c1112a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2388009900 tx=0x7f238802a3f0 comp rx=0 tx=0).stop 2024-04-26T23:43:24.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.952+0000 7f2393200640 1 -- 172.21.15.72:0/2463165944 shutdown_connections 2024-04-26T23:43:24.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.952+0000 7f2393200640 1 --2- 172.21.15.72:0/2463165944 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f238c10a980 0x7f238c1112a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:24.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.952+0000 7f2393200640 1 --2- 172.21.15.72:0/2463165944 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f238c109fc0 0x7f238c10a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:24.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.952+0000 7f2393200640 1 --2- 172.21.15.72:0/2463165944 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f238c108e70 0x7f238c109290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:24.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.952+0000 7f2393200640 1 -- 172.21.15.72:0/2463165944 >> 172.21.15.72:0/2463165944 conn(0x7f238c077e00 msgr2=0x7f238c07a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:24.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.953+0000 7f2393200640 1 -- 172.21.15.72:0/2463165944 shutdown_connections 2024-04-26T23:43:24.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.953+0000 7f2393200640 1 -- 172.21.15.72:0/2463165944 wait complete. 2024-04-26T23:43:24.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.953+0000 7f2393200640 1 Processor -- start 2024-04-26T23:43:24.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.954+0000 7f2393200640 1 -- start start 2024-04-26T23:43:24.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.954+0000 7f2393200640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f238c108e70 0x7f238c19efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:24.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.954+0000 7f2393200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f238c109fc0 0x7f238c19f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:24.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.954+0000 7f2393200640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f238c10a980 0x7f238c1a6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:24.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.954+0000 7f2393200640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f238c1149f0 con 0x7f238c10a980 2024-04-26T23:43:24.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.954+0000 7f2393200640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f238c114870 con 0x7f238c109fc0 2024-04-26T23:43:24.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.954+0000 7f2393200640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f238c114b70 con 0x7f238c108e70 2024-04-26T23:43:24.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.954+0000 7f2391400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f238c109fc0 0x7f238c19f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:24.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.954+0000 7f2391400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f238c109fc0 0x7f238c19f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:52502/0 (socket says 172.21.15.72:52502) 2024-04-26T23:43:24.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.954+0000 7f2391400640 1 -- 172.21.15.72:0/1166614606 learned_addr learned my addr 172.21.15.72:0/1166614606 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:43:24.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.954+0000 7f2392800640 1 --2- 172.21.15.72:0/1166614606 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f238c10a980 0x7f238c1a6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:24.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.954+0000 7f2391e00640 1 --2- 172.21.15.72:0/1166614606 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f238c108e70 0x7f238c19efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:24.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.955+0000 7f2391400640 1 -- 172.21.15.72:0/1166614606 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f238c108e70 msgr2=0x7f238c19efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:24.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.955+0000 7f2391400640 1 --2- 172.21.15.72:0/1166614606 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f238c108e70 0x7f238c19efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:24.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.955+0000 7f2391400640 1 -- 172.21.15.72:0/1166614606 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f238c10a980 msgr2=0x7f238c1a6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:24.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.955+0000 7f2391400640 1 --2- 172.21.15.72:0/1166614606 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f238c10a980 0x7f238c1a6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:24.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.955+0000 7f2391400640 1 -- 172.21.15.72:0/1166614606 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f238c1a6bd0 con 0x7f238c109fc0 2024-04-26T23:43:24.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.955+0000 7f2392800640 1 --2- 172.21.15.72:0/1166614606 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f238c10a980 0x7f238c1a6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:43:24.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.955+0000 7f2391400640 1 --2- 172.21.15.72:0/1166614606 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f238c109fc0 0x7f238c19f4f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f237c00e9c0 tx=0x7f237c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:24.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.955+0000 7f2382a00640 1 -- 172.21.15.72:0/1166614606 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f237c00cd80 con 0x7f238c109fc0 2024-04-26T23:43:24.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.955+0000 7f2393200640 1 -- 172.21.15.72:0/1166614606 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f238c1a6ec0 con 0x7f238c109fc0 2024-04-26T23:43:24.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.955+0000 7f2382a00640 1 -- 172.21.15.72:0/1166614606 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f237c0035f0 con 0x7f238c109fc0 2024-04-26T23:43:24.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.955+0000 7f2382a00640 1 -- 172.21.15.72:0/1166614606 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f237c010690 con 0x7f238c109fc0 2024-04-26T23:43:24.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.955+0000 7f2393200640 1 -- 172.21.15.72:0/1166614606 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f238c1a7380 con 0x7f238c109fc0 2024-04-26T23:43:24.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.957+0000 7f2393200640 1 -- 172.21.15.72:0/1166614606 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f238c1100b0 con 0x7f238c109fc0 2024-04-26T23:43:24.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.958+0000 7f2382a00640 1 -- 172.21.15.72:0/1166614606 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f237c0037a0 con 0x7f238c109fc0 2024-04-26T23:43:24.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.960+0000 7f2382a00640 1 --2- 172.21.15.72:0/1166614606 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f23780a3bb0 0x7f23780a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:24.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.961+0000 7f2382a00640 1 -- 172.21.15.72:0/1166614606 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f237c0d0a90 con 0x7f238c109fc0 2024-04-26T23:43:24.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.961+0000 7f2391e00640 1 --2- 172.21.15.72:0/1166614606 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f23780a3bb0 0x7f23780a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:24.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.961+0000 7f2391e00640 1 --2- 172.21.15.72:0/1166614606 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f23780a3bb0 0x7f23780a6070 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2384005fd0 tx=0x7f2384009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:24.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:24.964+0000 7f2382a00640 1 -- 172.21.15.72:0/1166614606 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f237c09d1d0 con 0x7f238c109fc0 2024-04-26T23:43:25.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:24 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/3976459702' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T23:43:25.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:24 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/1347712555' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T23:43:25.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:24 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/3976459702' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T23:43:25.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:24 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/1347712555' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T23:43:25.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.166+0000 7f2393200640 1 -- 172.21.15.72:0/1166614606 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f238c063530 con 0x7f238c109fc0 2024-04-26T23:43:25.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.167+0000 7f2382a00640 1 -- 172.21.15.72:0/1166614606 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) ==== 103+0+11 (secure 0 0 0) 0x7f237c09cad0 con 0x7f238c109fc0 2024-04-26T23:43:25.172 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-26T23:43:25.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.170+0000 7f2393200640 1 -- 172.21.15.72:0/1166614606 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f23780a3bb0 msgr2=0x7f23780a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:25.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.170+0000 7f2393200640 1 --2- 172.21.15.72:0/1166614606 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f23780a3bb0 0x7f23780a6070 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2384005fd0 tx=0x7f2384009500 comp rx=0 tx=0).stop 2024-04-26T23:43:25.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.170+0000 7f2393200640 1 -- 172.21.15.72:0/1166614606 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f238c109fc0 msgr2=0x7f238c19f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:25.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.170+0000 7f2393200640 1 --2- 172.21.15.72:0/1166614606 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f238c109fc0 0x7f238c19f4f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f237c00e9c0 tx=0x7f237c00ee80 comp rx=0 tx=0).stop 2024-04-26T23:43:25.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.170+0000 7f2393200640 1 -- 172.21.15.72:0/1166614606 shutdown_connections 2024-04-26T23:43:25.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.170+0000 7f2393200640 1 --2- 172.21.15.72:0/1166614606 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f23780a3bb0 0x7f23780a6070 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:25.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.170+0000 7f2393200640 1 --2- 172.21.15.72:0/1166614606 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f238c109fc0 0x7f238c19f4f0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:25.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.170+0000 7f2393200640 1 --2- 172.21.15.72:0/1166614606 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f238c10a980 0x7f238c1a6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:25.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.170+0000 7f2393200640 1 --2- 172.21.15.72:0/1166614606 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f238c108e70 0x7f238c19efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:25.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.170+0000 7f2393200640 1 -- 172.21.15.72:0/1166614606 >> 172.21.15.72:0/1166614606 conn(0x7f238c077e00 msgr2=0x7f238c10d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:25.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.170+0000 7f2393200640 1 -- 172.21.15.72:0/1166614606 shutdown_connections 2024-04-26T23:43:25.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.170+0000 7f2393200640 1 -- 172.21.15.72:0/1166614606 wait complete. 2024-04-26T23:43:25.191 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-26T23:43:25.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.386+0000 7f2f33000640 1 -- 172.21.15.72:0/2846704543 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2f2c1086f0 msgr2=0x7f2f2c10aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:25.395 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.386+0000 7f2f33000640 1 --2- 172.21.15.72:0/2846704543 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2f2c1086f0 0x7f2f2c10aae0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2f24009900 tx=0x7f2f2402a410 comp rx=0 tx=0).stop 2024-04-26T23:43:25.395 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.386+0000 7f2f33000640 1 -- 172.21.15.72:0/2846704543 shutdown_connections 2024-04-26T23:43:25.395 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.386+0000 7f2f33000640 1 --2- 172.21.15.72:0/2846704543 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2f2c1086f0 0x7f2f2c10aae0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:25.395 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.386+0000 7f2f33000640 1 --2- 172.21.15.72:0/2846704543 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2f2c105dc0 0x7f2f2c1081b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:25.395 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.386+0000 7f2f33000640 1 --2- 172.21.15.72:0/2846704543 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f2c103490 0x7f2f2c105880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:25.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.386+0000 7f2f33000640 1 -- 172.21.15.72:0/2846704543 >> 172.21.15.72:0/2846704543 conn(0x7f2f2c0fcde0 msgr2=0x7f2f2c0ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:25.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.386+0000 7f2f33000640 1 -- 172.21.15.72:0/2846704543 shutdown_connections 2024-04-26T23:43:25.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.386+0000 7f2f33000640 1 -- 172.21.15.72:0/2846704543 wait complete. 2024-04-26T23:43:25.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.387+0000 7f2f33000640 1 Processor -- start 2024-04-26T23:43:25.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.387+0000 7f2f33000640 1 -- start start 2024-04-26T23:43:25.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.387+0000 7f2f33000640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f2c103490 0x7f2f2c19adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:25.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.387+0000 7f2f33000640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2f2c105dc0 0x7f2f2c19b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:25.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.387+0000 7f2f33000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2f2c1086f0 0x7f2f2c1a2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:25.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.387+0000 7f2f33000640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f2f2c10e510 con 0x7f2f2c103490 2024-04-26T23:43:25.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.387+0000 7f2f33000640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2f2c10e390 con 0x7f2f2c1086f0 2024-04-26T23:43:25.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.387+0000 7f2f33000640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f2f2c10e690 con 0x7f2f2c105dc0 2024-04-26T23:43:25.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.388+0000 7f2f2be00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f2c103490 0x7f2f2c19adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:25.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.388+0000 7f2f2b400640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2f2c105dc0 0x7f2f2c19b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:25.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.388+0000 7f2f31000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2f2c1086f0 0x7f2f2c1a2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:25.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.388+0000 7f2f2b400640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2f2c105dc0 0x7f2f2c19b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.72:57734/0 (socket says 172.21.15.72:57734) 2024-04-26T23:43:25.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.388+0000 7f2f2b400640 1 -- 172.21.15.72:0/3976123095 learned_addr learned my addr 172.21.15.72:0/3976123095 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:43:25.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.388+0000 7f2f2b400640 1 -- 172.21.15.72:0/3976123095 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2f2c1086f0 msgr2=0x7f2f2c1a2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:25.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.388+0000 7f2f2b400640 1 --2- 172.21.15.72:0/3976123095 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2f2c1086f0 0x7f2f2c1a2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:25.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.388+0000 7f2f2b400640 1 -- 172.21.15.72:0/3976123095 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f2c103490 msgr2=0x7f2f2c19adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:25.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.388+0000 7f2f2b400640 1 --2- 172.21.15.72:0/3976123095 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f2c103490 0x7f2f2c19adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:25.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.388+0000 7f2f2b400640 1 -- 172.21.15.72:0/3976123095 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f2c1a29d0 con 0x7f2f2c105dc0 2024-04-26T23:43:25.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.388+0000 7f2f2b400640 1 --2- 172.21.15.72:0/3976123095 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2f2c105dc0 0x7f2f2c19b300 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c00ebc0 tx=0x7f2f1c00c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:25.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.389+0000 7f2f28c00640 1 -- 172.21.15.72:0/3976123095 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f1c004d20 con 0x7f2f2c105dc0 2024-04-26T23:43:25.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.389+0000 7f2f28c00640 1 -- 172.21.15.72:0/3976123095 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2f1c0054d0 con 0x7f2f2c105dc0 2024-04-26T23:43:25.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.389+0000 7f2f33000640 1 -- 172.21.15.72:0/3976123095 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f2c1a2cc0 con 0x7f2f2c105dc0 2024-04-26T23:43:25.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.389+0000 7f2f28c00640 1 -- 172.21.15.72:0/3976123095 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f1c010720 con 0x7f2f2c105dc0 2024-04-26T23:43:25.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.389+0000 7f2f33000640 1 -- 172.21.15.72:0/3976123095 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f2c1a31c0 con 0x7f2f2c105dc0 2024-04-26T23:43:25.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.390+0000 7f2f33000640 1 -- 172.21.15.72:0/3976123095 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f2c10a1e0 con 0x7f2f2c105dc0 2024-04-26T23:43:25.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.392+0000 7f2f28c00640 1 -- 172.21.15.72:0/3976123095 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2f1c005020 con 0x7f2f2c105dc0 2024-04-26T23:43:25.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.393+0000 7f2f28c00640 1 --2- 172.21.15.72:0/3976123095 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f2f100a3bb0 0x7f2f100a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:25.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.394+0000 7f2f28c00640 1 -- 172.21.15.72:0/3976123095 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f2f1c0d1420 con 0x7f2f2c105dc0 2024-04-26T23:43:25.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.394+0000 7f2f2be00640 1 --2- 172.21.15.72:0/3976123095 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f2f100a3bb0 0x7f2f100a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:25.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.394+0000 7f2f2be00640 1 --2- 172.21.15.72:0/3976123095 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f2f100a3bb0 0x7f2f100a6070 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c19c290 tx=0x7f2f14005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:25.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.396+0000 7f2f28c00640 1 -- 172.21.15.72:0/3976123095 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f2f1c09db60 con 0x7f2f2c105dc0 2024-04-26T23:43:25.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.614+0000 7f2f33000640 1 -- 172.21.15.72:0/3976123095 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f2f2c02dd20 con 0x7f2f2c105dc0 2024-04-26T23:43:25.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.615+0000 7f2f28c00640 1 -- 172.21.15.72:0/3976123095 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+591 (secure 0 0 0) 0x7f2f1c09d460 con 0x7f2f2c105dc0 2024-04-26T23:43:25.619 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:0/3625071330 2024-04-27T23:34:58.266218+0000 2024-04-26T23:43:25.619 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:6801/2903979134 2024-04-27T23:34:58.266218+0000 2024-04-26T23:43:25.619 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:6800/2903979134 2024-04-27T23:34:58.266218+0000 2024-04-26T23:43:25.619 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:0/1519142576 2024-04-27T23:34:58.266218+0000 2024-04-26T23:43:25.619 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:0/3669626460 2024-04-27T23:34:24.233319+0000 2024-04-26T23:43:25.619 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:0/945623297 2024-04-27T23:34:24.233319+0000 2024-04-26T23:43:25.619 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:0/3479150073 2024-04-27T23:34:24.233319+0000 2024-04-26T23:43:25.620 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:6800/4175896290 2024-04-27T23:34:24.233319+0000 2024-04-26T23:43:25.620 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:6801/4175896290 2024-04-27T23:34:24.233319+0000 2024-04-26T23:43:25.620 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:0/3783175012 2024-04-27T23:34:58.266218+0000 2024-04-26T23:43:25.620 INFO:teuthology.orchestra.run.smithi072.stderr:listed 10 entries 2024-04-26T23:43:25.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.618+0000 7f2f33000640 1 -- 172.21.15.72:0/3976123095 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f2f100a3bb0 msgr2=0x7f2f100a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:25.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.618+0000 7f2f33000640 1 --2- 172.21.15.72:0/3976123095 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f2f100a3bb0 0x7f2f100a6070 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c19c290 tx=0x7f2f14005f90 comp rx=0 tx=0).stop 2024-04-26T23:43:25.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.618+0000 7f2f33000640 1 -- 172.21.15.72:0/3976123095 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2f2c105dc0 msgr2=0x7f2f2c19b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:25.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.618+0000 7f2f33000640 1 --2- 172.21.15.72:0/3976123095 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2f2c105dc0 0x7f2f2c19b300 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c00ebc0 tx=0x7f2f1c00c6a0 comp rx=0 tx=0).stop 2024-04-26T23:43:25.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.618+0000 7f2f33000640 1 -- 172.21.15.72:0/3976123095 shutdown_connections 2024-04-26T23:43:25.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.618+0000 7f2f33000640 1 --2- 172.21.15.72:0/3976123095 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f2f100a3bb0 0x7f2f100a6070 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:25.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.618+0000 7f2f33000640 1 --2- 172.21.15.72:0/3976123095 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2f2c1086f0 0x7f2f2c1a2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:25.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.618+0000 7f2f33000640 1 --2- 172.21.15.72:0/3976123095 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2f2c105dc0 0x7f2f2c19b300 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:25.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.618+0000 7f2f33000640 1 --2- 172.21.15.72:0/3976123095 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f2c103490 0x7f2f2c19adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:25.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.618+0000 7f2f33000640 1 -- 172.21.15.72:0/3976123095 >> 172.21.15.72:0/3976123095 conn(0x7f2f2c0fcde0 msgr2=0x7f2f2c102bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:25.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.618+0000 7f2f33000640 1 -- 172.21.15.72:0/3976123095 shutdown_connections 2024-04-26T23:43:25.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:25.618+0000 7f2f33000640 1 -- 172.21.15.72:0/3976123095 wait complete. 2024-04-26T23:43:25.642 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi072.front.sepia.ceph.com... 2024-04-26T23:43:25.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:43:25.642 DEBUG:teuthology.orchestra.run.smithi072:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-04-26T23:43:25.719 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi072.stderr:ceph-fuse[70221]: fuse finished with error 0 and tester_r 0 2024-04-26T23:43:25.723 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T23:43:25.797 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi072.front.sepia.ceph.com 2024-04-26T23:43:25.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:43:25.797 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-04-26T23:43:25.861 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:43:25.928 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:25 smithi072 ceph-mon[28184]: pgmap v332: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T23:43:25.928 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:25 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/2276623695' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T23:43:25.928 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:25 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/1166614606' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T23:43:25.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:25 smithi076 ceph-mon[32774]: pgmap v332: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T23:43:25.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:25 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/2276623695' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T23:43:25.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:25 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/1166614606' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T23:43:26.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:25 smithi136 ceph-mon[33050]: pgmap v332: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T23:43:26.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:25 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/2276623695' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T23:43:26.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:25 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/1166614606' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T23:43:26.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.057+0000 7f8d93000640 1 -- 172.21.15.72:0/906218615 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d8c10a940 msgr2=0x7f8d8c1111d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:26.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.057+0000 7f8d93000640 1 --2- 172.21.15.72:0/906218615 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d8c10a940 0x7f8d8c1111d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8d84009ca0 tx=0x7f8d8402a8c0 comp rx=0 tx=0).stop 2024-04-26T23:43:26.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.057+0000 7f8d93000640 1 -- 172.21.15.72:0/906218615 shutdown_connections 2024-04-26T23:43:26.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.057+0000 7f8d93000640 1 --2- 172.21.15.72:0/906218615 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d8c10a940 0x7f8d8c1111d0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:26.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.057+0000 7f8d93000640 1 --2- 172.21.15.72:0/906218615 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8d8c109f80 0x7f8d8c10a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:26.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.057+0000 7f8d93000640 1 --2- 172.21.15.72:0/906218615 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8d8c108d90 0x7f8d8c1091b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:26.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.057+0000 7f8d93000640 1 -- 172.21.15.72:0/906218615 >> 172.21.15.72:0/906218615 conn(0x7f8d8c077d30 msgr2=0x7f8d8c07a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:26.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.057+0000 7f8d93000640 1 -- 172.21.15.72:0/906218615 shutdown_connections 2024-04-26T23:43:26.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.057+0000 7f8d93000640 1 -- 172.21.15.72:0/906218615 wait complete. 2024-04-26T23:43:26.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.058+0000 7f8d93000640 1 Processor -- start 2024-04-26T23:43:26.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.058+0000 7f8d93000640 1 -- start start 2024-04-26T23:43:26.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.059+0000 7f8d93000640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d8c108d90 0x7f8d8c19ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:26.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.059+0000 7f8d93000640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8d8c109f80 0x7f8d8c19f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:26.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.059+0000 7f8d93000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8d8c10a940 0x7f8d8c1a64f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:26.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.059+0000 7f8d93000640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f8d8c1148c0 con 0x7f8d8c108d90 2024-04-26T23:43:26.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.059+0000 7f8d93000640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8d8c114740 con 0x7f8d8c10a940 2024-04-26T23:43:26.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.059+0000 7f8d93000640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f8d8c114a40 con 0x7f8d8c109f80 2024-04-26T23:43:26.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.059+0000 7f8d91c00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d8c108d90 0x7f8d8c19ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:26.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.059+0000 7f8d91c00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d8c108d90 0x7f8d8c19ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56852/0 (socket says 172.21.15.72:56852) 2024-04-26T23:43:26.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.059+0000 7f8d91c00640 1 -- 172.21.15.72:0/2457993815 learned_addr learned my addr 172.21.15.72:0/2457993815 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:43:26.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.059+0000 7f8d92600640 1 --2- 172.21.15.72:0/2457993815 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8d8c10a940 0x7f8d8c1a64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:26.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.059+0000 7f8d91200640 1 --2- 172.21.15.72:0/2457993815 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8d8c109f80 0x7f8d8c19f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:26.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.060+0000 7f8d91c00640 1 -- 172.21.15.72:0/2457993815 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8d8c109f80 msgr2=0x7f8d8c19f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:26.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.060+0000 7f8d91c00640 1 --2- 172.21.15.72:0/2457993815 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8d8c109f80 0x7f8d8c19f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:26.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.060+0000 7f8d91c00640 1 -- 172.21.15.72:0/2457993815 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8d8c10a940 msgr2=0x7f8d8c1a64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:26.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.060+0000 7f8d91c00640 1 --2- 172.21.15.72:0/2457993815 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8d8c10a940 0x7f8d8c1a64f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:26.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.060+0000 7f8d91c00640 1 -- 172.21.15.72:0/2457993815 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d8c1a6b60 con 0x7f8d8c108d90 2024-04-26T23:43:26.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.060+0000 7f8d92600640 1 --2- 172.21.15.72:0/2457993815 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8d8c10a940 0x7f8d8c1a64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:43:26.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.060+0000 7f8d91c00640 1 --2- 172.21.15.72:0/2457993815 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d8c108d90 0x7f8d8c19ef40 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8d8000e9c0 tx=0x7f8d8000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:26.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.060+0000 7f8d8a000640 1 -- 172.21.15.72:0/2457993815 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d8000cd80 con 0x7f8d8c108d90 2024-04-26T23:43:26.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.060+0000 7f8d8a000640 1 -- 172.21.15.72:0/2457993815 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d800035f0 con 0x7f8d8c108d90 2024-04-26T23:43:26.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.060+0000 7f8d93000640 1 -- 172.21.15.72:0/2457993815 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d8c1a6e50 con 0x7f8d8c108d90 2024-04-26T23:43:26.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.060+0000 7f8d8a000640 1 -- 172.21.15.72:0/2457993815 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d80010690 con 0x7f8d8c108d90 2024-04-26T23:43:26.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.060+0000 7f8d93000640 1 -- 172.21.15.72:0/2457993815 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d8c1a7310 con 0x7f8d8c108d90 2024-04-26T23:43:26.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.061+0000 7f8d93000640 1 -- 172.21.15.72:0/2457993815 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d8c1100b0 con 0x7f8d8c108d90 2024-04-26T23:43:26.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.064+0000 7f8d8a000640 1 -- 172.21.15.72:0/2457993815 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8d800037a0 con 0x7f8d8c108d90 2024-04-26T23:43:26.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.065+0000 7f8d8a000640 1 --2- 172.21.15.72:0/2457993815 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f8d740a3c30 0x7f8d740a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:26.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.065+0000 7f8d8a000640 1 -- 172.21.15.72:0/2457993815 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f8d800d0980 con 0x7f8d8c108d90 2024-04-26T23:43:26.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.065+0000 7f8d91200640 1 --2- 172.21.15.72:0/2457993815 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f8d740a3c30 0x7f8d740a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:26.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.066+0000 7f8d91200640 1 --2- 172.21.15.72:0/2457993815 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f8d740a3c30 0x7f8d740a60f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c1a0450 tx=0x7f8d78005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:26.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.069+0000 7f8d8a000640 1 -- 172.21.15.72:0/2457993815 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8d8009d0c0 con 0x7f8d8c108d90 2024-04-26T23:43:26.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.273+0000 7f8d93000640 1 -- 172.21.15.72:0/2457993815 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8d8c0635d0 con 0x7f8d8c108d90 2024-04-26T23:43:26.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.277+0000 7f8d8a000640 1 -- 172.21.15.72:0/2457993815 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24081 (secure 0 0 0) 0x7f8d8009c9c0 con 0x7f8d8c108d90 2024-04-26T23:43:26.282 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:43:26.282 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"fsid":"3c8453ae-0425-11ef-bc93-c7b262605968","created":"2024-04-26T23:33:48.088300+0000","modified":"2024-04-26T23:43:07.990780+0000","last_up_change":"2024-04-26T23:42:04.482891+0000","last_in_change":"2024-04-26T23:41:44.784415+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:37:44.428727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:43:00.431829+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:43:01.872977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:43:05.512050+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e20c9ebf-56f6-47c1-a11b-e1e615af042b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6803","nonce":1639298875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6805","nonce":1639298875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6809","nonce":1639298875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6807","nonce":1639298875}]},"public_addr":"172.21.15.72:6803/1639298875","cluster_addr":"172.21.15.72:6805/1639298875","heartbeat_back_addr":"172.21.15.72:6809/1639298875","heartbeat_front_addr":"172.21.15.72:6807/1639298875","state":["exists","up"]},{"osd":1,"uuid":"e6cac469-780c-4d66-b560-b686072371fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6801","nonce":4185342193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6803","nonce":4185342193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6807","nonce":4185342193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6805","nonce":4185342193}]},"public_addr":"172.21.15.76:6801/4185342193","cluster_addr":"172.21.15.76:6803/4185342193","heartbeat_back_addr":"172.21.15.76:6807/4185342193","heartbeat_front_addr":"172.21.15.76:6805/4185342193","state":["exists","up"]},{"osd":2,"uuid":"838a49fc-18de-446c-8336-67b3e05da231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6801","nonce":3033860799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6803","nonce":3033860799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6807","nonce":3033860799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6805","nonce":3033860799}]},"public_addr":"172.21.15.136:6801/3033860799","cluster_addr":"172.21.15.136:6803/3033860799","heartbeat_back_addr":"172.21.15.136:6807/3033860799","heartbeat_front_addr":"172.21.15.136:6805/3033860799","state":["exists","up"]},{"osd":3,"uuid":"e6da89be-5ba7-48a4-9306-f75498660e0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6811","nonce":2071107815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6813","nonce":2071107815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6817","nonce":2071107815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6815","nonce":2071107815}]},"public_addr":"172.21.15.72:6811/2071107815","cluster_addr":"172.21.15.72:6813/2071107815","heartbeat_back_addr":"172.21.15.72:6817/2071107815","heartbeat_front_addr":"172.21.15.72:6815/2071107815","state":["exists","up"]},{"osd":4,"uuid":"0b17c491-d798-488f-892d-521ef2fbc51e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6809","nonce":147449238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6811","nonce":147449238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6815","nonce":147449238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6813","nonce":147449238}]},"public_addr":"172.21.15.76:6809/147449238","cluster_addr":"172.21.15.76:6811/147449238","heartbeat_back_addr":"172.21.15.76:6815/147449238","heartbeat_front_addr":"172.21.15.76:6813/147449238","state":["exists","up"]},{"osd":5,"uuid":"c7e887b3-ab38-4608-9bfd-1ed49956e8af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6809","nonce":2320149212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6811","nonce":2320149212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6815","nonce":2320149212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6813","nonce":2320149212}]},"public_addr":"172.21.15.136:6809/2320149212","cluster_addr":"172.21.15.136:6811/2320149212","heartbeat_back_addr":"172.21.15.136:6815/2320149212","heartbeat_front_addr":"172.21.15.136:6813/2320149212","state":["exists","up"]},{"osd":6,"uuid":"f0bd604b-0f57-4a38-9295-eb200264f1ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6819","nonce":389499207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6821","nonce":389499207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6825","nonce":389499207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6823","nonce":389499207}]},"public_addr":"172.21.15.72:6819/389499207","cluster_addr":"172.21.15.72:6821/389499207","heartbeat_back_addr":"172.21.15.72:6825/389499207","heartbeat_front_addr":"172.21.15.72:6823/389499207","state":["exists","up"]},{"osd":7,"uuid":"1879d4e4-038e-4cd6-a3ec-3a940bcc3797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6817","nonce":375888575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6819","nonce":375888575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6823","nonce":375888575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6821","nonce":375888575}]},"public_addr":"172.21.15.76:6817/375888575","cluster_addr":"172.21.15.76:6819/375888575","heartbeat_back_addr":"172.21.15.76:6823/375888575","heartbeat_front_addr":"172.21.15.76:6821/375888575","state":["exists","up"]},{"osd":8,"uuid":"6f404062-97e7-4ccb-a582-76dc1ffae548","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6817","nonce":3446774052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6819","nonce":3446774052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6823","nonce":3446774052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6821","nonce":3446774052}]},"public_addr":"172.21.15.136:6817/3446774052","cluster_addr":"172.21.15.136:6819/3446774052","heartbeat_back_addr":"172.21.15.136:6823/3446774052","heartbeat_front_addr":"172.21.15.136:6821/3446774052","state":["exists","up"]},{"osd":9,"uuid":"2989ca68-3164-4667-9bcf-230dcbb7b8f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6827","nonce":3423789152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6829","nonce":3423789152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6832","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6833","nonce":3423789152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6831","nonce":3423789152}]},"public_addr":"172.21.15.72:6827/3423789152","cluster_addr":"172.21.15.72:6829/3423789152","heartbeat_back_addr":"172.21.15.72:6833/3423789152","heartbeat_front_addr":"172.21.15.72:6831/3423789152","state":["exists","up"]},{"osd":10,"uuid":"9ec6d46a-0913-4208-97b5-167188ee8cb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6825","nonce":1120991574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6827","nonce":1120991574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6831","nonce":1120991574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6829","nonce":1120991574}]},"public_addr":"172.21.15.76:6825/1120991574","cluster_addr":"172.21.15.76:6827/1120991574","heartbeat_back_addr":"172.21.15.76:6831/1120991574","heartbeat_front_addr":"172.21.15.76:6829/1120991574","state":["exists","up"]},{"osd":11,"uuid":"bdd0774f-6e58-4918-b4cc-afd4f1118d3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6825","nonce":1135995193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6827","nonce":1135995193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6831","nonce":1135995193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6829","nonce":1135995193}]},"public_addr":"172.21.15.136:6825/1135995193","cluster_addr":"172.21.15.136:6827/1135995193","heartbeat_back_addr":"172.21.15.136:6831/1135995193","heartbeat_front_addr":"172.21.15.136:6829/1135995193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:36:52.128788+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:37:17.523654+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:37:41.634677+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:38:15.060797+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:38:41.234111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:39:06.783560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:39:40.884386+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:40:08.340465+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:40:34.344858+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:41:08.525309+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:41:36.782624+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:42:02.756888+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:0/3625071330":"2024-04-27T23:34:58.266218+0000","172.21.15.72:6801/2903979134":"2024-04-27T23:34:58.266218+0000","172.21.15.72:6800/2903979134":"2024-04-27T23:34:58.266218+0000","172.21.15.72:0/1519142576":"2024-04-27T23:34:58.266218+0000","172.21.15.72:0/3669626460":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/945623297":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/3479150073":"2024-04-27T23:34:24.233319+0000","172.21.15.72:6800/4175896290":"2024-04-27T23:34:24.233319+0000","172.21.15.72:6801/4175896290":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/3783175012":"2024-04-27T23:34:58.266218+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:43:26.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.280+0000 7f8d93000640 1 -- 172.21.15.72:0/2457993815 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f8d740a3c30 msgr2=0x7f8d740a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:26.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.280+0000 7f8d93000640 1 --2- 172.21.15.72:0/2457993815 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f8d740a3c30 0x7f8d740a60f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c1a0450 tx=0x7f8d78005f90 comp rx=0 tx=0).stop 2024-04-26T23:43:26.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.280+0000 7f8d93000640 1 -- 172.21.15.72:0/2457993815 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d8c108d90 msgr2=0x7f8d8c19ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:26.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.280+0000 7f8d93000640 1 --2- 172.21.15.72:0/2457993815 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d8c108d90 0x7f8d8c19ef40 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8d8000e9c0 tx=0x7f8d8000ee80 comp rx=0 tx=0).stop 2024-04-26T23:43:26.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.280+0000 7f8d93000640 1 -- 172.21.15.72:0/2457993815 shutdown_connections 2024-04-26T23:43:26.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.280+0000 7f8d93000640 1 --2- 172.21.15.72:0/2457993815 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f8d740a3c30 0x7f8d740a60f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:26.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.280+0000 7f8d93000640 1 --2- 172.21.15.72:0/2457993815 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8d8c10a940 0x7f8d8c1a64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:26.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.280+0000 7f8d93000640 1 --2- 172.21.15.72:0/2457993815 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8d8c109f80 0x7f8d8c19f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:26.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.280+0000 7f8d93000640 1 --2- 172.21.15.72:0/2457993815 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d8c108d90 0x7f8d8c19ef40 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:26.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.280+0000 7f8d93000640 1 -- 172.21.15.72:0/2457993815 >> 172.21.15.72:0/2457993815 conn(0x7f8d8c077d30 msgr2=0x7f8d8c10d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:26.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.280+0000 7f8d93000640 1 -- 172.21.15.72:0/2457993815 shutdown_connections 2024-04-26T23:43:26.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.280+0000 7f8d93000640 1 -- 172.21.15.72:0/2457993815 wait complete. 2024-04-26T23:43:26.304 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:43:26.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.508+0000 7fe57ea00640 1 -- 172.21.15.72:0/2758318646 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe578109f80 msgr2=0x7fe57810a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:26.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.508+0000 7fe57ea00640 1 --2- 172.21.15.72:0/2758318646 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe578109f80 0x7fe57810a400 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe564009860 tx=0x7fe56402a340 comp rx=0 tx=0).stop 2024-04-26T23:43:26.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.508+0000 7fe57ea00640 1 -- 172.21.15.72:0/2758318646 shutdown_connections 2024-04-26T23:43:26.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.508+0000 7fe57ea00640 1 --2- 172.21.15.72:0/2758318646 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe57810a940 0x7fe5781111d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:26.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.508+0000 7fe57ea00640 1 --2- 172.21.15.72:0/2758318646 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe578109f80 0x7fe57810a400 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:26.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.508+0000 7fe57ea00640 1 --2- 172.21.15.72:0/2758318646 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe578108d90 0x7fe5781091b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:26.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.508+0000 7fe57ea00640 1 -- 172.21.15.72:0/2758318646 >> 172.21.15.72:0/2758318646 conn(0x7fe578077d30 msgr2=0x7fe57807a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:26.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.508+0000 7fe57ea00640 1 -- 172.21.15.72:0/2758318646 shutdown_connections 2024-04-26T23:43:26.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.508+0000 7fe57ea00640 1 -- 172.21.15.72:0/2758318646 wait complete. 2024-04-26T23:43:26.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.509+0000 7fe57ea00640 1 Processor -- start 2024-04-26T23:43:26.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.509+0000 7fe57ea00640 1 -- start start 2024-04-26T23:43:26.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.510+0000 7fe57ea00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe578108d90 0x7fe57819ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:26.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.510+0000 7fe57ea00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe578109f80 0x7fe57819f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:26.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.510+0000 7fe57ea00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe57810a940 0x7fe5781a6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:26.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.510+0000 7fe57ea00640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fe5781148c0 con 0x7fe578108d90 2024-04-26T23:43:26.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.510+0000 7fe57ea00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe578114740 con 0x7fe578109f80 2024-04-26T23:43:26.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.510+0000 7fe57ea00640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fe578114a40 con 0x7fe57810a940 2024-04-26T23:43:26.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.510+0000 7fe57d600640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe578108d90 0x7fe57819ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:26.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.510+0000 7fe57d600640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe578108d90 0x7fe57819ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56860/0 (socket says 172.21.15.72:56860) 2024-04-26T23:43:26.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.510+0000 7fe57d600640 1 -- 172.21.15.72:0/3637334506 learned_addr learned my addr 172.21.15.72:0/3637334506 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:43:26.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.510+0000 7fe57cc00640 1 --2- 172.21.15.72:0/3637334506 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe578109f80 0x7fe57819f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:26.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.510+0000 7fe57e000640 1 --2- 172.21.15.72:0/3637334506 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe57810a940 0x7fe5781a6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:26.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.510+0000 7fe57d600640 1 -- 172.21.15.72:0/3637334506 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe57810a940 msgr2=0x7fe5781a6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:26.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.510+0000 7fe57d600640 1 --2- 172.21.15.72:0/3637334506 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe57810a940 0x7fe5781a6530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:26.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.510+0000 7fe57d600640 1 -- 172.21.15.72:0/3637334506 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe578109f80 msgr2=0x7fe57819f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:26.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.511+0000 7fe57d600640 1 --2- 172.21.15.72:0/3637334506 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe578109f80 0x7fe57819f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:26.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.511+0000 7fe57d600640 1 -- 172.21.15.72:0/3637334506 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5781a6ba0 con 0x7fe578108d90 2024-04-26T23:43:26.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.511+0000 7fe57d600640 1 --2- 172.21.15.72:0/3637334506 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe578108d90 0x7fe57819ef80 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe56c00e9f0 tx=0x7fe56c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:26.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.511+0000 7fe576000640 1 -- 172.21.15.72:0/3637334506 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe56c00ccc0 con 0x7fe578108d90 2024-04-26T23:43:26.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.511+0000 7fe576000640 1 -- 172.21.15.72:0/3637334506 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe56c0035f0 con 0x7fe578108d90 2024-04-26T23:43:26.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.511+0000 7fe57ea00640 1 -- 172.21.15.72:0/3637334506 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5781a6e90 con 0x7fe578108d90 2024-04-26T23:43:26.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.511+0000 7fe576000640 1 -- 172.21.15.72:0/3637334506 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe56c00ce60 con 0x7fe578108d90 2024-04-26T23:43:26.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.511+0000 7fe57ea00640 1 -- 172.21.15.72:0/3637334506 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5781a72f0 con 0x7fe578108d90 2024-04-26T23:43:26.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.512+0000 7fe57ea00640 1 -- 172.21.15.72:0/3637334506 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe578110180 con 0x7fe578108d90 2024-04-26T23:43:26.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.516+0000 7fe576000640 1 -- 172.21.15.72:0/3637334506 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe56c004000 con 0x7fe578108d90 2024-04-26T23:43:26.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.517+0000 7fe576000640 1 --2- 172.21.15.72:0/3637334506 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fe5600a3bb0 0x7fe5600a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:26.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.518+0000 7fe576000640 1 -- 172.21.15.72:0/3637334506 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fe56c0d0950 con 0x7fe578108d90 2024-04-26T23:43:26.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.518+0000 7fe57cc00640 1 --2- 172.21.15.72:0/3637334506 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fe5600a3bb0 0x7fe5600a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:26.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.518+0000 7fe57cc00640 1 --2- 172.21.15.72:0/3637334506 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fe5600a3bb0 0x7fe5600a6070 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe56402a930 tx=0x7fe564030450 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:26.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.520+0000 7fe576000640 1 -- 172.21.15.72:0/3637334506 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe56c09d090 con 0x7fe578108d90 2024-04-26T23:43:26.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.726+0000 7fe57ea00640 1 -- 172.21.15.72:0/3637334506 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe57802dd80 con 0x7fe578108d90 2024-04-26T23:43:26.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.729+0000 7fe576000640 1 -- 172.21.15.72:0/3637334506 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24081 (secure 0 0 0) 0x7fe56c09c990 con 0x7fe578108d90 2024-04-26T23:43:26.734 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:43:26.734 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"fsid":"3c8453ae-0425-11ef-bc93-c7b262605968","created":"2024-04-26T23:33:48.088300+0000","modified":"2024-04-26T23:43:07.990780+0000","last_up_change":"2024-04-26T23:42:04.482891+0000","last_in_change":"2024-04-26T23:41:44.784415+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:37:44.428727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:43:00.431829+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:43:01.872977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:43:05.512050+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e20c9ebf-56f6-47c1-a11b-e1e615af042b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6803","nonce":1639298875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6805","nonce":1639298875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6809","nonce":1639298875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6807","nonce":1639298875}]},"public_addr":"172.21.15.72:6803/1639298875","cluster_addr":"172.21.15.72:6805/1639298875","heartbeat_back_addr":"172.21.15.72:6809/1639298875","heartbeat_front_addr":"172.21.15.72:6807/1639298875","state":["exists","up"]},{"osd":1,"uuid":"e6cac469-780c-4d66-b560-b686072371fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6801","nonce":4185342193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6803","nonce":4185342193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6807","nonce":4185342193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6805","nonce":4185342193}]},"public_addr":"172.21.15.76:6801/4185342193","cluster_addr":"172.21.15.76:6803/4185342193","heartbeat_back_addr":"172.21.15.76:6807/4185342193","heartbeat_front_addr":"172.21.15.76:6805/4185342193","state":["exists","up"]},{"osd":2,"uuid":"838a49fc-18de-446c-8336-67b3e05da231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6801","nonce":3033860799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6803","nonce":3033860799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6807","nonce":3033860799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6805","nonce":3033860799}]},"public_addr":"172.21.15.136:6801/3033860799","cluster_addr":"172.21.15.136:6803/3033860799","heartbeat_back_addr":"172.21.15.136:6807/3033860799","heartbeat_front_addr":"172.21.15.136:6805/3033860799","state":["exists","up"]},{"osd":3,"uuid":"e6da89be-5ba7-48a4-9306-f75498660e0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6811","nonce":2071107815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6813","nonce":2071107815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6817","nonce":2071107815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6815","nonce":2071107815}]},"public_addr":"172.21.15.72:6811/2071107815","cluster_addr":"172.21.15.72:6813/2071107815","heartbeat_back_addr":"172.21.15.72:6817/2071107815","heartbeat_front_addr":"172.21.15.72:6815/2071107815","state":["exists","up"]},{"osd":4,"uuid":"0b17c491-d798-488f-892d-521ef2fbc51e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6809","nonce":147449238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6811","nonce":147449238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6815","nonce":147449238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6813","nonce":147449238}]},"public_addr":"172.21.15.76:6809/147449238","cluster_addr":"172.21.15.76:6811/147449238","heartbeat_back_addr":"172.21.15.76:6815/147449238","heartbeat_front_addr":"172.21.15.76:6813/147449238","state":["exists","up"]},{"osd":5,"uuid":"c7e887b3-ab38-4608-9bfd-1ed49956e8af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6809","nonce":2320149212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6811","nonce":2320149212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6815","nonce":2320149212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6813","nonce":2320149212}]},"public_addr":"172.21.15.136:6809/2320149212","cluster_addr":"172.21.15.136:6811/2320149212","heartbeat_back_addr":"172.21.15.136:6815/2320149212","heartbeat_front_addr":"172.21.15.136:6813/2320149212","state":["exists","up"]},{"osd":6,"uuid":"f0bd604b-0f57-4a38-9295-eb200264f1ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6819","nonce":389499207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6821","nonce":389499207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6825","nonce":389499207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6823","nonce":389499207}]},"public_addr":"172.21.15.72:6819/389499207","cluster_addr":"172.21.15.72:6821/389499207","heartbeat_back_addr":"172.21.15.72:6825/389499207","heartbeat_front_addr":"172.21.15.72:6823/389499207","state":["exists","up"]},{"osd":7,"uuid":"1879d4e4-038e-4cd6-a3ec-3a940bcc3797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6817","nonce":375888575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6819","nonce":375888575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6823","nonce":375888575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6821","nonce":375888575}]},"public_addr":"172.21.15.76:6817/375888575","cluster_addr":"172.21.15.76:6819/375888575","heartbeat_back_addr":"172.21.15.76:6823/375888575","heartbeat_front_addr":"172.21.15.76:6821/375888575","state":["exists","up"]},{"osd":8,"uuid":"6f404062-97e7-4ccb-a582-76dc1ffae548","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6817","nonce":3446774052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6819","nonce":3446774052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6823","nonce":3446774052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6821","nonce":3446774052}]},"public_addr":"172.21.15.136:6817/3446774052","cluster_addr":"172.21.15.136:6819/3446774052","heartbeat_back_addr":"172.21.15.136:6823/3446774052","heartbeat_front_addr":"172.21.15.136:6821/3446774052","state":["exists","up"]},{"osd":9,"uuid":"2989ca68-3164-4667-9bcf-230dcbb7b8f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6827","nonce":3423789152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6829","nonce":3423789152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6832","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6833","nonce":3423789152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6831","nonce":3423789152}]},"public_addr":"172.21.15.72:6827/3423789152","cluster_addr":"172.21.15.72:6829/3423789152","heartbeat_back_addr":"172.21.15.72:6833/3423789152","heartbeat_front_addr":"172.21.15.72:6831/3423789152","state":["exists","up"]},{"osd":10,"uuid":"9ec6d46a-0913-4208-97b5-167188ee8cb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6825","nonce":1120991574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6827","nonce":1120991574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6831","nonce":1120991574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6829","nonce":1120991574}]},"public_addr":"172.21.15.76:6825/1120991574","cluster_addr":"172.21.15.76:6827/1120991574","heartbeat_back_addr":"172.21.15.76:6831/1120991574","heartbeat_front_addr":"172.21.15.76:6829/1120991574","state":["exists","up"]},{"osd":11,"uuid":"bdd0774f-6e58-4918-b4cc-afd4f1118d3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6825","nonce":1135995193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6827","nonce":1135995193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6831","nonce":1135995193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6829","nonce":1135995193}]},"public_addr":"172.21.15.136:6825/1135995193","cluster_addr":"172.21.15.136:6827/1135995193","heartbeat_back_addr":"172.21.15.136:6831/1135995193","heartbeat_front_addr":"172.21.15.136:6829/1135995193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:36:52.128788+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:37:17.523654+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:37:41.634677+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:38:15.060797+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:38:41.234111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:39:06.783560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:39:40.884386+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:40:08.340465+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:40:34.344858+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:41:08.525309+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:41:36.782624+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:42:02.756888+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:0/3625071330":"2024-04-27T23:34:58.266218+0000","172.21.15.72:6801/2903979134":"2024-04-27T23:34:58.266218+0000","172.21.15.72:6800/2903979134":"2024-04-27T23:34:58.266218+0000","172.21.15.72:0/1519142576":"2024-04-27T23:34:58.266218+0000","172.21.15.72:0/3669626460":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/945623297":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/3479150073":"2024-04-27T23:34:24.233319+0000","172.21.15.72:6800/4175896290":"2024-04-27T23:34:24.233319+0000","172.21.15.72:6801/4175896290":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/3783175012":"2024-04-27T23:34:58.266218+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:43:26.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.731+0000 7fe57ea00640 1 -- 172.21.15.72:0/3637334506 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fe5600a3bb0 msgr2=0x7fe5600a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:26.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.731+0000 7fe57ea00640 1 --2- 172.21.15.72:0/3637334506 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fe5600a3bb0 0x7fe5600a6070 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe56402a930 tx=0x7fe564030450 comp rx=0 tx=0).stop 2024-04-26T23:43:26.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.731+0000 7fe57ea00640 1 -- 172.21.15.72:0/3637334506 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe578108d90 msgr2=0x7fe57819ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:26.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.731+0000 7fe57ea00640 1 --2- 172.21.15.72:0/3637334506 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe578108d90 0x7fe57819ef80 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe56c00e9f0 tx=0x7fe56c00eeb0 comp rx=0 tx=0).stop 2024-04-26T23:43:26.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.732+0000 7fe57ea00640 1 -- 172.21.15.72:0/3637334506 shutdown_connections 2024-04-26T23:43:26.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.732+0000 7fe57ea00640 1 --2- 172.21.15.72:0/3637334506 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fe5600a3bb0 0x7fe5600a6070 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:26.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.732+0000 7fe57ea00640 1 --2- 172.21.15.72:0/3637334506 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe578109f80 0x7fe57819f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:26.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.732+0000 7fe57ea00640 1 --2- 172.21.15.72:0/3637334506 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe57810a940 0x7fe5781a6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:26.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.732+0000 7fe57ea00640 1 --2- 172.21.15.72:0/3637334506 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe578108d90 0x7fe57819ef80 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:26.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.732+0000 7fe57ea00640 1 -- 172.21.15.72:0/3637334506 >> 172.21.15.72:0/3637334506 conn(0x7fe578077d30 msgr2=0x7fe57810d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:26.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.732+0000 7fe57ea00640 1 -- 172.21.15.72:0/3637334506 shutdown_connections 2024-04-26T23:43:26.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.732+0000 7fe57ea00640 1 -- 172.21.15.72:0/3637334506 wait complete. 2024-04-26T23:43:26.755 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-26T23:43:26.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:26 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/3976123095' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T23:43:26.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:26 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/2457993815' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:43:26.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.960+0000 7f9469e00640 1 -- 172.21.15.72:0/3489950156 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9464104a70 msgr2=0x7f9464104e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:26.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.960+0000 7f9469e00640 1 --2- 172.21.15.72:0/3489950156 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9464104a70 0x7f9464104e90 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9454009900 tx=0x7f945402a3f0 comp rx=0 tx=0).stop 2024-04-26T23:43:26.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.960+0000 7f9469e00640 1 -- 172.21.15.72:0/3489950156 shutdown_connections 2024-04-26T23:43:26.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.960+0000 7f9469e00640 1 --2- 172.21.15.72:0/3489950156 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9464106620 0x7f946410ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:26.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.960+0000 7f9469e00640 1 --2- 172.21.15.72:0/3489950156 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9464105c60 0x7f94641060e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:26.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.960+0000 7f9469e00640 1 --2- 172.21.15.72:0/3489950156 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9464104a70 0x7f9464104e90 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:26.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.960+0000 7f9469e00640 1 -- 172.21.15.72:0/3489950156 >> 172.21.15.72:0/3489950156 conn(0x7f94640fffe0 msgr2=0x7f9464102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:26.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.960+0000 7f9469e00640 1 -- 172.21.15.72:0/3489950156 shutdown_connections 2024-04-26T23:43:26.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.960+0000 7f9469e00640 1 -- 172.21.15.72:0/3489950156 wait complete. 2024-04-26T23:43:26.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.961+0000 7f9469e00640 1 Processor -- start 2024-04-26T23:43:26.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.961+0000 7f9469e00640 1 -- start start 2024-04-26T23:43:26.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.961+0000 7f9469e00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9464104a70 0x7f946419acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:26.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.961+0000 7f9469e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9464105c60 0x7f946419b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:26.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.961+0000 7f9469e00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9464106620 0x7f94641a2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:26.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.961+0000 7f9469e00640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f94641105e0 con 0x7f9464106620 2024-04-26T23:43:26.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.961+0000 7f9469e00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f9464110460 con 0x7f9464105c60 2024-04-26T23:43:26.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.961+0000 7f9469e00640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f9464110760 con 0x7f9464104a70 2024-04-26T23:43:26.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.962+0000 7f945be00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9464105c60 0x7f946419b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:26.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.962+0000 7f945be00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9464105c60 0x7f946419b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:57756/0 (socket says 172.21.15.72:57756) 2024-04-26T23:43:26.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.962+0000 7f945be00640 1 -- 172.21.15.72:0/1722648060 learned_addr learned my addr 172.21.15.72:0/1722648060 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:43:26.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.962+0000 7f9468a00640 1 --2- 172.21.15.72:0/1722648060 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9464104a70 0x7f946419acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:26.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.962+0000 7f9469400640 1 --2- 172.21.15.72:0/1722648060 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9464106620 0x7f94641a2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:26.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.962+0000 7f945be00640 1 -- 172.21.15.72:0/1722648060 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9464104a70 msgr2=0x7f946419acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:26.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.962+0000 7f945be00640 1 --2- 172.21.15.72:0/1722648060 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9464104a70 0x7f946419acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:26.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.962+0000 7f945be00640 1 -- 172.21.15.72:0/1722648060 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9464106620 msgr2=0x7f94641a2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:26.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.962+0000 7f945be00640 1 --2- 172.21.15.72:0/1722648060 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9464106620 0x7f94641a2250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:26.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.962+0000 7f945be00640 1 -- 172.21.15.72:0/1722648060 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94641a28f0 con 0x7f9464105c60 2024-04-26T23:43:26.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.962+0000 7f9469400640 1 --2- 172.21.15.72:0/1722648060 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9464106620 0x7f94641a2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:43:26.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.962+0000 7f945be00640 1 --2- 172.21.15.72:0/1722648060 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9464105c60 0x7f946419b1f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f944c00e9f0 tx=0x7f944c00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:26.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.963+0000 7f9462000640 1 -- 172.21.15.72:0/1722648060 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f944c00cd80 con 0x7f9464105c60 2024-04-26T23:43:26.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.963+0000 7f9469e00640 1 -- 172.21.15.72:0/1722648060 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94641a2be0 con 0x7f9464105c60 2024-04-26T23:43:26.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.963+0000 7f9462000640 1 -- 172.21.15.72:0/1722648060 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f944c0054d0 con 0x7f9464105c60 2024-04-26T23:43:26.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.963+0000 7f9462000640 1 -- 172.21.15.72:0/1722648060 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f944c010690 con 0x7f9464105c60 2024-04-26T23:43:26.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.963+0000 7f9469e00640 1 -- 172.21.15.72:0/1722648060 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94641a3100 con 0x7f9464105c60 2024-04-26T23:43:26.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.964+0000 7f9469e00640 1 -- 172.21.15.72:0/1722648060 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f946410be30 con 0x7f9464105c60 2024-04-26T23:43:26.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.966+0000 7f9462000640 1 -- 172.21.15.72:0/1722648060 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f944c004e90 con 0x7f9464105c60 2024-04-26T23:43:26.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.967+0000 7f9462000640 1 --2- 172.21.15.72:0/1722648060 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f94480a3bb0 0x7f94480a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:26.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.968+0000 7f9462000640 1 -- 172.21.15.72:0/1722648060 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f944c0d0a20 con 0x7f9464105c60 2024-04-26T23:43:26.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.968+0000 7f9468a00640 1 --2- 172.21.15.72:0/1722648060 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f94480a3bb0 0x7f94480a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:26.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.968+0000 7f9468a00640 1 --2- 172.21.15.72:0/1722648060 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f94480a3bb0 0x7f94480a6070 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9454005ab0 tx=0x7f9454005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:26.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:26.970+0000 7f9462000640 1 -- 172.21.15.72:0/1722648060 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f944c09d160 con 0x7f9464105c60 2024-04-26T23:43:27.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:26 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/3976123095' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T23:43:27.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:26 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/2457993815' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:43:27.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:26 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/3976123095' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T23:43:27.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:26 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/2457993815' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:43:27.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.211+0000 7f9469e00640 1 -- 172.21.15.72:0/1722648060 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f946402dad0 con 0x7f9464105c60 2024-04-26T23:43:27.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.611+0000 7f9462000640 1 -- 172.21.15.72:0/1722648060 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7f944c09ca60 con 0x7f9464105c60 2024-04-26T23:43:27.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.614+0000 7f9469e00640 1 -- 172.21.15.72:0/1722648060 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f94480a3bb0 msgr2=0x7f94480a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:27.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.614+0000 7f9469e00640 1 --2- 172.21.15.72:0/1722648060 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f94480a3bb0 0x7f94480a6070 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9454005ab0 tx=0x7f9454005a20 comp rx=0 tx=0).stop 2024-04-26T23:43:27.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.614+0000 7f9469e00640 1 -- 172.21.15.72:0/1722648060 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9464105c60 msgr2=0x7f946419b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:27.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.614+0000 7f9469e00640 1 --2- 172.21.15.72:0/1722648060 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9464105c60 0x7f946419b1f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f944c00e9f0 tx=0x7f944c00edf0 comp rx=0 tx=0).stop 2024-04-26T23:43:27.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.614+0000 7f9469e00640 1 -- 172.21.15.72:0/1722648060 shutdown_connections 2024-04-26T23:43:27.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.614+0000 7f9469e00640 1 --2- 172.21.15.72:0/1722648060 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f94480a3bb0 0x7f94480a6070 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:27.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.614+0000 7f9469e00640 1 --2- 172.21.15.72:0/1722648060 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9464105c60 0x7f946419b1f0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:27.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.615+0000 7f9469e00640 1 --2- 172.21.15.72:0/1722648060 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9464106620 0x7f94641a2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:27.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.615+0000 7f9469e00640 1 --2- 172.21.15.72:0/1722648060 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9464104a70 0x7f946419acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:27.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.615+0000 7f9469e00640 1 -- 172.21.15.72:0/1722648060 >> 172.21.15.72:0/1722648060 conn(0x7f94640fffe0 msgr2=0x7f9464109000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:27.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.615+0000 7f9469e00640 1 -- 172.21.15.72:0/1722648060 shutdown_connections 2024-04-26T23:43:27.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.615+0000 7f9469e00640 1 -- 172.21.15.72:0/1722648060 wait complete. 2024-04-26T23:43:27.640 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2024-04-26T23:43:27.640 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-04-26T23:43:27.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.834+0000 7fe194000640 1 -- 172.21.15.72:0/3052404966 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe18c105d80 msgr2=0x7fe18c106200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:27.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.834+0000 7fe194000640 1 --2- 172.21.15.72:0/3052404966 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe18c105d80 0x7fe18c106200 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe178009900 tx=0x7fe17802a410 comp rx=0 tx=0).stop 2024-04-26T23:43:27.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.835+0000 7fe194000640 1 -- 172.21.15.72:0/3052404966 shutdown_connections 2024-04-26T23:43:27.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.835+0000 7fe194000640 1 --2- 172.21.15.72:0/3052404966 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe18c106740 0x7fe18c10cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:27.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.835+0000 7fe194000640 1 --2- 172.21.15.72:0/3052404966 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe18c105d80 0x7fe18c106200 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:27.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.835+0000 7fe194000640 1 --2- 172.21.15.72:0/3052404966 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe18c104b90 0x7fe18c104fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:27.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.835+0000 7fe194000640 1 -- 172.21.15.72:0/3052404966 >> 172.21.15.72:0/3052404966 conn(0x7fe18c100120 msgr2=0x7fe18c102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:27.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.835+0000 7fe194000640 1 -- 172.21.15.72:0/3052404966 shutdown_connections 2024-04-26T23:43:27.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.835+0000 7fe194000640 1 -- 172.21.15.72:0/3052404966 wait complete. 2024-04-26T23:43:27.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.836+0000 7fe194000640 1 Processor -- start 2024-04-26T23:43:27.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.836+0000 7fe194000640 1 -- start start 2024-04-26T23:43:27.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.836+0000 7fe194000640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe18c104b90 0x7fe18c19ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:27.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.836+0000 7fe194000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe18c105d80 0x7fe18c19b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:27.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.836+0000 7fe194000640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe18c106740 0x7fe18c1a23b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:27.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.836+0000 7fe194000640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fe18c1106c0 con 0x7fe18c106740 2024-04-26T23:43:27.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.836+0000 7fe194000640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe18c110540 con 0x7fe18c105d80 2024-04-26T23:43:27.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.836+0000 7fe194000640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fe18c110840 con 0x7fe18c104b90 2024-04-26T23:43:27.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.837+0000 7fe191600640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe18c104b90 0x7fe18c19ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:27.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.837+0000 7fe191600640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe18c104b90 0x7fe18c19ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.72:46344/0 (socket says 172.21.15.72:46344) 2024-04-26T23:43:27.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.837+0000 7fe191600640 1 -- 172.21.15.72:0/1057616159 learned_addr learned my addr 172.21.15.72:0/1057616159 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:43:27.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.837+0000 7fe192000640 1 --2- 172.21.15.72:0/1057616159 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe18c106740 0x7fe18c1a23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:27.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.837+0000 7fe190c00640 1 --2- 172.21.15.72:0/1057616159 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe18c105d80 0x7fe18c19b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:27.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.837+0000 7fe191600640 1 -- 172.21.15.72:0/1057616159 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe18c105d80 msgr2=0x7fe18c19b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:27.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.837+0000 7fe191600640 1 --2- 172.21.15.72:0/1057616159 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe18c105d80 0x7fe18c19b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:27.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.837+0000 7fe191600640 1 -- 172.21.15.72:0/1057616159 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe18c106740 msgr2=0x7fe18c1a23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:27.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.837+0000 7fe191600640 1 --2- 172.21.15.72:0/1057616159 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe18c106740 0x7fe18c1a23b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:27.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.837+0000 7fe191600640 1 -- 172.21.15.72:0/1057616159 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe18c1a2a20 con 0x7fe18c104b90 2024-04-26T23:43:27.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.838+0000 7fe191600640 1 --2- 172.21.15.72:0/1057616159 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe18c104b90 0x7fe18c19ae00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe18000e9c0 tx=0x7fe18000edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:27.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.838+0000 7fe18a000640 1 -- 172.21.15.72:0/1057616159 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe18000ccc0 con 0x7fe18c104b90 2024-04-26T23:43:27.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.838+0000 7fe194000640 1 -- 172.21.15.72:0/1057616159 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe18c1a2d10 con 0x7fe18c104b90 2024-04-26T23:43:27.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.838+0000 7fe18a000640 1 -- 172.21.15.72:0/1057616159 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1800035f0 con 0x7fe18c104b90 2024-04-26T23:43:27.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.838+0000 7fe18a000640 1 -- 172.21.15.72:0/1057616159 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe18000ce60 con 0x7fe18c104b90 2024-04-26T23:43:27.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.838+0000 7fe194000640 1 -- 172.21.15.72:0/1057616159 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe18c1a3180 con 0x7fe18c104b90 2024-04-26T23:43:27.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.839+0000 7fe194000640 1 -- 172.21.15.72:0/1057616159 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe18c10bfd0 con 0x7fe18c104b90 2024-04-26T23:43:27.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.841+0000 7fe18a000640 1 -- 172.21.15.72:0/1057616159 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe180004000 con 0x7fe18c104b90 2024-04-26T23:43:27.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.842+0000 7fe18a000640 1 --2- 172.21.15.72:0/1057616159 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fe1740a3c30 0x7fe1740a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:27.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.843+0000 7fe18a000640 1 -- 172.21.15.72:0/1057616159 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fe1800d0af0 con 0x7fe18c104b90 2024-04-26T23:43:27.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.843+0000 7fe190c00640 1 --2- 172.21.15.72:0/1057616159 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fe1740a3c30 0x7fe1740a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:27.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.844+0000 7fe190c00640 1 --2- 172.21.15.72:0/1057616159 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fe1740a3c30 0x7fe1740a60f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe178000c00 tx=0x7fe1780314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:27.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:27.847+0000 7fe18a000640 1 -- 172.21.15.72:0/1057616159 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe18009d230 con 0x7fe18c104b90 2024-04-26T23:43:27.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:27 smithi076 ceph-mon[32774]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T23:43:27.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:27 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/3637334506' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:43:27.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:27 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/1722648060' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T23:43:27.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:27 smithi076 ceph-mon[32774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T23:43:28.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:27 smithi072 ceph-mon[28184]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T23:43:28.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:27 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/3637334506' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:43:28.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:27 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/1722648060' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T23:43:28.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:27 smithi072 ceph-mon[28184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T23:43:28.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:27 smithi136 ceph-mon[33050]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T23:43:28.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:27 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/3637334506' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:43:28.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:27 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/1722648060' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T23:43:28.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:27 smithi136 ceph-mon[33050]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T23:43:28.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.090+0000 7fe194000640 1 -- 172.21.15.72:0/1057616159 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7fe18c02dad0 con 0x7fe18c104b90 2024-04-26T23:43:28.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.170+0000 7fe18a000640 1 -- 172.21.15.72:0/1057616159 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v19) ==== 105+0+0 (secure 0 0 0) 0x7fe18009cb30 con 0x7fe18c104b90 2024-04-26T23:43:28.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.173+0000 7fe194000640 1 -- 172.21.15.72:0/1057616159 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fe1740a3c30 msgr2=0x7fe1740a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:28.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.173+0000 7fe194000640 1 --2- 172.21.15.72:0/1057616159 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fe1740a3c30 0x7fe1740a60f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe178000c00 tx=0x7fe1780314a0 comp rx=0 tx=0).stop 2024-04-26T23:43:28.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.173+0000 7fe194000640 1 -- 172.21.15.72:0/1057616159 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe18c104b90 msgr2=0x7fe18c19ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:28.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.173+0000 7fe194000640 1 --2- 172.21.15.72:0/1057616159 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe18c104b90 0x7fe18c19ae00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe18000e9c0 tx=0x7fe18000edc0 comp rx=0 tx=0).stop 2024-04-26T23:43:28.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.173+0000 7fe194000640 1 -- 172.21.15.72:0/1057616159 shutdown_connections 2024-04-26T23:43:28.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.173+0000 7fe194000640 1 --2- 172.21.15.72:0/1057616159 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fe1740a3c30 0x7fe1740a60f0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:28.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.173+0000 7fe194000640 1 --2- 172.21.15.72:0/1057616159 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe18c105d80 0x7fe18c19b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:28.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.173+0000 7fe194000640 1 --2- 172.21.15.72:0/1057616159 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe18c106740 0x7fe18c1a23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:28.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.173+0000 7fe194000640 1 --2- 172.21.15.72:0/1057616159 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe18c104b90 0x7fe18c19ae00 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:28.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.173+0000 7fe194000640 1 -- 172.21.15.72:0/1057616159 >> 172.21.15.72:0/1057616159 conn(0x7fe18c100120 msgr2=0x7fe18c1091a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:28.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.173+0000 7fe194000640 1 -- 172.21.15.72:0/1057616159 shutdown_connections 2024-04-26T23:43:28.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.174+0000 7fe194000640 1 -- 172.21.15.72:0/1057616159 wait complete. 2024-04-26T23:43:28.201 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-04-26T23:43:28.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.379+0000 7f66c1400640 1 -- 172.21.15.72:0/4106803939 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66bc105c60 msgr2=0x7f66bc1060e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:28.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.379+0000 7f66c1400640 1 --2- 172.21.15.72:0/4106803939 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66bc105c60 0x7f66bc1060e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f66a4009900 tx=0x7f66a402a410 comp rx=0 tx=0).stop 2024-04-26T23:43:28.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.379+0000 7f66c1400640 1 -- 172.21.15.72:0/4106803939 shutdown_connections 2024-04-26T23:43:28.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.379+0000 7f66c1400640 1 --2- 172.21.15.72:0/4106803939 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f66bc106620 0x7f66bc10ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:28.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.379+0000 7f66c1400640 1 --2- 172.21.15.72:0/4106803939 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66bc105c60 0x7f66bc1060e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:28.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.379+0000 7f66c1400640 1 --2- 172.21.15.72:0/4106803939 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f66bc104a70 0x7f66bc104e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:28.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.379+0000 7f66c1400640 1 -- 172.21.15.72:0/4106803939 >> 172.21.15.72:0/4106803939 conn(0x7f66bc0fffe0 msgr2=0x7f66bc102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:28.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.380+0000 7f66c1400640 1 -- 172.21.15.72:0/4106803939 shutdown_connections 2024-04-26T23:43:28.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.380+0000 7f66c1400640 1 -- 172.21.15.72:0/4106803939 wait complete. 2024-04-26T23:43:28.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.381+0000 7f66c1400640 1 Processor -- start 2024-04-26T23:43:28.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.381+0000 7f66c1400640 1 -- start start 2024-04-26T23:43:28.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.381+0000 7f66c1400640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66bc104a70 0x7f66bc19ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:28.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.381+0000 7f66c1400640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f66bc105c60 0x7f66bc19b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:28.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.381+0000 7f66c1400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f66bc106620 0x7f66bc1a2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:28.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.381+0000 7f66c1400640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f66bc1105d0 con 0x7f66bc104a70 2024-04-26T23:43:28.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.381+0000 7f66c1400640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f66bc110450 con 0x7f66bc106620 2024-04-26T23:43:28.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.381+0000 7f66c1400640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f66bc110750 con 0x7f66bc105c60 2024-04-26T23:43:28.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.381+0000 7f66bbe00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66bc104a70 0x7f66bc19ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:28.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.381+0000 7f66bbe00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66bc104a70 0x7f66bc19ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42366/0 (socket says 172.21.15.72:42366) 2024-04-26T23:43:28.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.381+0000 7f66bbe00640 1 -- 172.21.15.72:0/399102863 learned_addr learned my addr 172.21.15.72:0/399102863 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:43:28.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.381+0000 7f66c0a00640 1 --2- 172.21.15.72:0/399102863 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f66bc106620 0x7f66bc1a2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:28.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.382+0000 7f66bbe00640 1 -- 172.21.15.72:0/399102863 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f66bc105c60 msgr2=0x7f66bc19b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:28.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.382+0000 7f66bbe00640 1 --2- 172.21.15.72:0/399102863 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f66bc105c60 0x7f66bc19b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:28.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.382+0000 7f66bbe00640 1 -- 172.21.15.72:0/399102863 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f66bc106620 msgr2=0x7f66bc1a2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:28.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.382+0000 7f66bbe00640 1 --2- 172.21.15.72:0/399102863 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f66bc106620 0x7f66bc1a2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:28.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.382+0000 7f66bbe00640 1 -- 172.21.15.72:0/399102863 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66bc1a28c0 con 0x7f66bc104a70 2024-04-26T23:43:28.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.382+0000 7f66bbe00640 1 --2- 172.21.15.72:0/399102863 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66bc104a70 0x7f66bc19ac80 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f66ac00e9c0 tx=0x7f66ac00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:28.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.382+0000 7f66b9600640 1 -- 172.21.15.72:0/399102863 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66ac00cd80 con 0x7f66bc104a70 2024-04-26T23:43:28.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.382+0000 7f66b9600640 1 -- 172.21.15.72:0/399102863 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66ac0035f0 con 0x7f66bc104a70 2024-04-26T23:43:28.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.382+0000 7f66c1400640 1 -- 172.21.15.72:0/399102863 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66bc1a2bb0 con 0x7f66bc104a70 2024-04-26T23:43:28.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.382+0000 7f66b9600640 1 -- 172.21.15.72:0/399102863 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66ac010690 con 0x7f66bc104a70 2024-04-26T23:43:28.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.383+0000 7f66c1400640 1 -- 172.21.15.72:0/399102863 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66bc1a3100 con 0x7f66bc104a70 2024-04-26T23:43:28.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.384+0000 7f66c1400640 1 -- 172.21.15.72:0/399102863 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66bc10be10 con 0x7f66bc104a70 2024-04-26T23:43:28.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.386+0000 7f66b9600640 1 -- 172.21.15.72:0/399102863 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f66ac0037a0 con 0x7f66bc104a70 2024-04-26T23:43:28.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.387+0000 7f66b9600640 1 --2- 172.21.15.72:0/399102863 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f669c0a3bb0 0x7f669c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:28.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.387+0000 7f66b9600640 1 -- 172.21.15.72:0/399102863 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f66ac0d0a90 con 0x7f66bc104a70 2024-04-26T23:43:28.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.387+0000 7f66b3e00640 1 --2- 172.21.15.72:0/399102863 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f669c0a3bb0 0x7f669c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:28.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.388+0000 7f66b3e00640 1 --2- 172.21.15.72:0/399102863 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f669c0a3bb0 0x7f669c0a6070 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f66a4009230 tx=0x7f66a4000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:28.395 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.391+0000 7f66b9600640 1 -- 172.21.15.72:0/399102863 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f66ac09d1d0 con 0x7f66bc104a70 2024-04-26T23:43:28.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:28.641+0000 7f66c1400640 1 -- 172.21.15.72:0/399102863 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f66bc063530 con 0x7f66bc104a70 2024-04-26T23:43:28.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:28 smithi076 ceph-mon[32774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-26T23:43:28.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:28 smithi076 ceph-mon[32774]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T23:43:28.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:28 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/1057616159' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-26T23:43:28.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:28 smithi076 ceph-mon[32774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-26T23:43:28.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:28 smithi076 ceph-mon[32774]: daemon mds.a assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-26T23:43:28.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:28 smithi076 ceph-mon[32774]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-26T23:43:28.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:28 smithi076 ceph-mon[32774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-26T23:43:28.930 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:28 smithi076 ceph-mon[32774]: fsmap cephfs:2 {0=b=up:active,1=a=up:creating} 10 up:standby 2024-04-26T23:43:28.930 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:28 smithi076 ceph-mon[32774]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-04-26T23:43:28.930 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:28 smithi076 ceph-mon[32774]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T23:43:29.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:28 smithi072 ceph-mon[28184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-26T23:43:29.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:28 smithi072 ceph-mon[28184]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T23:43:29.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:28 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/1057616159' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-26T23:43:29.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:28 smithi072 ceph-mon[28184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-26T23:43:29.045 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:28 smithi072 ceph-mon[28184]: daemon mds.a assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-26T23:43:29.045 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:28 smithi072 ceph-mon[28184]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-26T23:43:29.045 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:28 smithi072 ceph-mon[28184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-26T23:43:29.045 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:28 smithi072 ceph-mon[28184]: fsmap cephfs:2 {0=b=up:active,1=a=up:creating} 10 up:standby 2024-04-26T23:43:29.045 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:28 smithi072 ceph-mon[28184]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-04-26T23:43:29.045 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:28 smithi072 ceph-mon[28184]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T23:43:29.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:28 smithi136 ceph-mon[33050]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-26T23:43:29.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:28 smithi136 ceph-mon[33050]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T23:43:29.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:28 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/1057616159' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-26T23:43:29.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:28 smithi136 ceph-mon[33050]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-26T23:43:29.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:28 smithi136 ceph-mon[33050]: daemon mds.a assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-26T23:43:29.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:28 smithi136 ceph-mon[33050]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-26T23:43:29.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:28 smithi136 ceph-mon[33050]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-26T23:43:29.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:28 smithi136 ceph-mon[33050]: fsmap cephfs:2 {0=b=up:active,1=a=up:creating} 10 up:standby 2024-04-26T23:43:29.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:28 smithi136 ceph-mon[33050]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-04-26T23:43:29.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:28 smithi136 ceph-mon[33050]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T23:43:29.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.169+0000 7f66b9600640 1 -- 172.21.15.72:0/399102863 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v20) ==== 121+0+0 (secure 0 0 0) 0x7f66ac09cad0 con 0x7f66bc104a70 2024-04-26T23:43:29.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.173+0000 7f66c1400640 1 -- 172.21.15.72:0/399102863 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f669c0a3bb0 msgr2=0x7f669c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:29.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.173+0000 7f66c1400640 1 --2- 172.21.15.72:0/399102863 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f669c0a3bb0 0x7f669c0a6070 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f66a4009230 tx=0x7f66a4000bc0 comp rx=0 tx=0).stop 2024-04-26T23:43:29.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.173+0000 7f66c1400640 1 -- 172.21.15.72:0/399102863 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66bc104a70 msgr2=0x7f66bc19ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:29.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.173+0000 7f66c1400640 1 --2- 172.21.15.72:0/399102863 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66bc104a70 0x7f66bc19ac80 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f66ac00e9c0 tx=0x7f66ac00ee80 comp rx=0 tx=0).stop 2024-04-26T23:43:29.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.174+0000 7f66c1400640 1 -- 172.21.15.72:0/399102863 shutdown_connections 2024-04-26T23:43:29.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.174+0000 7f66c1400640 1 --2- 172.21.15.72:0/399102863 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f669c0a3bb0 0x7f669c0a6070 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:29.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.174+0000 7f66c1400640 1 --2- 172.21.15.72:0/399102863 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f66bc106620 0x7f66bc1a2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:29.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.174+0000 7f66c1400640 1 --2- 172.21.15.72:0/399102863 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f66bc105c60 0x7f66bc19b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:29.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.174+0000 7f66c1400640 1 --2- 172.21.15.72:0/399102863 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66bc104a70 0x7f66bc19ac80 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:29.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.174+0000 7f66c1400640 1 -- 172.21.15.72:0/399102863 >> 172.21.15.72:0/399102863 conn(0x7f66bc0fffe0 msgr2=0x7f66bc108fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:29.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.174+0000 7f66c1400640 1 -- 172.21.15.72:0/399102863 shutdown_connections 2024-04-26T23:43:29.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.174+0000 7f66c1400640 1 -- 172.21.15.72:0/399102863 wait complete. 2024-04-26T23:43:29.204 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-26T23:43:29.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.379+0000 7fd056c00640 1 -- 172.21.15.72:0/538104057 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd050105c60 msgr2=0x7fd0501060e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:29.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.379+0000 7fd056c00640 1 --2- 172.21.15.72:0/538104057 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd050105c60 0x7fd0501060e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd03c009900 tx=0x7fd03c02a400 comp rx=0 tx=0).stop 2024-04-26T23:43:29.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.379+0000 7fd056c00640 1 -- 172.21.15.72:0/538104057 shutdown_connections 2024-04-26T23:43:29.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.379+0000 7fd056c00640 1 --2- 172.21.15.72:0/538104057 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd050106620 0x7fd05010ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:29.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.380+0000 7fd056c00640 1 --2- 172.21.15.72:0/538104057 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd050105c60 0x7fd0501060e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:29.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.380+0000 7fd056c00640 1 --2- 172.21.15.72:0/538104057 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd050104a70 0x7fd050104e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:29.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.380+0000 7fd056c00640 1 -- 172.21.15.72:0/538104057 >> 172.21.15.72:0/538104057 conn(0x7fd0500fffe0 msgr2=0x7fd050102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:29.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.380+0000 7fd056c00640 1 -- 172.21.15.72:0/538104057 shutdown_connections 2024-04-26T23:43:29.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.380+0000 7fd056c00640 1 -- 172.21.15.72:0/538104057 wait complete. 2024-04-26T23:43:29.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.381+0000 7fd056c00640 1 Processor -- start 2024-04-26T23:43:29.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.381+0000 7fd056c00640 1 -- start start 2024-04-26T23:43:29.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.381+0000 7fd056c00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd050104a70 0x7fd05019ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:29.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.381+0000 7fd056c00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd050105c60 0x7fd05019b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:29.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.381+0000 7fd056c00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd050106620 0x7fd0501a2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:29.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.381+0000 7fd056c00640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fd0501105d0 con 0x7fd050105c60 2024-04-26T23:43:29.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.381+0000 7fd056c00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fd050110450 con 0x7fd050104a70 2024-04-26T23:43:29.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.381+0000 7fd056c00640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd050110750 con 0x7fd050106620 2024-04-26T23:43:29.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.382+0000 7fd054e00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd050105c60 0x7fd05019b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:29.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.382+0000 7fd054e00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd050105c60 0x7fd05019b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42384/0 (socket says 172.21.15.72:42384) 2024-04-26T23:43:29.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.382+0000 7fd054e00640 1 -- 172.21.15.72:0/1831221341 learned_addr learned my addr 172.21.15.72:0/1831221341 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:43:29.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.382+0000 7fd055800640 1 --2- 172.21.15.72:0/1831221341 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd050104a70 0x7fd05019ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:29.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.382+0000 7fd056200640 1 --2- 172.21.15.72:0/1831221341 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd050106620 0x7fd0501a2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:29.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.382+0000 7fd054e00640 1 -- 172.21.15.72:0/1831221341 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd050106620 msgr2=0x7fd0501a2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:29.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.382+0000 7fd054e00640 1 --2- 172.21.15.72:0/1831221341 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd050106620 0x7fd0501a2210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:29.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.382+0000 7fd054e00640 1 -- 172.21.15.72:0/1831221341 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd050104a70 msgr2=0x7fd05019ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:29.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.382+0000 7fd054e00640 1 --2- 172.21.15.72:0/1831221341 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd050104a70 0x7fd05019ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:29.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.382+0000 7fd054e00640 1 -- 172.21.15.72:0/1831221341 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0501a28b0 con 0x7fd050105c60 2024-04-26T23:43:29.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.382+0000 7fd055800640 1 --2- 172.21.15.72:0/1831221341 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd050104a70 0x7fd05019ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:43:29.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.382+0000 7fd054e00640 1 --2- 172.21.15.72:0/1831221341 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd050105c60 0x7fd05019b1b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd03c005ab0 tx=0x7fd03c003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:29.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.383+0000 7fd04e000640 1 -- 172.21.15.72:0/1831221341 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd03c03f070 con 0x7fd050105c60 2024-04-26T23:43:29.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.383+0000 7fd04e000640 1 -- 172.21.15.72:0/1831221341 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd03c031c50 con 0x7fd050105c60 2024-04-26T23:43:29.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.383+0000 7fd04e000640 1 -- 172.21.15.72:0/1831221341 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd03c035030 con 0x7fd050105c60 2024-04-26T23:43:29.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.383+0000 7fd056c00640 1 -- 172.21.15.72:0/1831221341 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0501a2b40 con 0x7fd050105c60 2024-04-26T23:43:29.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.383+0000 7fd056c00640 1 -- 172.21.15.72:0/1831221341 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0501a3030 con 0x7fd050105c60 2024-04-26T23:43:29.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.384+0000 7fd056c00640 1 -- 172.21.15.72:0/1831221341 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd05010be10 con 0x7fd050105c60 2024-04-26T23:43:29.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.386+0000 7fd04e000640 1 -- 172.21.15.72:0/1831221341 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd03c042070 con 0x7fd050105c60 2024-04-26T23:43:29.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.387+0000 7fd04e000640 1 --2- 172.21.15.72:0/1831221341 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fd0380a3bb0 0x7fd0380a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:29.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.387+0000 7fd04e000640 1 -- 172.21.15.72:0/1831221341 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fd03c0f0d80 con 0x7fd050105c60 2024-04-26T23:43:29.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.387+0000 7fd055800640 1 --2- 172.21.15.72:0/1831221341 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fd0380a3bb0 0x7fd0380a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:29.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.388+0000 7fd055800640 1 --2- 172.21.15.72:0/1831221341 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fd0380a3bb0 0x7fd0380a6070 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd0440095f0 tx=0x7fd044005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:29.395 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.392+0000 7fd04e000640 1 -- 172.21.15.72:0/1831221341 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd03c0bce70 con 0x7fd050105c60 2024-04-26T23:43:29.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:29.631+0000 7fd056c00640 1 -- 172.21.15.72:0/1831221341 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fd050063530 con 0x7fd050105c60 2024-04-26T23:43:29.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:29 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/399102863' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T23:43:29.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:29 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/399102863' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-26T23:43:29.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:29 smithi076 ceph-mon[32774]: mds.? [v2:172.21.15.72:6840/1356345077,v1:172.21.15.72:6841/1356345077] up:active 2024-04-26T23:43:29.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:29 smithi076 ceph-mon[32774]: daemon mds.g assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-26T23:43:29.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:29 smithi076 ceph-mon[32774]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-26T23:43:29.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:29 smithi076 ceph-mon[32774]: Cluster is now healthy 2024-04-26T23:43:29.930 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:29 smithi076 ceph-mon[32774]: fsmap cephfs:2 {0=b=up:active,1=a=up:active} 10 up:standby 2024-04-26T23:43:29.930 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:29 smithi076 ceph-mon[32774]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:creating} 9 up:standby 2024-04-26T23:43:29.930 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:29 smithi076 ceph-mon[32774]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-04-26T23:43:30.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:29 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/399102863' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T23:43:30.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:29 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/399102863' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-26T23:43:30.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:29 smithi072 ceph-mon[28184]: mds.? [v2:172.21.15.72:6840/1356345077,v1:172.21.15.72:6841/1356345077] up:active 2024-04-26T23:43:30.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:29 smithi072 ceph-mon[28184]: daemon mds.g assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-26T23:43:30.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:29 smithi072 ceph-mon[28184]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-26T23:43:30.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:29 smithi072 ceph-mon[28184]: Cluster is now healthy 2024-04-26T23:43:30.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:29 smithi072 ceph-mon[28184]: fsmap cephfs:2 {0=b=up:active,1=a=up:active} 10 up:standby 2024-04-26T23:43:30.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:29 smithi072 ceph-mon[28184]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:creating} 9 up:standby 2024-04-26T23:43:30.045 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:29 smithi072 ceph-mon[28184]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-04-26T23:43:30.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:29 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/399102863' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T23:43:30.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:29 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/399102863' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-26T23:43:30.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:29 smithi136 ceph-mon[33050]: mds.? [v2:172.21.15.72:6840/1356345077,v1:172.21.15.72:6841/1356345077] up:active 2024-04-26T23:43:30.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:29 smithi136 ceph-mon[33050]: daemon mds.g assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-26T23:43:30.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:29 smithi136 ceph-mon[33050]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-26T23:43:30.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:29 smithi136 ceph-mon[33050]: Cluster is now healthy 2024-04-26T23:43:30.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:29 smithi136 ceph-mon[33050]: fsmap cephfs:2 {0=b=up:active,1=a=up:active} 10 up:standby 2024-04-26T23:43:30.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:29 smithi136 ceph-mon[33050]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:creating} 9 up:standby 2024-04-26T23:43:30.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:29 smithi136 ceph-mon[33050]: daemon mds.g is now active in filesystem cephfs as rank 2 2024-04-26T23:43:30.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.183+0000 7fd04e000640 1 -- 172.21.15.72:0/1831221341 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v22) ==== 115+0+0 (secure 0 0 0) 0x7fd03c0c1170 con 0x7fd050105c60 2024-04-26T23:43:30.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.187+0000 7fd056c00640 1 -- 172.21.15.72:0/1831221341 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fd0380a3bb0 msgr2=0x7fd0380a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:30.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.187+0000 7fd056c00640 1 --2- 172.21.15.72:0/1831221341 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fd0380a3bb0 0x7fd0380a6070 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd0440095f0 tx=0x7fd044005f90 comp rx=0 tx=0).stop 2024-04-26T23:43:30.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.187+0000 7fd056c00640 1 -- 172.21.15.72:0/1831221341 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd050105c60 msgr2=0x7fd05019b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:30.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.187+0000 7fd056c00640 1 --2- 172.21.15.72:0/1831221341 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd050105c60 0x7fd05019b1b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd03c005ab0 tx=0x7fd03c003b90 comp rx=0 tx=0).stop 2024-04-26T23:43:30.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.188+0000 7fd056c00640 1 -- 172.21.15.72:0/1831221341 shutdown_connections 2024-04-26T23:43:30.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.188+0000 7fd056c00640 1 --2- 172.21.15.72:0/1831221341 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fd0380a3bb0 0x7fd0380a6070 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:30.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.188+0000 7fd056c00640 1 --2- 172.21.15.72:0/1831221341 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd050106620 0x7fd0501a2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:30.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.188+0000 7fd056c00640 1 --2- 172.21.15.72:0/1831221341 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd050105c60 0x7fd05019b1b0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:30.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.188+0000 7fd056c00640 1 --2- 172.21.15.72:0/1831221341 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd050104a70 0x7fd05019ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:30.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.188+0000 7fd056c00640 1 -- 172.21.15.72:0/1831221341 >> 172.21.15.72:0/1831221341 conn(0x7fd0500fffe0 msgr2=0x7fd050108fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:30.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.188+0000 7fd056c00640 1 -- 172.21.15.72:0/1831221341 shutdown_connections 2024-04-26T23:43:30.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.188+0000 7fd056c00640 1 -- 172.21.15.72:0/1831221341 wait complete. 2024-04-26T23:43:30.213 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2024-04-26T23:43:30.213 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-04-26T23:43:30.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.403+0000 7f6843000640 1 -- 172.21.15.72:0/1545858133 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f683c105c60 msgr2=0x7f683c1060e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:30.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.403+0000 7f6843000640 1 --2- 172.21.15.72:0/1545858133 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f683c105c60 0x7f683c1060e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6828009900 tx=0x7f682802a3f0 comp rx=0 tx=0).stop 2024-04-26T23:43:30.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.403+0000 7f6843000640 1 -- 172.21.15.72:0/1545858133 shutdown_connections 2024-04-26T23:43:30.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.403+0000 7f6843000640 1 --2- 172.21.15.72:0/1545858133 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f683c105c60 0x7f683c1060e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:30.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.403+0000 7f6843000640 1 --2- 172.21.15.72:0/1545858133 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f683c106620 0x7f683c10ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:30.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.403+0000 7f6843000640 1 --2- 172.21.15.72:0/1545858133 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f683c104a70 0x7f683c104e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:30.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.403+0000 7f6843000640 1 -- 172.21.15.72:0/1545858133 >> 172.21.15.72:0/1545858133 conn(0x7f683c0fffe0 msgr2=0x7f683c102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:30.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.403+0000 7f6843000640 1 -- 172.21.15.72:0/1545858133 shutdown_connections 2024-04-26T23:43:30.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.403+0000 7f6843000640 1 -- 172.21.15.72:0/1545858133 wait complete. 2024-04-26T23:43:30.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.404+0000 7f6843000640 1 Processor -- start 2024-04-26T23:43:30.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.404+0000 7f6843000640 1 -- start start 2024-04-26T23:43:30.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.404+0000 7f6843000640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f683c104a70 0x7f683c19ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:30.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.404+0000 7f6843000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f683c105c60 0x7f683c19b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:30.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.405+0000 7f6843000640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f683c106620 0x7f683c1a21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:30.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.405+0000 7f6843000640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f683c1105d0 con 0x7f683c104a70 2024-04-26T23:43:30.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.405+0000 7f6843000640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f683c110450 con 0x7f683c105c60 2024-04-26T23:43:30.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.405+0000 7f6843000640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f683c110750 con 0x7f683c106620 2024-04-26T23:43:30.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.405+0000 7f6841200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f683c105c60 0x7f683c19b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:30.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.405+0000 7f6841c00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f683c104a70 0x7f683c19ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:30.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.405+0000 7f6841200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f683c105c60 0x7f683c19b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:57838/0 (socket says 172.21.15.72:57838) 2024-04-26T23:43:30.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.405+0000 7f6841200640 1 -- 172.21.15.72:0/3482942081 learned_addr learned my addr 172.21.15.72:0/3482942081 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:43:30.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.405+0000 7f6841c00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f683c104a70 0x7f683c19ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42416/0 (socket says 172.21.15.72:42416) 2024-04-26T23:43:30.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.405+0000 7f6842600640 1 --2- 172.21.15.72:0/3482942081 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f683c106620 0x7f683c1a21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:30.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.405+0000 7f6841200640 1 -- 172.21.15.72:0/3482942081 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f683c106620 msgr2=0x7f683c1a21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:30.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.405+0000 7f6841200640 1 --2- 172.21.15.72:0/3482942081 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f683c106620 0x7f683c1a21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:30.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.405+0000 7f6841200640 1 -- 172.21.15.72:0/3482942081 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f683c104a70 msgr2=0x7f683c19ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:30.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.405+0000 7f6841200640 1 --2- 172.21.15.72:0/3482942081 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f683c104a70 0x7f683c19ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:30.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.405+0000 7f6841200640 1 -- 172.21.15.72:0/3482942081 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f683c1a2890 con 0x7f683c105c60 2024-04-26T23:43:30.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.406+0000 7f6841200640 1 --2- 172.21.15.72:0/3482942081 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f683c105c60 0x7f683c19b260 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6828005ab0 tx=0x7f6828003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:30.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.406+0000 7f683a000640 1 -- 172.21.15.72:0/3482942081 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f682803f070 con 0x7f683c105c60 2024-04-26T23:43:30.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.406+0000 7f683a000640 1 -- 172.21.15.72:0/3482942081 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6828031c50 con 0x7f683c105c60 2024-04-26T23:43:30.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.406+0000 7f6843000640 1 -- 172.21.15.72:0/3482942081 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f683c1a2b20 con 0x7f683c105c60 2024-04-26T23:43:30.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.406+0000 7f683a000640 1 -- 172.21.15.72:0/3482942081 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6828035030 con 0x7f683c105c60 2024-04-26T23:43:30.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.406+0000 7f6843000640 1 -- 172.21.15.72:0/3482942081 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f683c1a3060 con 0x7f683c105c60 2024-04-26T23:43:30.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.407+0000 7f6843000640 1 -- 172.21.15.72:0/3482942081 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f683c10bd90 con 0x7f683c105c60 2024-04-26T23:43:30.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.409+0000 7f683a000640 1 -- 172.21.15.72:0/3482942081 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6828042070 con 0x7f683c105c60 2024-04-26T23:43:30.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.410+0000 7f683a000640 1 --2- 172.21.15.72:0/3482942081 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f68240a3c30 0x7f68240a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:30.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.410+0000 7f683a000640 1 -- 172.21.15.72:0/3482942081 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f68280f0e50 con 0x7f683c105c60 2024-04-26T23:43:30.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.410+0000 7f6841c00640 1 --2- 172.21.15.72:0/3482942081 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f68240a3c30 0x7f68240a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:30.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.411+0000 7f6841c00640 1 --2- 172.21.15.72:0/3482942081 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f68240a3c30 0x7f68240a60f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f683c19c150 tx=0x7f683000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:30.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.414+0000 7f683a000640 1 -- 172.21.15.72:0/3482942081 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f68280bcf40 con 0x7f683c105c60 2024-04-26T23:43:30.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.648+0000 7f6843000640 1 -- 172.21.15.72:0/3482942081 --> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f683c062d70 con 0x7f68240a3c30 2024-04-26T23:43:30.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.682+0000 7f683a000640 1 -- 172.21.15.72:0/3482942081 <== mgr.14152 v2:172.21.15.72:6800/4281177709 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f683c062d70 con 0x7f68240a3c30 2024-04-26T23:43:30.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.685+0000 7f6843000640 1 -- 172.21.15.72:0/3482942081 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f68240a3c30 msgr2=0x7f68240a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:30.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.685+0000 7f6843000640 1 --2- 172.21.15.72:0/3482942081 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f68240a3c30 0x7f68240a60f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f683c19c150 tx=0x7f683000b410 comp rx=0 tx=0).stop 2024-04-26T23:43:30.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.685+0000 7f6843000640 1 -- 172.21.15.72:0/3482942081 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f683c105c60 msgr2=0x7f683c19b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:30.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.685+0000 7f6843000640 1 --2- 172.21.15.72:0/3482942081 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f683c105c60 0x7f683c19b260 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6828005ab0 tx=0x7f6828003b90 comp rx=0 tx=0).stop 2024-04-26T23:43:30.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.685+0000 7f6843000640 1 -- 172.21.15.72:0/3482942081 shutdown_connections 2024-04-26T23:43:30.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.685+0000 7f6843000640 1 --2- 172.21.15.72:0/3482942081 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f68240a3c30 0x7f68240a60f0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:30.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.685+0000 7f6843000640 1 --2- 172.21.15.72:0/3482942081 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f683c105c60 0x7f683c19b260 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:30.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.685+0000 7f6843000640 1 --2- 172.21.15.72:0/3482942081 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f683c106620 0x7f683c1a21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:30.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.685+0000 7f6843000640 1 --2- 172.21.15.72:0/3482942081 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f683c104a70 0x7f683c19ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:30.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.685+0000 7f6843000640 1 -- 172.21.15.72:0/3482942081 >> 172.21.15.72:0/3482942081 conn(0x7f683c0fffe0 msgr2=0x7f683c108f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:30.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.685+0000 7f6843000640 1 -- 172.21.15.72:0/3482942081 shutdown_connections 2024-04-26T23:43:30.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.685+0000 7f6843000640 1 -- 172.21.15.72:0/3482942081 wait complete. 2024-04-26T23:43:30.707 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-26T23:43:30.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.886+0000 7ffbea800640 1 -- 172.21.15.72:0/543848537 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffbe4105d80 msgr2=0x7ffbe4106200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:30.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.886+0000 7ffbea800640 1 --2- 172.21.15.72:0/543848537 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffbe4105d80 0x7ffbe4106200 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ffbd0009900 tx=0x7ffbd002a3f0 comp rx=0 tx=0).stop 2024-04-26T23:43:30.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.886+0000 7ffbea800640 1 -- 172.21.15.72:0/543848537 shutdown_connections 2024-04-26T23:43:30.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.886+0000 7ffbea800640 1 --2- 172.21.15.72:0/543848537 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffbe4105d80 0x7ffbe4106200 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:30.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.886+0000 7ffbea800640 1 --2- 172.21.15.72:0/543848537 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffbe4106740 0x7ffbe410cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:30.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.886+0000 7ffbea800640 1 --2- 172.21.15.72:0/543848537 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffbe4104b90 0x7ffbe4104fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:30.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.886+0000 7ffbea800640 1 -- 172.21.15.72:0/543848537 >> 172.21.15.72:0/543848537 conn(0x7ffbe4100120 msgr2=0x7ffbe4102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:30.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.886+0000 7ffbea800640 1 -- 172.21.15.72:0/543848537 shutdown_connections 2024-04-26T23:43:30.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.887+0000 7ffbea800640 1 -- 172.21.15.72:0/543848537 wait complete. 2024-04-26T23:43:30.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.888+0000 7ffbea800640 1 Processor -- start 2024-04-26T23:43:30.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.888+0000 7ffbea800640 1 -- start start 2024-04-26T23:43:30.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.888+0000 7ffbea800640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffbe4104b90 0x7ffbe419ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:30.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.888+0000 7ffbea800640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffbe4105d80 0x7ffbe419b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:30.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.888+0000 7ffbea800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffbe4106740 0x7ffbe41a23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:30.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.888+0000 7ffbea800640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe41106f0 con 0x7ffbe4105d80 2024-04-26T23:43:30.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.888+0000 7ffbea800640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe4110570 con 0x7ffbe4106740 2024-04-26T23:43:30.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.888+0000 7ffbea800640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe4110870 con 0x7ffbe4104b90 2024-04-26T23:43:30.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.889+0000 7ffbe2a00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffbe4105d80 0x7ffbe419b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:30.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.889+0000 7ffbe3400640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffbe4104b90 0x7ffbe419ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:30.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.889+0000 7ffbe3400640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffbe4104b90 0x7ffbe419ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.72:46464/0 (socket says 172.21.15.72:46464) 2024-04-26T23:43:30.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.889+0000 7ffbe3e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffbe4106740 0x7ffbe41a23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:30.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.889+0000 7ffbe3400640 1 -- 172.21.15.72:0/1007447358 learned_addr learned my addr 172.21.15.72:0/1007447358 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:43:30.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.889+0000 7ffbe2a00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffbe4105d80 0x7ffbe419b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42458/0 (socket says 172.21.15.72:42458) 2024-04-26T23:43:30.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.889+0000 7ffbe3400640 1 -- 172.21.15.72:0/1007447358 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffbe4106740 msgr2=0x7ffbe41a23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:30.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.889+0000 7ffbe3400640 1 --2- 172.21.15.72:0/1007447358 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffbe4106740 0x7ffbe41a23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:30.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.889+0000 7ffbe3400640 1 -- 172.21.15.72:0/1007447358 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffbe4105d80 msgr2=0x7ffbe419b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:30.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.889+0000 7ffbe3400640 1 --2- 172.21.15.72:0/1007447358 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffbe4105d80 0x7ffbe419b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:30.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.889+0000 7ffbe3400640 1 -- 172.21.15.72:0/1007447358 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbe41a2a40 con 0x7ffbe4104b90 2024-04-26T23:43:30.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.889+0000 7ffbe2a00640 1 --2- 172.21.15.72:0/1007447358 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffbe4105d80 0x7ffbe419b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:43:30.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.889+0000 7ffbe3e00640 1 --2- 172.21.15.72:0/1007447358 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffbe4106740 0x7ffbe41a23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:43:30.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.889+0000 7ffbe3400640 1 --2- 172.21.15.72:0/1007447358 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffbe4104b90 0x7ffbe419ae20 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ffbd400e9c0 tx=0x7ffbd400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:30.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.890+0000 7ffbdbe00640 1 -- 172.21.15.72:0/1007447358 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbd400cd80 con 0x7ffbe4104b90 2024-04-26T23:43:30.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.890+0000 7ffbea800640 1 -- 172.21.15.72:0/1007447358 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbe41a2d30 con 0x7ffbe4104b90 2024-04-26T23:43:30.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.890+0000 7ffbdbe00640 1 -- 172.21.15.72:0/1007447358 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffbd40035f0 con 0x7ffbe4104b90 2024-04-26T23:43:30.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.890+0000 7ffbdbe00640 1 -- 172.21.15.72:0/1007447358 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbd4010690 con 0x7ffbe4104b90 2024-04-26T23:43:30.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.890+0000 7ffbea800640 1 -- 172.21.15.72:0/1007447358 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbe41a31a0 con 0x7ffbe4104b90 2024-04-26T23:43:30.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.891+0000 7ffbea800640 1 -- 172.21.15.72:0/1007447358 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbe410bff0 con 0x7ffbe4104b90 2024-04-26T23:43:30.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.893+0000 7ffbdbe00640 1 -- 172.21.15.72:0/1007447358 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ffbd40037a0 con 0x7ffbe4104b90 2024-04-26T23:43:30.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.894+0000 7ffbdbe00640 1 --2- 172.21.15.72:0/1007447358 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ffbc80a3bb0 0x7ffbc80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:30.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.894+0000 7ffbdbe00640 1 -- 172.21.15.72:0/1007447358 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7ffbd40d09f0 con 0x7ffbe4104b90 2024-04-26T23:43:30.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.895+0000 7ffbe2a00640 1 --2- 172.21.15.72:0/1007447358 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ffbc80a3bb0 0x7ffbc80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:30.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.895+0000 7ffbe2a00640 1 --2- 172.21.15.72:0/1007447358 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ffbc80a3bb0 0x7ffbc80a6070 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ffbd0031500 tx=0x7ffbd0005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:30.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:30.896+0000 7ffbdbe00640 1 -- 172.21.15.72:0/1007447358 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ffbd409d130 con 0x7ffbe4104b90 2024-04-26T23:43:30.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:30 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/1831221341' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T23:43:30.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:30 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/1831221341' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-26T23:43:30.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:30 smithi076 ceph-mon[32774]: mds.? [v2:172.21.15.72:6838/1403162102,v1:172.21.15.72:6839/1403162102] up:active 2024-04-26T23:43:30.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:30 smithi076 ceph-mon[32774]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 9 up:standby 2024-04-26T23:43:30.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:30 smithi076 ceph-mon[32774]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T23:43:30.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:30 smithi076 ceph-mon[32774]: pgmap v335: 129 pgs: 129 active+clean; 583 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 3 op/s 2024-04-26T23:43:31.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:30 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/1831221341' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T23:43:31.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:30 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/1831221341' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-26T23:43:31.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:30 smithi072 ceph-mon[28184]: mds.? [v2:172.21.15.72:6838/1403162102,v1:172.21.15.72:6839/1403162102] up:active 2024-04-26T23:43:31.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:30 smithi072 ceph-mon[28184]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 9 up:standby 2024-04-26T23:43:31.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:30 smithi072 ceph-mon[28184]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T23:43:31.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:30 smithi072 ceph-mon[28184]: pgmap v335: 129 pgs: 129 active+clean; 583 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 3 op/s 2024-04-26T23:43:31.044 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:43:30 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:43:30.657+0000 7f6eb3a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T23:43:31.044 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:43:30 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:43:30.657+0000 7f6eb3a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T23:43:31.045 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:43:30 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:43:30.657+0000 7f6eb3a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T23:43:31.045 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:43:30 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:43:30.657+0000 7f6eb3a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T23:43:31.045 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 26 23:43:30 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-26T23:43:30.657+0000 7f6eb3a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T23:43:31.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:30 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/1831221341' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T23:43:31.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:30 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/1831221341' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-26T23:43:31.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:30 smithi136 ceph-mon[33050]: mds.? [v2:172.21.15.72:6838/1403162102,v1:172.21.15.72:6839/1403162102] up:active 2024-04-26T23:43:31.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:30 smithi136 ceph-mon[33050]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 9 up:standby 2024-04-26T23:43:31.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:30 smithi136 ceph-mon[33050]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T23:43:31.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:30 smithi136 ceph-mon[33050]: pgmap v335: 129 pgs: 129 active+clean; 583 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 3 op/s 2024-04-26T23:43:31.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.130+0000 7ffbea800640 1 -- 172.21.15.72:0/1007447358 --> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7ffbe4062d70 con 0x7ffbc80a3bb0 2024-04-26T23:43:31.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.163+0000 7ffbdbe00640 1 -- 172.21.15.72:0/1007447358 <== mgr.14152 v2:172.21.15.72:6800/4281177709 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ffbe4062d70 con 0x7ffbc80a3bb0 2024-04-26T23:43:31.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.165+0000 7ffbea800640 1 -- 172.21.15.72:0/1007447358 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ffbc80a3bb0 msgr2=0x7ffbc80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:31.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.165+0000 7ffbea800640 1 --2- 172.21.15.72:0/1007447358 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ffbc80a3bb0 0x7ffbc80a6070 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ffbd0031500 tx=0x7ffbd0005a20 comp rx=0 tx=0).stop 2024-04-26T23:43:31.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.165+0000 7ffbea800640 1 -- 172.21.15.72:0/1007447358 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffbe4104b90 msgr2=0x7ffbe419ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:31.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.165+0000 7ffbea800640 1 --2- 172.21.15.72:0/1007447358 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffbe4104b90 0x7ffbe419ae20 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ffbd400e9c0 tx=0x7ffbd400ee80 comp rx=0 tx=0).stop 2024-04-26T23:43:31.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.165+0000 7ffbea800640 1 -- 172.21.15.72:0/1007447358 shutdown_connections 2024-04-26T23:43:31.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.165+0000 7ffbea800640 1 --2- 172.21.15.72:0/1007447358 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ffbc80a3bb0 0x7ffbc80a6070 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:31.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.165+0000 7ffbea800640 1 --2- 172.21.15.72:0/1007447358 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffbe4106740 0x7ffbe41a23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:31.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.165+0000 7ffbea800640 1 --2- 172.21.15.72:0/1007447358 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffbe4105d80 0x7ffbe419b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:31.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.165+0000 7ffbea800640 1 --2- 172.21.15.72:0/1007447358 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffbe4104b90 0x7ffbe419ae20 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:31.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.165+0000 7ffbea800640 1 -- 172.21.15.72:0/1007447358 >> 172.21.15.72:0/1007447358 conn(0x7ffbe4100120 msgr2=0x7ffbe41091c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:31.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.165+0000 7ffbea800640 1 -- 172.21.15.72:0/1007447358 shutdown_connections 2024-04-26T23:43:31.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.165+0000 7ffbea800640 1 -- 172.21.15.72:0/1007447358 wait complete. 2024-04-26T23:43:31.188 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-26T23:43:31.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.380+0000 7f9fe0e00640 1 -- 172.21.15.72:0/3922126850 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9fdc109fc0 msgr2=0x7f9fdc10a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:31.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.380+0000 7f9fe0e00640 1 --2- 172.21.15.72:0/3922126850 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9fdc109fc0 0x7f9fdc10a440 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9fc4009900 tx=0x7f9fc402a410 comp rx=0 tx=0).stop 2024-04-26T23:43:31.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.380+0000 7f9fe0e00640 1 -- 172.21.15.72:0/3922126850 shutdown_connections 2024-04-26T23:43:31.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.380+0000 7f9fe0e00640 1 --2- 172.21.15.72:0/3922126850 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9fdc10a980 0x7f9fdc1112a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:31.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.380+0000 7f9fe0e00640 1 --2- 172.21.15.72:0/3922126850 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9fdc109fc0 0x7f9fdc10a440 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:31.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.380+0000 7f9fe0e00640 1 --2- 172.21.15.72:0/3922126850 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9fdc108e70 0x7f9fdc109290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:31.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.380+0000 7f9fe0e00640 1 -- 172.21.15.72:0/3922126850 >> 172.21.15.72:0/3922126850 conn(0x7f9fdc077e00 msgr2=0x7f9fdc07a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:31.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.380+0000 7f9fe0e00640 1 -- 172.21.15.72:0/3922126850 shutdown_connections 2024-04-26T23:43:31.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.380+0000 7f9fe0e00640 1 -- 172.21.15.72:0/3922126850 wait complete. 2024-04-26T23:43:31.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.382+0000 7f9fe0e00640 1 Processor -- start 2024-04-26T23:43:31.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.382+0000 7f9fe0e00640 1 -- start start 2024-04-26T23:43:31.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.382+0000 7f9fe0e00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9fdc108e70 0x7f9fdc19f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:31.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.382+0000 7f9fe0e00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9fdc109fc0 0x7f9fdc19f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:31.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.382+0000 7f9fe0e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9fdc10a980 0x7f9fdc1a6650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:31.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.382+0000 7f9fe0e00640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f9fdc1149a0 con 0x7f9fdc109fc0 2024-04-26T23:43:31.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.382+0000 7f9fe0e00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f9fdc114820 con 0x7f9fdc10a980 2024-04-26T23:43:31.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.382+0000 7f9fe0e00640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f9fdc114b20 con 0x7f9fdc108e70 2024-04-26T23:43:31.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.382+0000 7f9fd3e00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9fdc109fc0 0x7f9fdc19f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:31.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.382+0000 7f9fdb400640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9fdc108e70 0x7f9fdc19f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:31.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.383+0000 7f9fd3e00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9fdc109fc0 0x7f9fdc19f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42476/0 (socket says 172.21.15.72:42476) 2024-04-26T23:43:31.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.383+0000 7f9fdb400640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9fdc108e70 0x7f9fdc19f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.72:46492/0 (socket says 172.21.15.72:46492) 2024-04-26T23:43:31.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.383+0000 7f9fd3e00640 1 -- 172.21.15.72:0/4020910194 learned_addr learned my addr 172.21.15.72:0/4020910194 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:43:31.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.383+0000 7f9fdbe00640 1 --2- 172.21.15.72:0/4020910194 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9fdc10a980 0x7f9fdc1a6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:31.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.383+0000 7f9fdb400640 1 -- 172.21.15.72:0/4020910194 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9fdc10a980 msgr2=0x7f9fdc1a6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:31.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.383+0000 7f9fdb400640 1 --2- 172.21.15.72:0/4020910194 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9fdc10a980 0x7f9fdc1a6650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:31.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.383+0000 7f9fdb400640 1 -- 172.21.15.72:0/4020910194 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9fdc109fc0 msgr2=0x7f9fdc19f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:31.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.383+0000 7f9fdb400640 1 --2- 172.21.15.72:0/4020910194 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9fdc109fc0 0x7f9fdc19f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:31.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.383+0000 7f9fdb400640 1 -- 172.21.15.72:0/4020910194 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fdc1a6cc0 con 0x7f9fdc108e70 2024-04-26T23:43:31.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.383+0000 7f9fdb400640 1 --2- 172.21.15.72:0/4020910194 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9fdc108e70 0x7f9fdc19f0a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc00e9f0 tx=0x7f9fcc00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:31.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.384+0000 7f9fd8c00640 1 -- 172.21.15.72:0/4020910194 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fcc00ccc0 con 0x7f9fdc108e70 2024-04-26T23:43:31.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.384+0000 7f9fe0e00640 1 -- 172.21.15.72:0/4020910194 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fdc1a6fb0 con 0x7f9fdc108e70 2024-04-26T23:43:31.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.384+0000 7f9fd8c00640 1 -- 172.21.15.72:0/4020910194 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9fcc0108e0 con 0x7f9fdc108e70 2024-04-26T23:43:31.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.384+0000 7f9fd8c00640 1 -- 172.21.15.72:0/4020910194 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fcc018840 con 0x7f9fdc108e70 2024-04-26T23:43:31.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.384+0000 7f9fe0e00640 1 -- 172.21.15.72:0/4020910194 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fdc1a7420 con 0x7f9fdc108e70 2024-04-26T23:43:31.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.385+0000 7f9fe0e00640 1 -- 172.21.15.72:0/4020910194 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fdc110200 con 0x7f9fdc108e70 2024-04-26T23:43:31.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.387+0000 7f9fd8c00640 1 -- 172.21.15.72:0/4020910194 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9fcc010430 con 0x7f9fdc108e70 2024-04-26T23:43:31.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.388+0000 7f9fd8c00640 1 --2- 172.21.15.72:0/4020910194 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f9fbc0a3bb0 0x7f9fbc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:31.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.389+0000 7f9fd8c00640 1 -- 172.21.15.72:0/4020910194 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f9fcc0d08d0 con 0x7f9fdc108e70 2024-04-26T23:43:31.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.389+0000 7f9fd3e00640 1 --2- 172.21.15.72:0/4020910194 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f9fbc0a3bb0 0x7f9fbc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:31.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.389+0000 7f9fd3e00640 1 --2- 172.21.15.72:0/4020910194 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f9fbc0a3bb0 0x7f9fbc0a6070 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc1a05b0 tx=0x7f9fc40314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:31.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.393+0000 7f9fd8c00640 1 -- 172.21.15.72:0/4020910194 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9fcc09d010 con 0x7f9fdc108e70 2024-04-26T23:43:31.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.634+0000 7f9fe0e00640 1 -- 172.21.15.72:0/4020910194 --> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f9fdc062d70 con 0x7f9fbc0a3bb0 2024-04-26T23:43:31.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.641+0000 7f9fd8c00640 1 -- 172.21.15.72:0/4020910194 <== mgr.14152 v2:172.21.15.72:6800/4281177709 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f9fdc062d70 con 0x7f9fbc0a3bb0 2024-04-26T23:43:31.645 INFO:teuthology.orchestra.run.smithi072.stdout:/volumes/qa/sv_0/dc331c62-b6ab-4d28-8c32-22a1f0c2f5d7 2024-04-26T23:43:31.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.643+0000 7f9fe0e00640 1 -- 172.21.15.72:0/4020910194 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f9fbc0a3bb0 msgr2=0x7f9fbc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:31.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.643+0000 7f9fe0e00640 1 --2- 172.21.15.72:0/4020910194 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f9fbc0a3bb0 0x7f9fbc0a6070 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc1a05b0 tx=0x7f9fc40314a0 comp rx=0 tx=0).stop 2024-04-26T23:43:31.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.643+0000 7f9fe0e00640 1 -- 172.21.15.72:0/4020910194 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9fdc108e70 msgr2=0x7f9fdc19f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:31.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.643+0000 7f9fe0e00640 1 --2- 172.21.15.72:0/4020910194 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9fdc108e70 0x7f9fdc19f0a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc00e9f0 tx=0x7f9fcc00edf0 comp rx=0 tx=0).stop 2024-04-26T23:43:31.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.644+0000 7f9fe0e00640 1 -- 172.21.15.72:0/4020910194 shutdown_connections 2024-04-26T23:43:31.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.644+0000 7f9fe0e00640 1 --2- 172.21.15.72:0/4020910194 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f9fbc0a3bb0 0x7f9fbc0a6070 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:31.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.644+0000 7f9fe0e00640 1 --2- 172.21.15.72:0/4020910194 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9fdc10a980 0x7f9fdc1a6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:31.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.644+0000 7f9fe0e00640 1 --2- 172.21.15.72:0/4020910194 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9fdc109fc0 0x7f9fdc19f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:31.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.644+0000 7f9fe0e00640 1 --2- 172.21.15.72:0/4020910194 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9fdc108e70 0x7f9fdc19f0a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:31.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.644+0000 7f9fe0e00640 1 -- 172.21.15.72:0/4020910194 >> 172.21.15.72:0/4020910194 conn(0x7f9fdc077e00 msgr2=0x7f9fdc10d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:31.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.644+0000 7f9fe0e00640 1 -- 172.21.15.72:0/4020910194 shutdown_connections 2024-04-26T23:43:31.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.644+0000 7f9fe0e00640 1 -- 172.21.15.72:0/4020910194 wait complete. 2024-04-26T23:43:31.666 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-26T23:43:31.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.866+0000 7f2652e00640 1 -- 172.21.15.72:0/3581838927 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f264c104a70 msgr2=0x7f264c104e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:31.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.866+0000 7f2652e00640 1 --2- 172.21.15.72:0/3581838927 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f264c104a70 0x7f264c104e90 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f2640009900 tx=0x7f264002a410 comp rx=0 tx=0).stop 2024-04-26T23:43:31.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.866+0000 7f2652e00640 1 -- 172.21.15.72:0/3581838927 shutdown_connections 2024-04-26T23:43:31.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.866+0000 7f2652e00640 1 --2- 172.21.15.72:0/3581838927 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f264c105c60 0x7f264c1060e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:31.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.866+0000 7f2652e00640 1 --2- 172.21.15.72:0/3581838927 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f264c106620 0x7f264c10ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:31.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.866+0000 7f2652e00640 1 --2- 172.21.15.72:0/3581838927 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f264c104a70 0x7f264c104e90 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:31.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.866+0000 7f2652e00640 1 -- 172.21.15.72:0/3581838927 >> 172.21.15.72:0/3581838927 conn(0x7f264c0fffe0 msgr2=0x7f264c102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:31.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.866+0000 7f2652e00640 1 -- 172.21.15.72:0/3581838927 shutdown_connections 2024-04-26T23:43:31.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.866+0000 7f2652e00640 1 -- 172.21.15.72:0/3581838927 wait complete. 2024-04-26T23:43:31.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.867+0000 7f2652e00640 1 Processor -- start 2024-04-26T23:43:31.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.867+0000 7f2652e00640 1 -- start start 2024-04-26T23:43:31.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.867+0000 7f2652e00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f264c104a70 0x7f264c19ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:31.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.867+0000 7f2652e00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f264c105c60 0x7f264c19b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:31.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.868+0000 7f2652e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f264c106620 0x7f264c1a2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:31.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.868+0000 7f2652e00640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f264c110660 con 0x7f264c104a70 2024-04-26T23:43:31.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.868+0000 7f2652e00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f264c1104e0 con 0x7f264c106620 2024-04-26T23:43:31.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.868+0000 7f2652e00640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f264c1107e0 con 0x7f264c105c60 2024-04-26T23:43:31.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.868+0000 7f2651a00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f264c104a70 0x7f264c19ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:31.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.868+0000 7f2651a00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f264c104a70 0x7f264c19ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42496/0 (socket says 172.21.15.72:42496) 2024-04-26T23:43:31.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.868+0000 7f2651a00640 1 -- 172.21.15.72:0/2523053973 learned_addr learned my addr 172.21.15.72:0/2523053973 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:43:31.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.868+0000 7f2651000640 1 --2- 172.21.15.72:0/2523053973 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f264c105c60 0x7f264c19b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:31.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.868+0000 7f2652400640 1 --2- 172.21.15.72:0/2523053973 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f264c106620 0x7f264c1a2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:31.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.868+0000 7f2651a00640 1 -- 172.21.15.72:0/2523053973 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f264c105c60 msgr2=0x7f264c19b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:31.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.868+0000 7f2651a00640 1 --2- 172.21.15.72:0/2523053973 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f264c105c60 0x7f264c19b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:31.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.868+0000 7f2651a00640 1 -- 172.21.15.72:0/2523053973 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f264c106620 msgr2=0x7f264c1a2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:31.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.868+0000 7f2651a00640 1 --2- 172.21.15.72:0/2523053973 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f264c106620 0x7f264c1a2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:31.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.868+0000 7f2651a00640 1 -- 172.21.15.72:0/2523053973 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f264c1a2980 con 0x7f264c104a70 2024-04-26T23:43:31.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.869+0000 7f2651a00640 1 --2- 172.21.15.72:0/2523053973 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f264c104a70 0x7f264c19ad60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2640005ab0 tx=0x7f2640003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:31.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.869+0000 7f264a000640 1 -- 172.21.15.72:0/2523053973 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f264003f070 con 0x7f264c104a70 2024-04-26T23:43:31.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.869+0000 7f264a000640 1 -- 172.21.15.72:0/2523053973 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2640031c50 con 0x7f264c104a70 2024-04-26T23:43:31.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.869+0000 7f2652e00640 1 -- 172.21.15.72:0/2523053973 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f264c1a2c10 con 0x7f264c104a70 2024-04-26T23:43:31.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.869+0000 7f264a000640 1 -- 172.21.15.72:0/2523053973 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2640035030 con 0x7f264c104a70 2024-04-26T23:43:31.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.869+0000 7f2652e00640 1 -- 172.21.15.72:0/2523053973 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f264c1a3100 con 0x7f264c104a70 2024-04-26T23:43:31.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.870+0000 7f2652e00640 1 -- 172.21.15.72:0/2523053973 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f264c10be00 con 0x7f264c104a70 2024-04-26T23:43:31.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.872+0000 7f264a000640 1 -- 172.21.15.72:0/2523053973 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2640042070 con 0x7f264c104a70 2024-04-26T23:43:31.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.873+0000 7f264a000640 1 --2- 172.21.15.72:0/2523053973 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f26340a3bb0 0x7f26340a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:31.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.873+0000 7f264a000640 1 -- 172.21.15.72:0/2523053973 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f26400f0e00 con 0x7f264c104a70 2024-04-26T23:43:31.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.874+0000 7f2651000640 1 --2- 172.21.15.72:0/2523053973 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f26340a3bb0 0x7f26340a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:31.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.874+0000 7f2651000640 1 --2- 172.21.15.72:0/2523053973 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f26340a3bb0 0x7f26340a6070 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f264c19c270 tx=0x7f2638005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:31.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:31.878+0000 7f264a000640 1 -- 172.21.15.72:0/2523053973 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f26400bcef0 con 0x7f264c104a70 2024-04-26T23:43:32.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.093+0000 7f2652e00640 1 -- 172.21.15.72:0/2523053973 --> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f264c062e00 con 0x7f26340a3bb0 2024-04-26T23:43:32.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.132+0000 7f264a000640 1 -- 172.21.15.72:0/2523053973 <== mgr.14152 v2:172.21.15.72:6800/4281177709 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f264c062e00 con 0x7f26340a3bb0 2024-04-26T23:43:32.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.134+0000 7f2652e00640 1 -- 172.21.15.72:0/2523053973 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f26340a3bb0 msgr2=0x7f26340a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:32.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.134+0000 7f2652e00640 1 --2- 172.21.15.72:0/2523053973 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f26340a3bb0 0x7f26340a6070 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f264c19c270 tx=0x7f2638005f90 comp rx=0 tx=0).stop 2024-04-26T23:43:32.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.134+0000 7f2652e00640 1 -- 172.21.15.72:0/2523053973 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f264c104a70 msgr2=0x7f264c19ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:32.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.134+0000 7f2652e00640 1 --2- 172.21.15.72:0/2523053973 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f264c104a70 0x7f264c19ad60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2640005ab0 tx=0x7f2640003c10 comp rx=0 tx=0).stop 2024-04-26T23:43:32.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.134+0000 7f2652e00640 1 -- 172.21.15.72:0/2523053973 shutdown_connections 2024-04-26T23:43:32.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.134+0000 7f2652e00640 1 --2- 172.21.15.72:0/2523053973 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f26340a3bb0 0x7f26340a6070 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:32.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.134+0000 7f2652e00640 1 --2- 172.21.15.72:0/2523053973 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f264c106620 0x7f264c1a2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:32.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.134+0000 7f2652e00640 1 --2- 172.21.15.72:0/2523053973 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f264c105c60 0x7f264c19b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:32.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.134+0000 7f2652e00640 1 --2- 172.21.15.72:0/2523053973 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f264c104a70 0x7f264c19ad60 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:32.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.134+0000 7f2652e00640 1 -- 172.21.15.72:0/2523053973 >> 172.21.15.72:0/2523053973 conn(0x7f264c0fffe0 msgr2=0x7f264c108fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:32.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.135+0000 7f2652e00640 1 -- 172.21.15.72:0/2523053973 shutdown_connections 2024-04-26T23:43:32.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.135+0000 7f2652e00640 1 -- 172.21.15.72:0/2523053973 wait complete. 2024-04-26T23:43:32.157 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-26T23:43:32.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.358+0000 7f357f200640 1 -- 172.21.15.72:0/3814752044 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3580109fc0 msgr2=0x7f358010a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:32.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.358+0000 7f357f200640 1 --2- 172.21.15.72:0/3814752044 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3580109fc0 0x7f358010a440 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3568009900 tx=0x7f356802a410 comp rx=0 tx=0).stop 2024-04-26T23:43:32.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.358+0000 7f357f200640 1 -- 172.21.15.72:0/3814752044 shutdown_connections 2024-04-26T23:43:32.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.358+0000 7f357f200640 1 --2- 172.21.15.72:0/3814752044 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f358010a980 0x7f35801112a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:32.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.358+0000 7f357f200640 1 --2- 172.21.15.72:0/3814752044 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3580109fc0 0x7f358010a440 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:32.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.358+0000 7f357f200640 1 --2- 172.21.15.72:0/3814752044 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3580108e70 0x7f3580109290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:32.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.358+0000 7f357f200640 1 -- 172.21.15.72:0/3814752044 >> 172.21.15.72:0/3814752044 conn(0x7f3580077e00 msgr2=0x7f358007a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:32.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.359+0000 7f357f200640 1 -- 172.21.15.72:0/3814752044 shutdown_connections 2024-04-26T23:43:32.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.359+0000 7f357f200640 1 -- 172.21.15.72:0/3814752044 wait complete. 2024-04-26T23:43:32.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.360+0000 7f357f200640 1 Processor -- start 2024-04-26T23:43:32.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.360+0000 7f357f200640 1 -- start start 2024-04-26T23:43:32.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.360+0000 7f357f200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3580108e70 0x7f358019f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:32.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.360+0000 7f357f200640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3580109fc0 0x7f358019f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:32.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.360+0000 7f357f200640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f358010a980 0x7f35801a6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:32.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.360+0000 7f357f200640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f35801149f0 con 0x7f3580109fc0 2024-04-26T23:43:32.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.360+0000 7f357f200640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3580114870 con 0x7f3580108e70 2024-04-26T23:43:32.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.360+0000 7f357f200640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f3580114b70 con 0x7f358010a980 2024-04-26T23:43:32.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.360+0000 7f3577e00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3580109fc0 0x7f358019f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:32.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.360+0000 7f3577e00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3580109fc0 0x7f358019f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42518/0 (socket says 172.21.15.72:42518) 2024-04-26T23:43:32.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.360+0000 7f3577e00640 1 -- 172.21.15.72:0/2220424239 learned_addr learned my addr 172.21.15.72:0/2220424239 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:43:32.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.360+0000 7f357de00640 1 --2- 172.21.15.72:0/2220424239 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3580108e70 0x7f358019f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:32.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.360+0000 7f357e800640 1 --2- 172.21.15.72:0/2220424239 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f358010a980 0x7f35801a6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:32.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.361+0000 7f3577e00640 1 -- 172.21.15.72:0/2220424239 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f358010a980 msgr2=0x7f35801a6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:32.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.361+0000 7f3577e00640 1 --2- 172.21.15.72:0/2220424239 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f358010a980 0x7f35801a6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:32.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.361+0000 7f3577e00640 1 -- 172.21.15.72:0/2220424239 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3580108e70 msgr2=0x7f358019f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:32.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.361+0000 7f3577e00640 1 --2- 172.21.15.72:0/2220424239 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3580108e70 0x7f358019f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:32.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.361+0000 7f3577e00640 1 -- 172.21.15.72:0/2220424239 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35801a6c80 con 0x7f3580109fc0 2024-04-26T23:43:32.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.361+0000 7f3577e00640 1 --2- 172.21.15.72:0/2220424239 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3580109fc0 0x7f358019f5a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3568031500 tx=0x7f3568003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:32.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.361+0000 7f3576a00640 1 -- 172.21.15.72:0/2220424239 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f356803f070 con 0x7f3580109fc0 2024-04-26T23:43:32.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.361+0000 7f3576a00640 1 -- 172.21.15.72:0/2220424239 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3568031c50 con 0x7f3580109fc0 2024-04-26T23:43:32.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.361+0000 7f3576a00640 1 -- 172.21.15.72:0/2220424239 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3568035030 con 0x7f3580109fc0 2024-04-26T23:43:32.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.362+0000 7f357f200640 1 -- 172.21.15.72:0/2220424239 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35801a6f10 con 0x7f3580109fc0 2024-04-26T23:43:32.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.362+0000 7f357f200640 1 -- 172.21.15.72:0/2220424239 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35801a72c0 con 0x7f3580109fc0 2024-04-26T23:43:32.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.363+0000 7f357f200640 1 -- 172.21.15.72:0/2220424239 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3580110180 con 0x7f3580109fc0 2024-04-26T23:43:32.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.365+0000 7f3576a00640 1 -- 172.21.15.72:0/2220424239 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3568042070 con 0x7f3580109fc0 2024-04-26T23:43:32.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.367+0000 7f3576a00640 1 --2- 172.21.15.72:0/2220424239 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f35640a3c30 0x7f35640a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:32.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.367+0000 7f3576a00640 1 -- 172.21.15.72:0/2220424239 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f35680f0d50 con 0x7f3580109fc0 2024-04-26T23:43:32.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.367+0000 7f357de00640 1 --2- 172.21.15.72:0/2220424239 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f35640a3c30 0x7f35640a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:32.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.368+0000 7f357de00640 1 --2- 172.21.15.72:0/2220424239 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f35640a3c30 0x7f35640a60f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3570009bd0 tx=0x7f3570005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:32.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.371+0000 7f3576a00640 1 -- 172.21.15.72:0/2220424239 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f35680bce40 con 0x7f3580109fc0 2024-04-26T23:43:32.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:32 smithi072 ceph-mon[28184]: from='client.24830 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:43:32.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:32 smithi072 ceph-mon[28184]: from='client.24820 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:43:32.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:32 smithi136 ceph-mon[33050]: from='client.24830 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:43:32.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:32 smithi136 ceph-mon[33050]: from='client.24820 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:43:32.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.610+0000 7f357f200640 1 -- 172.21.15.72:0/2220424239 --> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f3580062e00 con 0x7f35640a3c30 2024-04-26T23:43:32.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.622+0000 7f3576a00640 1 -- 172.21.15.72:0/2220424239 <== mgr.14152 v2:172.21.15.72:6800/4281177709 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f3580062e00 con 0x7f35640a3c30 2024-04-26T23:43:32.626 INFO:teuthology.orchestra.run.smithi072.stdout:/volumes/qa/sv_1/b5db3a0e-4c33-4081-af16-2a62694d4438 2024-04-26T23:43:32.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.624+0000 7f357f200640 1 -- 172.21.15.72:0/2220424239 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f35640a3c30 msgr2=0x7f35640a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:32.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.624+0000 7f357f200640 1 --2- 172.21.15.72:0/2220424239 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f35640a3c30 0x7f35640a60f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3570009bd0 tx=0x7f3570005f90 comp rx=0 tx=0).stop 2024-04-26T23:43:32.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.624+0000 7f357f200640 1 -- 172.21.15.72:0/2220424239 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3580109fc0 msgr2=0x7f358019f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:32.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.624+0000 7f357f200640 1 --2- 172.21.15.72:0/2220424239 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3580109fc0 0x7f358019f5a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3568031500 tx=0x7f3568003c10 comp rx=0 tx=0).stop 2024-04-26T23:43:32.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.625+0000 7f357f200640 1 -- 172.21.15.72:0/2220424239 shutdown_connections 2024-04-26T23:43:32.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.625+0000 7f357f200640 1 --2- 172.21.15.72:0/2220424239 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f35640a3c30 0x7f35640a60f0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:32.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.625+0000 7f357f200640 1 --2- 172.21.15.72:0/2220424239 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f358010a980 0x7f35801a6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:32.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.625+0000 7f357f200640 1 --2- 172.21.15.72:0/2220424239 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3580109fc0 0x7f358019f5a0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:32.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.625+0000 7f357f200640 1 --2- 172.21.15.72:0/2220424239 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3580108e70 0x7f358019f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:32.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.625+0000 7f357f200640 1 -- 172.21.15.72:0/2220424239 >> 172.21.15.72:0/2220424239 conn(0x7f3580077e00 msgr2=0x7f358010d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:32.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.625+0000 7f357f200640 1 -- 172.21.15.72:0/2220424239 shutdown_connections 2024-04-26T23:43:32.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.625+0000 7f357f200640 1 -- 172.21.15.72:0/2220424239 wait complete. 2024-04-26T23:43:32.647 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:43:32.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:32 smithi076 ceph-mon[32774]: from='client.24830 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:43:32.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:32 smithi076 ceph-mon[32774]: from='client.24820 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:43:32.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.838+0000 7f6c1b800640 1 -- 172.21.15.72:0/4187407485 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6c14104b90 msgr2=0x7f6c14104fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:32.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.838+0000 7f6c1b800640 1 --2- 172.21.15.72:0/4187407485 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6c14104b90 0x7f6c14104fb0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6c08009900 tx=0x7f6c0802a410 comp rx=0 tx=0).stop 2024-04-26T23:43:32.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.839+0000 7f6c1b800640 1 -- 172.21.15.72:0/4187407485 shutdown_connections 2024-04-26T23:43:32.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.839+0000 7f6c1b800640 1 --2- 172.21.15.72:0/4187407485 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6c14106740 0x7f6c1410cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:32.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.839+0000 7f6c1b800640 1 --2- 172.21.15.72:0/4187407485 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c14105d80 0x7f6c14106200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:32.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.839+0000 7f6c1b800640 1 --2- 172.21.15.72:0/4187407485 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6c14104b90 0x7f6c14104fb0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:32.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.839+0000 7f6c1b800640 1 -- 172.21.15.72:0/4187407485 >> 172.21.15.72:0/4187407485 conn(0x7f6c14100120 msgr2=0x7f6c14102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:32.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.839+0000 7f6c1b800640 1 -- 172.21.15.72:0/4187407485 shutdown_connections 2024-04-26T23:43:32.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.839+0000 7f6c1b800640 1 -- 172.21.15.72:0/4187407485 wait complete. 2024-04-26T23:43:32.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.840+0000 7f6c1b800640 1 Processor -- start 2024-04-26T23:43:32.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.840+0000 7f6c1b800640 1 -- start start 2024-04-26T23:43:32.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.840+0000 7f6c1b800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6c14104b90 0x7f6c1419adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:32.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.840+0000 7f6c1b800640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c14105d80 0x7f6c1419b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:32.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.840+0000 7f6c1b800640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6c14106740 0x7f6c141a2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:32.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.840+0000 7f6c1b800640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f6c14110700 con 0x7f6c14106740 2024-04-26T23:43:32.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.840+0000 7f6c1b800640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f6c14110580 con 0x7f6c14104b90 2024-04-26T23:43:32.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.840+0000 7f6c1b800640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f6c14110880 con 0x7f6c14105d80 2024-04-26T23:43:32.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.840+0000 7f6c18e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6c14104b90 0x7f6c1419adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:32.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.840+0000 7f6c19800640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6c14106740 0x7f6c141a2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:32.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.840+0000 7f6c18e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6c14104b90 0x7f6c1419adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:57942/0 (socket says 172.21.15.72:57942) 2024-04-26T23:43:32.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.840+0000 7f6c18e00640 1 -- 172.21.15.72:0/2811662932 learned_addr learned my addr 172.21.15.72:0/2811662932 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:43:32.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.840+0000 7f6c13e00640 1 --2- 172.21.15.72:0/2811662932 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c14105d80 0x7f6c1419b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:32.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.841+0000 7f6c18e00640 1 -- 172.21.15.72:0/2811662932 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c14105d80 msgr2=0x7f6c1419b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:32.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.841+0000 7f6c18e00640 1 --2- 172.21.15.72:0/2811662932 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c14105d80 0x7f6c1419b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:32.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.841+0000 7f6c18e00640 1 -- 172.21.15.72:0/2811662932 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6c14106740 msgr2=0x7f6c141a2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:32.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.841+0000 7f6c18e00640 1 --2- 172.21.15.72:0/2811662932 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6c14106740 0x7f6c141a2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:32.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.841+0000 7f6c18e00640 1 -- 172.21.15.72:0/2811662932 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c141a29d0 con 0x7f6c14104b90 2024-04-26T23:43:32.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.841+0000 7f6c19800640 1 --2- 172.21.15.72:0/2811662932 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6c14106740 0x7f6c141a2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:43:32.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.841+0000 7f6c13e00640 1 --2- 172.21.15.72:0/2811662932 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c14105d80 0x7f6c1419b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:43:32.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.841+0000 7f6c18e00640 1 --2- 172.21.15.72:0/2811662932 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6c14104b90 0x7f6c1419adc0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f6c08031500 tx=0x7f6c08003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:32.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.841+0000 7f6c11600640 1 -- 172.21.15.72:0/2811662932 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c0803f070 con 0x7f6c14104b90 2024-04-26T23:43:32.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.841+0000 7f6c1b800640 1 -- 172.21.15.72:0/2811662932 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c141a2c60 con 0x7f6c14104b90 2024-04-26T23:43:32.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.841+0000 7f6c11600640 1 -- 172.21.15.72:0/2811662932 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c08031c50 con 0x7f6c14104b90 2024-04-26T23:43:32.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.841+0000 7f6c11600640 1 -- 172.21.15.72:0/2811662932 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c08035030 con 0x7f6c14104b90 2024-04-26T23:43:32.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.842+0000 7f6c1b800640 1 -- 172.21.15.72:0/2811662932 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c141a31a0 con 0x7f6c14104b90 2024-04-26T23:43:32.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.843+0000 7f6c1b800640 1 -- 172.21.15.72:0/2811662932 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c1410bf30 con 0x7f6c14104b90 2024-04-26T23:43:32.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.845+0000 7f6c11600640 1 -- 172.21.15.72:0/2811662932 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6c08042070 con 0x7f6c14104b90 2024-04-26T23:43:32.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.847+0000 7f6c11600640 1 --2- 172.21.15.72:0/2811662932 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6bfc0a3bb0 0x7f6bfc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:32.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.847+0000 7f6c11600640 1 -- 172.21.15.72:0/2811662932 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f6c080f1540 con 0x7f6c14104b90 2024-04-26T23:43:32.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.847+0000 7f6c13e00640 1 --2- 172.21.15.72:0/2811662932 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6bfc0a3bb0 0x7f6bfc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:32.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.848+0000 7f6c13e00640 1 --2- 172.21.15.72:0/2811662932 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6bfc0a3bb0 0x7f6bfc0a6070 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6c00009fc0 tx=0x7f6c00005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:32.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:32.848+0000 7f6c11600640 1 -- 172.21.15.72:0/2811662932 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6c080bd630 con 0x7f6c14104b90 2024-04-26T23:43:33.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.092+0000 7f6c1b800640 1 -- 172.21.15.72:0/2811662932 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6c1402dad0 con 0x7f6c14104b90 2024-04-26T23:43:33.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.094+0000 7f6c11600640 1 -- 172.21.15.72:0/2811662932 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11874 (secure 0 0 0) 0x7f6c080c1930 con 0x7f6c14104b90 2024-04-26T23:43:33.098 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:43:33.099 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/2386456166","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2386456166},{"type":"v1","addr":"172.21.15.72:6835","nonce":2386456166}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1528342053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1528342053},{"type":"v1","addr":"172.21.15.136:6833","nonce":1528342053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6833/2427238559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":2427238559},{"type":"v1","addr":"172.21.15.76:6833","nonce":2427238559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24491,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/4050689894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":4050689894},{"type":"v1","addr":"172.21.15.72:6837","nonce":4050689894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3681925367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3681925367},{"type":"v1","addr":"172.21.15.136:6835","nonce":3681925367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4251109085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4251109085},{"type":"v1","addr":"172.21.15.76:6835","nonce":4251109085}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T23:43:02.928649+0000","modified":"2024-04-26T23:43:30.190394+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24590,"mds_1":24571,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24514":{"gid":24514,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6837/3639308555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3639308555},{"type":"v1","addr":"172.21.15.136:6837","nonce":3639308555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.76:6837/2506930945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2506930945},{"type":"v1","addr":"172.21.15.76:6837","nonce":2506930945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6839/3938677630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3938677630},{"type":"v1","addr":"172.21.15.136:6839","nonce":3938677630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.72:6839/1403162102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6838","nonce":1403162102},{"type":"v1","addr":"172.21.15.72:6839","nonce":1403162102}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24571":{"gid":24571,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.72:6841/1356345077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6840","nonce":1356345077},{"type":"v1","addr":"172.21.15.72:6841","nonce":1356345077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.76:6839/3405829058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":3405829058},{"type":"v1","addr":"172.21.15.76:6839","nonce":3405829058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24553,24571,24590]},"id":1}]} 2024-04-26T23:43:33.099 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 23 2024-04-26T23:43:33.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.097+0000 7f6c1b800640 1 -- 172.21.15.72:0/2811662932 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6bfc0a3bb0 msgr2=0x7f6bfc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:33.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.097+0000 7f6c1b800640 1 --2- 172.21.15.72:0/2811662932 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6bfc0a3bb0 0x7f6bfc0a6070 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6c00009fc0 tx=0x7f6c00005f90 comp rx=0 tx=0).stop 2024-04-26T23:43:33.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.097+0000 7f6c1b800640 1 -- 172.21.15.72:0/2811662932 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6c14104b90 msgr2=0x7f6c1419adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:33.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.097+0000 7f6c1b800640 1 --2- 172.21.15.72:0/2811662932 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6c14104b90 0x7f6c1419adc0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f6c08031500 tx=0x7f6c08003c10 comp rx=0 tx=0).stop 2024-04-26T23:43:33.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.097+0000 7f6c1b800640 1 -- 172.21.15.72:0/2811662932 shutdown_connections 2024-04-26T23:43:33.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.097+0000 7f6c1b800640 1 --2- 172.21.15.72:0/2811662932 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6bfc0a3bb0 0x7f6bfc0a6070 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:33.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.097+0000 7f6c1b800640 1 --2- 172.21.15.72:0/2811662932 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6c14106740 0x7f6c141a2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:33.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.097+0000 7f6c1b800640 1 --2- 172.21.15.72:0/2811662932 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c14105d80 0x7f6c1419b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:33.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.097+0000 7f6c1b800640 1 --2- 172.21.15.72:0/2811662932 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6c14104b90 0x7f6c1419adc0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:33.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.097+0000 7f6c1b800640 1 -- 172.21.15.72:0/2811662932 >> 172.21.15.72:0/2811662932 conn(0x7f6c14100120 msgr2=0x7f6c14109100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:33.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.097+0000 7f6c1b800640 1 -- 172.21.15.72:0/2811662932 shutdown_connections 2024-04-26T23:43:33.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.097+0000 7f6c1b800640 1 -- 172.21.15.72:0/2811662932 wait complete. 2024-04-26T23:43:33.121 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:43:33.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.316+0000 7f205ee00640 1 -- 172.21.15.72:0/2080893879 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2058108f90 msgr2=0x7f20581093b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:33.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.316+0000 7f205ee00640 1 --2- 172.21.15.72:0/2080893879 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2058108f90 0x7f20581093b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2048009860 tx=0x7f204802a340 comp rx=0 tx=0).stop 2024-04-26T23:43:33.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.316+0000 7f205ee00640 1 -- 172.21.15.72:0/2080893879 shutdown_connections 2024-04-26T23:43:33.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.316+0000 7f205ee00640 1 --2- 172.21.15.72:0/2080893879 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f205810ab40 0x7f20581113d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:33.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.316+0000 7f205ee00640 1 --2- 172.21.15.72:0/2080893879 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f205810a180 0x7f205810a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:33.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.316+0000 7f205ee00640 1 --2- 172.21.15.72:0/2080893879 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2058108f90 0x7f20581093b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:33.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.316+0000 7f205ee00640 1 -- 172.21.15.72:0/2080893879 >> 172.21.15.72:0/2080893879 conn(0x7f2058077f40 msgr2=0x7f205807a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:33.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.317+0000 7f205ee00640 1 -- 172.21.15.72:0/2080893879 shutdown_connections 2024-04-26T23:43:33.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.317+0000 7f205ee00640 1 -- 172.21.15.72:0/2080893879 wait complete. 2024-04-26T23:43:33.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.318+0000 7f205ee00640 1 Processor -- start 2024-04-26T23:43:33.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.318+0000 7f205ee00640 1 -- start start 2024-04-26T23:43:33.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.318+0000 7f205ee00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2058108f90 0x7f205819f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:33.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.318+0000 7f205ee00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f205810a180 0x7f205819f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:33.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.318+0000 7f205ee00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f205810ab40 0x7f20581a6760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:33.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.318+0000 7f205ee00640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f2058114ad0 con 0x7f205810a180 2024-04-26T23:43:33.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.318+0000 7f205ee00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2058114950 con 0x7f2058108f90 2024-04-26T23:43:33.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.318+0000 7f205ee00640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f2058114c50 con 0x7f205810ab40 2024-04-26T23:43:33.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.318+0000 7f204f400640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f205810a180 0x7f205819f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:33.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.318+0000 7f2057e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2058108f90 0x7f205819f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:33.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.318+0000 7f2057e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2058108f90 0x7f205819f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:57970/0 (socket says 172.21.15.72:57970) 2024-04-26T23:43:33.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.318+0000 7f204f400640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f205810a180 0x7f205819f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42544/0 (socket says 172.21.15.72:42544) 2024-04-26T23:43:33.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.318+0000 7f2057e00640 1 -- 172.21.15.72:0/1673951662 learned_addr learned my addr 172.21.15.72:0/1673951662 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:43:33.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.319+0000 7f205ce00640 1 --2- 172.21.15.72:0/1673951662 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f205810ab40 0x7f20581a6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:33.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.319+0000 7f2057e00640 1 -- 172.21.15.72:0/1673951662 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f205810ab40 msgr2=0x7f20581a6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:33.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.319+0000 7f2057e00640 1 --2- 172.21.15.72:0/1673951662 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f205810ab40 0x7f20581a6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:33.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.319+0000 7f2057e00640 1 -- 172.21.15.72:0/1673951662 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f205810a180 msgr2=0x7f205819f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:33.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.319+0000 7f2057e00640 1 --2- 172.21.15.72:0/1673951662 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f205810a180 0x7f205819f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:33.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.319+0000 7f2057e00640 1 -- 172.21.15.72:0/1673951662 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20581a6dd0 con 0x7f2058108f90 2024-04-26T23:43:33.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.319+0000 7f205ce00640 1 --2- 172.21.15.72:0/1673951662 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f205810ab40 0x7f20581a6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:43:33.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.319+0000 7f2057e00640 1 --2- 172.21.15.72:0/1673951662 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2058108f90 0x7f205819f1b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2048005940 tx=0x7f2048032dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:33.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.319+0000 7f2055600640 1 -- 172.21.15.72:0/1673951662 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2048040020 con 0x7f2058108f90 2024-04-26T23:43:33.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.319+0000 7f2055600640 1 -- 172.21.15.72:0/1673951662 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2048004040 con 0x7f2058108f90 2024-04-26T23:43:33.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.320+0000 7f205ee00640 1 -- 172.21.15.72:0/1673951662 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20581a7060 con 0x7f2058108f90 2024-04-26T23:43:33.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.320+0000 7f2055600640 1 -- 172.21.15.72:0/1673951662 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2048037040 con 0x7f2058108f90 2024-04-26T23:43:33.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.320+0000 7f205ee00640 1 -- 172.21.15.72:0/1673951662 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20581a7550 con 0x7f2058108f90 2024-04-26T23:43:33.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.321+0000 7f205ee00640 1 -- 172.21.15.72:0/1673951662 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2058110380 con 0x7f2058108f90 2024-04-26T23:43:33.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.322+0000 7f2055600640 1 -- 172.21.15.72:0/1673951662 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f2048043020 con 0x7f2058108f90 2024-04-26T23:43:33.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.324+0000 7f2055600640 1 --2- 172.21.15.72:0/1673951662 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f203c0a3c40 0x7f203c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:33.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.324+0000 7f2055600640 1 -- 172.21.15.72:0/1673951662 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f20480f0570 con 0x7f2058108f90 2024-04-26T23:43:33.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.324+0000 7f204f400640 1 --2- 172.21.15.72:0/1673951662 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f203c0a3c40 0x7f203c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:33.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.325+0000 7f204f400640 1 --2- 172.21.15.72:0/1673951662 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f203c0a3c40 0x7f203c0a6100 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f20581a06c0 tx=0x7f2040005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:33.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.328+0000 7f2055600640 1 -- 172.21.15.72:0/1673951662 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f204804a020 con 0x7f2058108f90 2024-04-26T23:43:33.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.531+0000 7f205ee00640 1 -- 172.21.15.72:0/1673951662 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2058063530 con 0x7f2058108f90 2024-04-26T23:43:33.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.534+0000 7f2055600640 1 -- 172.21.15.72:0/1673951662 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24081 (secure 0 0 0) 0x7f20480bcc60 con 0x7f2058108f90 2024-04-26T23:43:33.538 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:43:33.538 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"fsid":"3c8453ae-0425-11ef-bc93-c7b262605968","created":"2024-04-26T23:33:48.088300+0000","modified":"2024-04-26T23:43:07.990780+0000","last_up_change":"2024-04-26T23:42:04.482891+0000","last_in_change":"2024-04-26T23:41:44.784415+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:37:44.428727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:43:00.431829+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:43:01.872977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:43:05.512050+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e20c9ebf-56f6-47c1-a11b-e1e615af042b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6803","nonce":1639298875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6805","nonce":1639298875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6809","nonce":1639298875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6807","nonce":1639298875}]},"public_addr":"172.21.15.72:6803/1639298875","cluster_addr":"172.21.15.72:6805/1639298875","heartbeat_back_addr":"172.21.15.72:6809/1639298875","heartbeat_front_addr":"172.21.15.72:6807/1639298875","state":["exists","up"]},{"osd":1,"uuid":"e6cac469-780c-4d66-b560-b686072371fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6801","nonce":4185342193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6803","nonce":4185342193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6807","nonce":4185342193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6805","nonce":4185342193}]},"public_addr":"172.21.15.76:6801/4185342193","cluster_addr":"172.21.15.76:6803/4185342193","heartbeat_back_addr":"172.21.15.76:6807/4185342193","heartbeat_front_addr":"172.21.15.76:6805/4185342193","state":["exists","up"]},{"osd":2,"uuid":"838a49fc-18de-446c-8336-67b3e05da231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6801","nonce":3033860799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6803","nonce":3033860799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6807","nonce":3033860799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6805","nonce":3033860799}]},"public_addr":"172.21.15.136:6801/3033860799","cluster_addr":"172.21.15.136:6803/3033860799","heartbeat_back_addr":"172.21.15.136:6807/3033860799","heartbeat_front_addr":"172.21.15.136:6805/3033860799","state":["exists","up"]},{"osd":3,"uuid":"e6da89be-5ba7-48a4-9306-f75498660e0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6811","nonce":2071107815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6813","nonce":2071107815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6817","nonce":2071107815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6815","nonce":2071107815}]},"public_addr":"172.21.15.72:6811/2071107815","cluster_addr":"172.21.15.72:6813/2071107815","heartbeat_back_addr":"172.21.15.72:6817/2071107815","heartbeat_front_addr":"172.21.15.72:6815/2071107815","state":["exists","up"]},{"osd":4,"uuid":"0b17c491-d798-488f-892d-521ef2fbc51e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6809","nonce":147449238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6811","nonce":147449238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6815","nonce":147449238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6813","nonce":147449238}]},"public_addr":"172.21.15.76:6809/147449238","cluster_addr":"172.21.15.76:6811/147449238","heartbeat_back_addr":"172.21.15.76:6815/147449238","heartbeat_front_addr":"172.21.15.76:6813/147449238","state":["exists","up"]},{"osd":5,"uuid":"c7e887b3-ab38-4608-9bfd-1ed49956e8af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6809","nonce":2320149212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6811","nonce":2320149212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6815","nonce":2320149212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6813","nonce":2320149212}]},"public_addr":"172.21.15.136:6809/2320149212","cluster_addr":"172.21.15.136:6811/2320149212","heartbeat_back_addr":"172.21.15.136:6815/2320149212","heartbeat_front_addr":"172.21.15.136:6813/2320149212","state":["exists","up"]},{"osd":6,"uuid":"f0bd604b-0f57-4a38-9295-eb200264f1ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6819","nonce":389499207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6821","nonce":389499207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6825","nonce":389499207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6823","nonce":389499207}]},"public_addr":"172.21.15.72:6819/389499207","cluster_addr":"172.21.15.72:6821/389499207","heartbeat_back_addr":"172.21.15.72:6825/389499207","heartbeat_front_addr":"172.21.15.72:6823/389499207","state":["exists","up"]},{"osd":7,"uuid":"1879d4e4-038e-4cd6-a3ec-3a940bcc3797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6817","nonce":375888575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6819","nonce":375888575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6823","nonce":375888575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6821","nonce":375888575}]},"public_addr":"172.21.15.76:6817/375888575","cluster_addr":"172.21.15.76:6819/375888575","heartbeat_back_addr":"172.21.15.76:6823/375888575","heartbeat_front_addr":"172.21.15.76:6821/375888575","state":["exists","up"]},{"osd":8,"uuid":"6f404062-97e7-4ccb-a582-76dc1ffae548","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6817","nonce":3446774052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6819","nonce":3446774052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6823","nonce":3446774052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6821","nonce":3446774052}]},"public_addr":"172.21.15.136:6817/3446774052","cluster_addr":"172.21.15.136:6819/3446774052","heartbeat_back_addr":"172.21.15.136:6823/3446774052","heartbeat_front_addr":"172.21.15.136:6821/3446774052","state":["exists","up"]},{"osd":9,"uuid":"2989ca68-3164-4667-9bcf-230dcbb7b8f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6827","nonce":3423789152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6829","nonce":3423789152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6832","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6833","nonce":3423789152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6831","nonce":3423789152}]},"public_addr":"172.21.15.72:6827/3423789152","cluster_addr":"172.21.15.72:6829/3423789152","heartbeat_back_addr":"172.21.15.72:6833/3423789152","heartbeat_front_addr":"172.21.15.72:6831/3423789152","state":["exists","up"]},{"osd":10,"uuid":"9ec6d46a-0913-4208-97b5-167188ee8cb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6825","nonce":1120991574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6827","nonce":1120991574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6831","nonce":1120991574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6829","nonce":1120991574}]},"public_addr":"172.21.15.76:6825/1120991574","cluster_addr":"172.21.15.76:6827/1120991574","heartbeat_back_addr":"172.21.15.76:6831/1120991574","heartbeat_front_addr":"172.21.15.76:6829/1120991574","state":["exists","up"]},{"osd":11,"uuid":"bdd0774f-6e58-4918-b4cc-afd4f1118d3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6825","nonce":1135995193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6827","nonce":1135995193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6831","nonce":1135995193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6829","nonce":1135995193}]},"public_addr":"172.21.15.136:6825/1135995193","cluster_addr":"172.21.15.136:6827/1135995193","heartbeat_back_addr":"172.21.15.136:6831/1135995193","heartbeat_front_addr":"172.21.15.136:6829/1135995193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:36:52.128788+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:37:17.523654+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:37:41.634677+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:38:15.060797+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:38:41.234111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:39:06.783560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:39:40.884386+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:40:08.340465+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:40:34.344858+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:41:08.525309+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:41:36.782624+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:42:02.756888+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:0/3625071330":"2024-04-27T23:34:58.266218+0000","172.21.15.72:6801/2903979134":"2024-04-27T23:34:58.266218+0000","172.21.15.72:6800/2903979134":"2024-04-27T23:34:58.266218+0000","172.21.15.72:0/1519142576":"2024-04-27T23:34:58.266218+0000","172.21.15.72:0/3669626460":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/945623297":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/3479150073":"2024-04-27T23:34:24.233319+0000","172.21.15.72:6800/4175896290":"2024-04-27T23:34:24.233319+0000","172.21.15.72:6801/4175896290":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/3783175012":"2024-04-27T23:34:58.266218+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:43:33.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.536+0000 7f205ee00640 1 -- 172.21.15.72:0/1673951662 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f203c0a3c40 msgr2=0x7f203c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:33.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.536+0000 7f205ee00640 1 --2- 172.21.15.72:0/1673951662 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f203c0a3c40 0x7f203c0a6100 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f20581a06c0 tx=0x7f2040005f90 comp rx=0 tx=0).stop 2024-04-26T23:43:33.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.536+0000 7f205ee00640 1 -- 172.21.15.72:0/1673951662 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2058108f90 msgr2=0x7f205819f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:33.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.536+0000 7f205ee00640 1 --2- 172.21.15.72:0/1673951662 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2058108f90 0x7f205819f1b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2048005940 tx=0x7f2048032dc0 comp rx=0 tx=0).stop 2024-04-26T23:43:33.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.537+0000 7f205ee00640 1 -- 172.21.15.72:0/1673951662 shutdown_connections 2024-04-26T23:43:33.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.537+0000 7f205ee00640 1 --2- 172.21.15.72:0/1673951662 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f203c0a3c40 0x7f203c0a6100 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:33.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.537+0000 7f205ee00640 1 --2- 172.21.15.72:0/1673951662 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f205810ab40 0x7f20581a6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:33.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.537+0000 7f205ee00640 1 --2- 172.21.15.72:0/1673951662 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f205810a180 0x7f205819f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:33.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.537+0000 7f205ee00640 1 --2- 172.21.15.72:0/1673951662 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2058108f90 0x7f205819f1b0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:33.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.537+0000 7f205ee00640 1 -- 172.21.15.72:0/1673951662 >> 172.21.15.72:0/1673951662 conn(0x7f2058077f40 msgr2=0x7f205810d550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:33.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.537+0000 7f205ee00640 1 -- 172.21.15.72:0/1673951662 shutdown_connections 2024-04-26T23:43:33.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.537+0000 7f205ee00640 1 -- 172.21.15.72:0/1673951662 wait complete. 2024-04-26T23:43:33.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:33 smithi072 ceph-mon[28184]: from='client.24826 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:43:33.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:33 smithi072 ceph-mon[28184]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:43:33.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:33 smithi072 ceph-mon[28184]: pgmap v336: 129 pgs: 129 active+clean; 583 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 2.1 KiB/s wr, 6 op/s 2024-04-26T23:43:33.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:33 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/2811662932' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:43:33.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:33 smithi136 ceph-mon[33050]: from='client.24826 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:43:33.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:33 smithi136 ceph-mon[33050]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:43:33.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:33 smithi136 ceph-mon[33050]: pgmap v336: 129 pgs: 129 active+clean; 583 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 2.1 KiB/s wr, 6 op/s 2024-04-26T23:43:33.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:33 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/2811662932' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:43:33.562 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-26T23:43:33.562 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T23:43:33.679 INFO:journalctl@ceph.mds.b.smithi076.stdout:Apr 26 23:43:33 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-b[62253]: 2024-04-26T23:43:33.378+0000 7ff94af66700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-26T23:43:33.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:33 smithi076 ceph-mon[32774]: from='client.24826 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:43:33.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:33 smithi076 ceph-mon[32774]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:43:33.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:33 smithi076 ceph-mon[32774]: pgmap v336: 129 pgs: 129 active+clean; 583 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 2.1 KiB/s wr, 6 op/s 2024-04-26T23:43:33.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:33 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/2811662932' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:43:33.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.761+0000 7f4185200640 1 -- 172.21.15.72:0/2164210277 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4180078ed0 msgr2=0x7f4180079770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:33.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.761+0000 7f4185200640 1 --2- 172.21.15.72:0/2164210277 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4180078ed0 0x7f4180079770 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4178009ca0 tx=0x7f417802a8c0 comp rx=0 tx=0).stop 2024-04-26T23:43:33.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.761+0000 7f4185200640 1 -- 172.21.15.72:0/2164210277 shutdown_connections 2024-04-26T23:43:33.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.761+0000 7f4185200640 1 --2- 172.21.15.72:0/2164210277 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4180078ed0 0x7f4180079770 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:33.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.761+0000 7f4185200640 1 --2- 172.21.15.72:0/2164210277 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4180078530 0x7f4180078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:33.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.761+0000 7f4185200640 1 --2- 172.21.15.72:0/2164210277 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f41800772f0 0x7f4180077710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:33.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.761+0000 7f4185200640 1 -- 172.21.15.72:0/2164210277 >> 172.21.15.72:0/2164210277 conn(0x7f4180100120 msgr2=0x7f4180102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:33.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.761+0000 7f4185200640 1 -- 172.21.15.72:0/2164210277 shutdown_connections 2024-04-26T23:43:33.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.761+0000 7f4185200640 1 -- 172.21.15.72:0/2164210277 wait complete. 2024-04-26T23:43:33.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.762+0000 7f4185200640 1 Processor -- start 2024-04-26T23:43:33.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.762+0000 7f4185200640 1 -- start start 2024-04-26T23:43:33.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.762+0000 7f4185200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f41800772f0 0x7f418019f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:33.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.762+0000 7f4185200640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4180078530 0x7f418019f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:33.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.762+0000 7f4185200640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4180078ed0 0x7f41801a66f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:33.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.762+0000 7f4185200640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f4180114ab0 con 0x7f4180078ed0 2024-04-26T23:43:33.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.762+0000 7f4185200640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f4180114930 con 0x7f41800772f0 2024-04-26T23:43:33.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.762+0000 7f4185200640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f4180114c30 con 0x7f4180078530 2024-04-26T23:43:33.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.762+0000 7f417e800640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4180078ed0 0x7f41801a66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:33.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.762+0000 7f417d400640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4180078530 0x7f418019f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:33.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.763+0000 7f417de00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f41800772f0 0x7f418019f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:33.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.763+0000 7f417d400640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4180078530 0x7f418019f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.72:46596/0 (socket says 172.21.15.72:46596) 2024-04-26T23:43:33.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.763+0000 7f417d400640 1 -- 172.21.15.72:0/2971196406 learned_addr learned my addr 172.21.15.72:0/2971196406 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:43:33.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.763+0000 7f417d400640 1 -- 172.21.15.72:0/2971196406 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f41800772f0 msgr2=0x7f418019f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:33.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.763+0000 7f417d400640 1 --2- 172.21.15.72:0/2971196406 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f41800772f0 0x7f418019f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:33.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.763+0000 7f417d400640 1 -- 172.21.15.72:0/2971196406 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4180078ed0 msgr2=0x7f41801a66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:33.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.763+0000 7f417d400640 1 --2- 172.21.15.72:0/2971196406 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4180078ed0 0x7f41801a66f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:33.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.763+0000 7f417d400640 1 -- 172.21.15.72:0/2971196406 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41801a6d60 con 0x7f4180078530 2024-04-26T23:43:33.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.763+0000 7f417d400640 1 --2- 172.21.15.72:0/2971196406 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4180078530 0x7f418019f680 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f416c00e9f0 tx=0x7f416c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:33.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.763+0000 7f4176a00640 1 -- 172.21.15.72:0/2971196406 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f416c00cd80 con 0x7f4180078530 2024-04-26T23:43:33.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.763+0000 7f4176a00640 1 -- 172.21.15.72:0/2971196406 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f416c0108e0 con 0x7f4180078530 2024-04-26T23:43:33.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.764+0000 7f4185200640 1 -- 172.21.15.72:0/2971196406 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41801a6ff0 con 0x7f4180078530 2024-04-26T23:43:33.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.764+0000 7f4176a00640 1 -- 172.21.15.72:0/2971196406 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f416c018980 con 0x7f4180078530 2024-04-26T23:43:33.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.764+0000 7f4185200640 1 -- 172.21.15.72:0/2971196406 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41801a7510 con 0x7f4180078530 2024-04-26T23:43:33.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.765+0000 7f4185200640 1 -- 172.21.15.72:0/2971196406 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41801114e0 con 0x7f4180078530 2024-04-26T23:43:33.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.767+0000 7f4176a00640 1 -- 172.21.15.72:0/2971196406 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f416c010430 con 0x7f4180078530 2024-04-26T23:43:33.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.769+0000 7f4176a00640 1 --2- 172.21.15.72:0/2971196406 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f41640a3c40 0x7f41640a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:33.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.769+0000 7f4176a00640 1 -- 172.21.15.72:0/2971196406 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f416c0d0d30 con 0x7f4180078530 2024-04-26T23:43:33.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.769+0000 7f417de00640 1 --2- 172.21.15.72:0/2971196406 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f41640a3c40 0x7f41640a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:33.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.770+0000 7f417de00640 1 --2- 172.21.15.72:0/2971196406 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f41640a3c40 0x7f41640a6100 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4170009bd0 tx=0x7f4170005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:33.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.771+0000 7f4176a00640 1 -- 172.21.15.72:0/2971196406 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f416c09ccc0 con 0x7f4180078530 2024-04-26T23:43:33.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.969+0000 7f4185200640 1 -- 172.21.15.72:0/2971196406 --> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4180062d70 con 0x7f41640a3c40 2024-04-26T23:43:33.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.984+0000 7f4176a00640 1 -- 172.21.15.72:0/2971196406 <== mgr.14152 v2:172.21.15.72:6800/4281177709 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340568 (secure 0 0 0) 0x7f4180062d70 con 0x7f41640a3c40 2024-04-26T23:43:33.991 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:43:33.999 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-04-26T23:43:34.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.995+0000 7f4185200640 1 -- 172.21.15.72:0/2971196406 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f41640a3c40 msgr2=0x7f41640a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:34.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.995+0000 7f4185200640 1 --2- 172.21.15.72:0/2971196406 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f41640a3c40 0x7f41640a6100 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4170009bd0 tx=0x7f4170005f90 comp rx=0 tx=0).stop 2024-04-26T23:43:34.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.995+0000 7f4185200640 1 -- 172.21.15.72:0/2971196406 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4180078530 msgr2=0x7f418019f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:34.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.995+0000 7f4185200640 1 --2- 172.21.15.72:0/2971196406 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4180078530 0x7f418019f680 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f416c00e9f0 tx=0x7f416c00eeb0 comp rx=0 tx=0).stop 2024-04-26T23:43:34.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.995+0000 7f4185200640 1 -- 172.21.15.72:0/2971196406 shutdown_connections 2024-04-26T23:43:34.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.995+0000 7f4185200640 1 --2- 172.21.15.72:0/2971196406 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f41640a3c40 0x7f41640a6100 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:34.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.995+0000 7f4185200640 1 --2- 172.21.15.72:0/2971196406 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4180078ed0 0x7f41801a66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:34.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.995+0000 7f4185200640 1 --2- 172.21.15.72:0/2971196406 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4180078530 0x7f418019f680 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:34.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.995+0000 7f4185200640 1 --2- 172.21.15.72:0/2971196406 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f41800772f0 0x7f418019f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:34.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.995+0000 7f4185200640 1 -- 172.21.15.72:0/2971196406 >> 172.21.15.72:0/2971196406 conn(0x7f4180100120 msgr2=0x7f4180101c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:34.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.995+0000 7f4185200640 1 -- 172.21.15.72:0/2971196406 shutdown_connections 2024-04-26T23:43:34.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:33.995+0000 7f4185200640 1 -- 172.21.15.72:0/2971196406 wait complete. 2024-04-26T23:43:34.019 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":336,"stamp":"2024-04-26T23:43:32.416557+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597108,"num_objects":55,"num_object_clones":0,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":132,"num_read_kb":115,"num_write":297,"num_write_kb":2978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":239,"ondisk_log_size":239,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":356608,"kb_used_data":11596,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124585216,"statfs":{"total":1151940427776,"available":1151575261184,"internally_reserved":0,"allocated":11874304,"data_stored":7515862,"data_compressed":283293,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19139,"internal_metadata":353154365},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3881,"num_objects":31,"num_object_clones":0,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":70,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":22,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004830"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.417345+0000","last_change":"2024-04-26T23:43:03.301992+0000","last_active":"2024-04-26T23:43:08.417345+0000","last_peered":"2024-04-26T23:43:08.417345+0000","last_clean":"2024-04-26T23:43:08.417345+0000","last_became_active":"2024-04-26T23:43:03.301498+0000","last_became_peered":"2024-04-26T23:43:03.301498+0000","last_unstale":"2024-04-26T23:43:08.417345+0000","last_undegraded":"2024-04-26T23:43:08.417345+0000","last_fullsized":"2024-04-26T23:43:08.417345+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:40:53.567805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.165116+0000","last_change":"2024-04-26T23:43:03.284964+0000","last_active":"2024-04-26T23:43:08.165116+0000","last_peered":"2024-04-26T23:43:08.165116+0000","last_clean":"2024-04-26T23:43:08.165116+0000","last_became_active":"2024-04-26T23:43:03.284185+0000","last_became_peered":"2024-04-26T23:43:03.284185+0000","last_unstale":"2024-04-26T23:43:08.165116+0000","last_undegraded":"2024-04-26T23:43:08.165116+0000","last_fullsized":"2024-04-26T23:43:08.165116+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:55:35.325775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.165383+0000","last_change":"2024-04-26T23:43:03.300333+0000","last_active":"2024-04-26T23:43:08.165383+0000","last_peered":"2024-04-26T23:43:08.165383+0000","last_clean":"2024-04-26T23:43:08.165383+0000","last_became_active":"2024-04-26T23:43:03.295779+0000","last_became_peered":"2024-04-26T23:43:03.295779+0000","last_unstale":"2024-04-26T23:43:08.165383+0000","last_undegraded":"2024-04-26T23:43:08.165383+0000","last_fullsized":"2024-04-26T23:43:08.165383+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:45:06.598862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.416873+0000","last_change":"2024-04-26T23:43:03.290211+0000","last_active":"2024-04-26T23:43:08.416873+0000","last_peered":"2024-04-26T23:43:08.416873+0000","last_clean":"2024-04-26T23:43:08.416873+0000","last_became_active":"2024-04-26T23:43:03.289853+0000","last_became_peered":"2024-04-26T23:43:03.289853+0000","last_unstale":"2024-04-26T23:43:08.416873+0000","last_undegraded":"2024-04-26T23:43:08.416873+0000","last_fullsized":"2024-04-26T23:43:08.416873+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:14:45.032300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.416540+0000","last_change":"2024-04-26T23:43:03.309667+0000","last_active":"2024-04-26T23:43:08.416540+0000","last_peered":"2024-04-26T23:43:08.416540+0000","last_clean":"2024-04-26T23:43:08.416540+0000","last_became_active":"2024-04-26T23:43:03.309305+0000","last_became_peered":"2024-04-26T23:43:03.309305+0000","last_unstale":"2024-04-26T23:43:08.416540+0000","last_undegraded":"2024-04-26T23:43:08.416540+0000","last_fullsized":"2024-04-26T23:43:08.416540+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:31:56.739866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.416669+0000","last_change":"2024-04-26T23:43:03.292107+0000","last_active":"2024-04-26T23:43:08.416669+0000","last_peered":"2024-04-26T23:43:08.416669+0000","last_clean":"2024-04-26T23:43:08.416669+0000","last_became_active":"2024-04-26T23:43:03.291760+0000","last_became_peered":"2024-04-26T23:43:03.291760+0000","last_unstale":"2024-04-26T23:43:08.416669+0000","last_undegraded":"2024-04-26T23:43:08.416669+0000","last_fullsized":"2024-04-26T23:43:08.416669+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:31:54.654379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.162471+0000","last_change":"2024-04-26T23:43:03.294923+0000","last_active":"2024-04-26T23:43:08.162471+0000","last_peered":"2024-04-26T23:43:08.162471+0000","last_clean":"2024-04-26T23:43:08.162471+0000","last_became_active":"2024-04-26T23:43:03.294523+0000","last_became_peered":"2024-04-26T23:43:03.294523+0000","last_unstale":"2024-04-26T23:43:08.162471+0000","last_undegraded":"2024-04-26T23:43:08.162471+0000","last_fullsized":"2024-04-26T23:43:08.162471+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:36:19.419773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.419625+0000","last_change":"2024-04-26T23:43:03.301335+0000","last_active":"2024-04-26T23:43:08.419625+0000","last_peered":"2024-04-26T23:43:08.419625+0000","last_clean":"2024-04-26T23:43:08.419625+0000","last_became_active":"2024-04-26T23:43:03.300918+0000","last_became_peered":"2024-04-26T23:43:03.300918+0000","last_unstale":"2024-04-26T23:43:08.419625+0000","last_undegraded":"2024-04-26T23:43:08.419625+0000","last_fullsized":"2024-04-26T23:43:08.419625+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:15:05.719452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.238112+0000","last_change":"2024-04-26T23:43:03.285114+0000","last_active":"2024-04-26T23:43:08.238112+0000","last_peered":"2024-04-26T23:43:08.238112+0000","last_clean":"2024-04-26T23:43:08.238112+0000","last_became_active":"2024-04-26T23:43:03.284744+0000","last_became_peered":"2024-04-26T23:43:03.284744+0000","last_unstale":"2024-04-26T23:43:08.238112+0000","last_undegraded":"2024-04-26T23:43:08.238112+0000","last_fullsized":"2024-04-26T23:43:08.238112+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:48:52.936951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.417313+0000","last_change":"2024-04-26T23:43:03.287643+0000","last_active":"2024-04-26T23:43:08.417313+0000","last_peered":"2024-04-26T23:43:08.417313+0000","last_clean":"2024-04-26T23:43:08.417313+0000","last_became_active":"2024-04-26T23:43:03.287324+0000","last_became_peered":"2024-04-26T23:43:03.287324+0000","last_unstale":"2024-04-26T23:43:08.417313+0000","last_undegraded":"2024-04-26T23:43:08.417313+0000","last_fullsized":"2024-04-26T23:43:08.417313+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:50:14.816548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.163323+0000","last_change":"2024-04-26T23:43:03.296421+0000","last_active":"2024-04-26T23:43:08.163323+0000","last_peered":"2024-04-26T23:43:08.163323+0000","last_clean":"2024-04-26T23:43:08.163323+0000","last_became_active":"2024-04-26T23:43:03.296060+0000","last_became_peered":"2024-04-26T23:43:03.296060+0000","last_unstale":"2024-04-26T23:43:08.163323+0000","last_undegraded":"2024-04-26T23:43:08.163323+0000","last_fullsized":"2024-04-26T23:43:08.163323+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:50:44.393446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.416764+0000","last_change":"2024-04-26T23:43:03.288562+0000","last_active":"2024-04-26T23:43:08.416764+0000","last_peered":"2024-04-26T23:43:08.416764+0000","last_clean":"2024-04-26T23:43:08.416764+0000","last_became_active":"2024-04-26T23:43:03.288219+0000","last_became_peered":"2024-04-26T23:43:03.288219+0000","last_unstale":"2024-04-26T23:43:08.416764+0000","last_undegraded":"2024-04-26T23:43:08.416764+0000","last_fullsized":"2024-04-26T23:43:08.416764+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:54:32.107146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.422359+0000","last_change":"2024-04-26T23:43:03.296840+0000","last_active":"2024-04-26T23:43:08.422359+0000","last_peered":"2024-04-26T23:43:08.422359+0000","last_clean":"2024-04-26T23:43:08.422359+0000","last_became_active":"2024-04-26T23:43:03.296501+0000","last_became_peered":"2024-04-26T23:43:03.296501+0000","last_unstale":"2024-04-26T23:43:08.422359+0000","last_undegraded":"2024-04-26T23:43:08.422359+0000","last_fullsized":"2024-04-26T23:43:08.422359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:07:05.293388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.164489+0000","last_change":"2024-04-26T23:43:03.291739+0000","last_active":"2024-04-26T23:43:08.164489+0000","last_peered":"2024-04-26T23:43:08.164489+0000","last_clean":"2024-04-26T23:43:08.164489+0000","last_became_active":"2024-04-26T23:43:03.287839+0000","last_became_peered":"2024-04-26T23:43:03.287839+0000","last_unstale":"2024-04-26T23:43:08.164489+0000","last_undegraded":"2024-04-26T23:43:08.164489+0000","last_fullsized":"2024-04-26T23:43:08.164489+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:10:56.698324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.164219+0000","last_change":"2024-04-26T23:43:03.295767+0000","last_active":"2024-04-26T23:43:08.164219+0000","last_peered":"2024-04-26T23:43:08.164219+0000","last_clean":"2024-04-26T23:43:08.164219+0000","last_became_active":"2024-04-26T23:43:03.295339+0000","last_became_peered":"2024-04-26T23:43:03.295339+0000","last_unstale":"2024-04-26T23:43:08.164219+0000","last_undegraded":"2024-04-26T23:43:08.164219+0000","last_fullsized":"2024-04-26T23:43:08.164219+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:21:44.410953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.237529+0000","last_change":"2024-04-26T23:43:03.301687+0000","last_active":"2024-04-26T23:43:08.237529+0000","last_peered":"2024-04-26T23:43:08.237529+0000","last_clean":"2024-04-26T23:43:08.237529+0000","last_became_active":"2024-04-26T23:43:03.301355+0000","last_became_peered":"2024-04-26T23:43:03.301355+0000","last_unstale":"2024-04-26T23:43:08.237529+0000","last_undegraded":"2024-04-26T23:43:08.237529+0000","last_fullsized":"2024-04-26T23:43:08.237529+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:00:57.991022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.416993+0000","last_change":"2024-04-26T23:43:03.302109+0000","last_active":"2024-04-26T23:43:08.416993+0000","last_peered":"2024-04-26T23:43:08.416993+0000","last_clean":"2024-04-26T23:43:08.416993+0000","last_became_active":"2024-04-26T23:43:03.301725+0000","last_became_peered":"2024-04-26T23:43:03.301725+0000","last_unstale":"2024-04-26T23:43:08.416993+0000","last_undegraded":"2024-04-26T23:43:08.416993+0000","last_fullsized":"2024-04-26T23:43:08.416993+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:36:44.630125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.423155+0000","last_change":"2024-04-26T23:43:03.298880+0000","last_active":"2024-04-26T23:43:08.423155+0000","last_peered":"2024-04-26T23:43:08.423155+0000","last_clean":"2024-04-26T23:43:08.423155+0000","last_became_active":"2024-04-26T23:43:03.298464+0000","last_became_peered":"2024-04-26T23:43:03.298464+0000","last_unstale":"2024-04-26T23:43:08.423155+0000","last_undegraded":"2024-04-26T23:43:08.423155+0000","last_fullsized":"2024-04-26T23:43:08.423155+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:42:54.745002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.163034+0000","last_change":"2024-04-26T23:43:03.309454+0000","last_active":"2024-04-26T23:43:08.163034+0000","last_peered":"2024-04-26T23:43:08.163034+0000","last_clean":"2024-04-26T23:43:08.163034+0000","last_became_active":"2024-04-26T23:43:03.309091+0000","last_became_peered":"2024-04-26T23:43:03.309091+0000","last_unstale":"2024-04-26T23:43:08.163034+0000","last_undegraded":"2024-04-26T23:43:08.163034+0000","last_fullsized":"2024-04-26T23:43:08.163034+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:46:12.333410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.162389+0000","last_change":"2024-04-26T23:43:07.541317+0000","last_active":"2024-04-26T23:43:08.162389+0000","last_peered":"2024-04-26T23:43:08.162389+0000","last_clean":"2024-04-26T23:43:08.162389+0000","last_became_active":"2024-04-26T23:43:07.540760+0000","last_became_peered":"2024-04-26T23:43:07.540760+0000","last_unstale":"2024-04-26T23:43:08.162389+0000","last_undegraded":"2024-04-26T23:43:08.162389+0000","last_fullsized":"2024-04-26T23:43:08.162389+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:28:55.545416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:29.204260+0000","last_change":"2024-04-26T23:43:02.450956+0000","last_active":"2024-04-26T23:43:29.204260+0000","last_peered":"2024-04-26T23:43:29.204260+0000","last_clean":"2024-04-26T23:43:29.204260+0000","last_became_active":"2024-04-26T23:43:02.446733+0000","last_became_peered":"2024-04-26T23:43:02.446733+0000","last_unstale":"2024-04-26T23:43:29.204260+0000","last_undegraded":"2024-04-26T23:43:29.204260+0000","last_fullsized":"2024-04-26T23:43:29.204260+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:58:47.613575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.420791+0000","last_change":"2024-04-26T23:43:03.311681+0000","last_active":"2024-04-26T23:43:08.420791+0000","last_peered":"2024-04-26T23:43:08.420791+0000","last_clean":"2024-04-26T23:43:08.420791+0000","last_became_active":"2024-04-26T23:43:03.311087+0000","last_became_peered":"2024-04-26T23:43:03.311087+0000","last_unstale":"2024-04-26T23:43:08.420791+0000","last_undegraded":"2024-04-26T23:43:08.420791+0000","last_fullsized":"2024-04-26T23:43:08.420791+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:13:00.744062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.417017+0000","last_change":"2024-04-26T23:43:07.543547+0000","last_active":"2024-04-26T23:43:08.417017+0000","last_peered":"2024-04-26T23:43:08.417017+0000","last_clean":"2024-04-26T23:43:08.417017+0000","last_became_active":"2024-04-26T23:43:07.543073+0000","last_became_peered":"2024-04-26T23:43:07.543073+0000","last_unstale":"2024-04-26T23:43:08.417017+0000","last_undegraded":"2024-04-26T23:43:08.417017+0000","last_fullsized":"2024-04-26T23:43:08.417017+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:30:05.418875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.421015+0000","last_change":"2024-04-26T23:43:02.460459+0000","last_active":"2024-04-26T23:43:08.421015+0000","last_peered":"2024-04-26T23:43:08.421015+0000","last_clean":"2024-04-26T23:43:08.421015+0000","last_became_active":"2024-04-26T23:43:02.459652+0000","last_became_peered":"2024-04-26T23:43:02.459652+0000","last_unstale":"2024-04-26T23:43:08.421015+0000","last_undegraded":"2024-04-26T23:43:08.421015+0000","last_fullsized":"2024-04-26T23:43:08.421015+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:27:19.166748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.165811+0000","last_change":"2024-04-26T23:43:03.300078+0000","last_active":"2024-04-26T23:43:08.165811+0000","last_peered":"2024-04-26T23:43:08.165811+0000","last_clean":"2024-04-26T23:43:08.165811+0000","last_became_active":"2024-04-26T23:43:03.296319+0000","last_became_peered":"2024-04-26T23:43:03.296319+0000","last_unstale":"2024-04-26T23:43:08.165811+0000","last_undegraded":"2024-04-26T23:43:08.165811+0000","last_fullsized":"2024-04-26T23:43:08.165811+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:33:51.859539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.416408+0000","last_change":"2024-04-26T23:43:07.543494+0000","last_active":"2024-04-26T23:43:08.416408+0000","last_peered":"2024-04-26T23:43:08.416408+0000","last_clean":"2024-04-26T23:43:08.416408+0000","last_became_active":"2024-04-26T23:43:07.542806+0000","last_became_peered":"2024-04-26T23:43:07.542806+0000","last_unstale":"2024-04-26T23:43:08.416408+0000","last_undegraded":"2024-04-26T23:43:08.416408+0000","last_fullsized":"2024-04-26T23:43:08.416408+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:54:40.522615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:29.201942+0000","last_change":"2024-04-26T23:43:02.438943+0000","last_active":"2024-04-26T23:43:29.201942+0000","last_peered":"2024-04-26T23:43:29.201942+0000","last_clean":"2024-04-26T23:43:29.201942+0000","last_became_active":"2024-04-26T23:43:02.438604+0000","last_became_peered":"2024-04-26T23:43:02.438604+0000","last_unstale":"2024-04-26T23:43:29.201942+0000","last_undegraded":"2024-04-26T23:43:29.201942+0000","last_fullsized":"2024-04-26T23:43:29.201942+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:06:51.054342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.237792+0000","last_change":"2024-04-26T23:43:03.293856+0000","last_active":"2024-04-26T23:43:08.237792+0000","last_peered":"2024-04-26T23:43:08.237792+0000","last_clean":"2024-04-26T23:43:08.237792+0000","last_became_active":"2024-04-26T23:43:03.293189+0000","last_became_peered":"2024-04-26T23:43:03.293189+0000","last_unstale":"2024-04-26T23:43:08.237792+0000","last_undegraded":"2024-04-26T23:43:08.237792+0000","last_fullsized":"2024-04-26T23:43:08.237792+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:50:27.545567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.416144+0000","last_change":"2024-04-26T23:43:07.535396+0000","last_active":"2024-04-26T23:43:08.416144+0000","last_peered":"2024-04-26T23:43:08.416144+0000","last_clean":"2024-04-26T23:43:08.416144+0000","last_became_active":"2024-04-26T23:43:07.534914+0000","last_became_peered":"2024-04-26T23:43:07.534914+0000","last_unstale":"2024-04-26T23:43:08.416144+0000","last_undegraded":"2024-04-26T23:43:08.416144+0000","last_fullsized":"2024-04-26T23:43:08.416144+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:27:31.199079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.416306+0000","last_change":"2024-04-26T23:43:02.442935+0000","last_active":"2024-04-26T23:43:08.416306+0000","last_peered":"2024-04-26T23:43:08.416306+0000","last_clean":"2024-04-26T23:43:08.416306+0000","last_became_active":"2024-04-26T23:43:02.442588+0000","last_became_peered":"2024-04-26T23:43:02.442588+0000","last_unstale":"2024-04-26T23:43:08.416306+0000","last_undegraded":"2024-04-26T23:43:08.416306+0000","last_fullsized":"2024-04-26T23:43:08.416306+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:05:26.442001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.418012+0000","last_change":"2024-04-26T23:43:03.289407+0000","last_active":"2024-04-26T23:43:08.418012+0000","last_peered":"2024-04-26T23:43:08.418012+0000","last_clean":"2024-04-26T23:43:08.418012+0000","last_became_active":"2024-04-26T23:43:03.287951+0000","last_became_peered":"2024-04-26T23:43:03.287951+0000","last_unstale":"2024-04-26T23:43:08.418012+0000","last_undegraded":"2024-04-26T23:43:08.418012+0000","last_fullsized":"2024-04-26T23:43:08.418012+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:24:42.687653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.162911+0000","last_change":"2024-04-26T23:43:07.527019+0000","last_active":"2024-04-26T23:43:08.162911+0000","last_peered":"2024-04-26T23:43:08.162911+0000","last_clean":"2024-04-26T23:43:08.162911+0000","last_became_active":"2024-04-26T23:43:07.526516+0000","last_became_peered":"2024-04-26T23:43:07.526516+0000","last_unstale":"2024-04-26T23:43:08.162911+0000","last_undegraded":"2024-04-26T23:43:08.162911+0000","last_fullsized":"2024-04-26T23:43:08.162911+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:44:41.267144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:28.187431+0000","last_change":"2024-04-26T23:43:02.445159+0000","last_active":"2024-04-26T23:43:28.187431+0000","last_peered":"2024-04-26T23:43:28.187431+0000","last_clean":"2024-04-26T23:43:28.187431+0000","last_became_active":"2024-04-26T23:43:02.444781+0000","last_became_peered":"2024-04-26T23:43:02.444781+0000","last_unstale":"2024-04-26T23:43:28.187431+0000","last_undegraded":"2024-04-26T23:43:28.187431+0000","last_fullsized":"2024-04-26T23:43:28.187431+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:45:16.509790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.417133+0000","last_change":"2024-04-26T23:43:03.292717+0000","last_active":"2024-04-26T23:43:08.417133+0000","last_peered":"2024-04-26T23:43:08.417133+0000","last_clean":"2024-04-26T23:43:08.417133+0000","last_became_active":"2024-04-26T23:43:03.292355+0000","last_became_peered":"2024-04-26T23:43:03.292355+0000","last_unstale":"2024-04-26T23:43:08.417133+0000","last_undegraded":"2024-04-26T23:43:08.417133+0000","last_fullsized":"2024-04-26T23:43:08.417133+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:06:02.063165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.162988+0000","last_change":"2024-04-26T23:43:07.540758+0000","last_active":"2024-04-26T23:43:08.162988+0000","last_peered":"2024-04-26T23:43:08.162988+0000","last_clean":"2024-04-26T23:43:08.162988+0000","last_became_active":"2024-04-26T23:43:07.540305+0000","last_became_peered":"2024-04-26T23:43:07.540305+0000","last_unstale":"2024-04-26T23:43:08.162988+0000","last_undegraded":"2024-04-26T23:43:08.162988+0000","last_fullsized":"2024-04-26T23:43:08.162988+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:08:04.491544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.416884+0000","last_change":"2024-04-26T23:43:02.453650+0000","last_active":"2024-04-26T23:43:08.416884+0000","last_peered":"2024-04-26T23:43:08.416884+0000","last_clean":"2024-04-26T23:43:08.416884+0000","last_became_active":"2024-04-26T23:43:02.453319+0000","last_became_peered":"2024-04-26T23:43:02.453319+0000","last_unstale":"2024-04-26T23:43:08.416884+0000","last_undegraded":"2024-04-26T23:43:08.416884+0000","last_fullsized":"2024-04-26T23:43:08.416884+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:23:27.570603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.417670+0000","last_change":"2024-04-26T23:43:03.300758+0000","last_active":"2024-04-26T23:43:08.417670+0000","last_peered":"2024-04-26T23:43:08.417670+0000","last_clean":"2024-04-26T23:43:08.417670+0000","last_became_active":"2024-04-26T23:43:03.300193+0000","last_became_peered":"2024-04-26T23:43:03.300193+0000","last_unstale":"2024-04-26T23:43:08.417670+0000","last_undegraded":"2024-04-26T23:43:08.417670+0000","last_fullsized":"2024-04-26T23:43:08.417670+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:37:43.289361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.417265+0000","last_change":"2024-04-26T23:43:07.546026+0000","last_active":"2024-04-26T23:43:08.417265+0000","last_peered":"2024-04-26T23:43:08.417265+0000","last_clean":"2024-04-26T23:43:08.417265+0000","last_became_active":"2024-04-26T23:43:07.545668+0000","last_became_peered":"2024-04-26T23:43:07.545668+0000","last_unstale":"2024-04-26T23:43:08.417265+0000","last_undegraded":"2024-04-26T23:43:08.417265+0000","last_fullsized":"2024-04-26T23:43:08.417265+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:54:20.580267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.421162+0000","last_change":"2024-04-26T23:43:02.454937+0000","last_active":"2024-04-26T23:43:08.421162+0000","last_peered":"2024-04-26T23:43:08.421162+0000","last_clean":"2024-04-26T23:43:08.421162+0000","last_became_active":"2024-04-26T23:43:02.454585+0000","last_became_peered":"2024-04-26T23:43:02.454585+0000","last_unstale":"2024-04-26T23:43:08.421162+0000","last_undegraded":"2024-04-26T23:43:08.421162+0000","last_fullsized":"2024-04-26T23:43:08.421162+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:55:54.999411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.419333+0000","last_change":"2024-04-26T23:43:03.312622+0000","last_active":"2024-04-26T23:43:08.419333+0000","last_peered":"2024-04-26T23:43:08.419333+0000","last_clean":"2024-04-26T23:43:08.419333+0000","last_became_active":"2024-04-26T23:43:03.310774+0000","last_became_peered":"2024-04-26T23:43:03.310774+0000","last_unstale":"2024-04-26T23:43:08.419333+0000","last_undegraded":"2024-04-26T23:43:08.419333+0000","last_fullsized":"2024-04-26T23:43:08.419333+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:31:11.627552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.165398+0000","last_change":"2024-04-26T23:43:07.535682+0000","last_active":"2024-04-26T23:43:08.165398+0000","last_peered":"2024-04-26T23:43:08.165398+0000","last_clean":"2024-04-26T23:43:08.165398+0000","last_became_active":"2024-04-26T23:43:07.535221+0000","last_became_peered":"2024-04-26T23:43:07.535221+0000","last_unstale":"2024-04-26T23:43:08.165398+0000","last_undegraded":"2024-04-26T23:43:08.165398+0000","last_fullsized":"2024-04-26T23:43:08.165398+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:33:54.105329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.417216+0000","last_change":"2024-04-26T23:43:02.443541+0000","last_active":"2024-04-26T23:43:08.417216+0000","last_peered":"2024-04-26T23:43:08.417216+0000","last_clean":"2024-04-26T23:43:08.417216+0000","last_became_active":"2024-04-26T23:43:02.442948+0000","last_became_peered":"2024-04-26T23:43:02.442948+0000","last_unstale":"2024-04-26T23:43:08.417216+0000","last_undegraded":"2024-04-26T23:43:08.417216+0000","last_fullsized":"2024-04-26T23:43:08.417216+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:58:44.595575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.422169+0000","last_change":"2024-04-26T23:43:03.311029+0000","last_active":"2024-04-26T23:43:08.422169+0000","last_peered":"2024-04-26T23:43:08.422169+0000","last_clean":"2024-04-26T23:43:08.422169+0000","last_became_active":"2024-04-26T23:43:03.310722+0000","last_became_peered":"2024-04-26T23:43:03.310722+0000","last_unstale":"2024-04-26T23:43:08.422169+0000","last_undegraded":"2024-04-26T23:43:08.422169+0000","last_fullsized":"2024-04-26T23:43:08.422169+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:07:54.449492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.421647+0000","last_change":"2024-04-26T23:43:07.537391+0000","last_active":"2024-04-26T23:43:08.421647+0000","last_peered":"2024-04-26T23:43:08.421647+0000","last_clean":"2024-04-26T23:43:08.421647+0000","last_became_active":"2024-04-26T23:43:07.536216+0000","last_became_peered":"2024-04-26T23:43:07.536216+0000","last_unstale":"2024-04-26T23:43:08.421647+0000","last_undegraded":"2024-04-26T23:43:08.421647+0000","last_fullsized":"2024-04-26T23:43:08.421647+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:45:55.326283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:28.185888+0000","last_change":"2024-04-26T23:43:02.451929+0000","last_active":"2024-04-26T23:43:28.185888+0000","last_peered":"2024-04-26T23:43:28.185888+0000","last_clean":"2024-04-26T23:43:28.185888+0000","last_became_active":"2024-04-26T23:43:02.451700+0000","last_became_peered":"2024-04-26T23:43:02.451700+0000","last_unstale":"2024-04-26T23:43:28.185888+0000","last_undegraded":"2024-04-26T23:43:28.185888+0000","last_fullsized":"2024-04-26T23:43:28.185888+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:21:31.279681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.238028+0000","last_change":"2024-04-26T23:43:03.295462+0000","last_active":"2024-04-26T23:43:08.238028+0000","last_peered":"2024-04-26T23:43:08.238028+0000","last_clean":"2024-04-26T23:43:08.238028+0000","last_became_active":"2024-04-26T23:43:03.295044+0000","last_became_peered":"2024-04-26T23:43:03.295044+0000","last_unstale":"2024-04-26T23:43:08.238028+0000","last_undegraded":"2024-04-26T23:43:08.238028+0000","last_fullsized":"2024-04-26T23:43:08.238028+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:56:19.445166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.419113+0000","last_change":"2024-04-26T23:43:07.539397+0000","last_active":"2024-04-26T23:43:08.419113+0000","last_peered":"2024-04-26T23:43:08.419113+0000","last_clean":"2024-04-26T23:43:08.419113+0000","last_became_active":"2024-04-26T23:43:07.533081+0000","last_became_peered":"2024-04-26T23:43:07.533081+0000","last_unstale":"2024-04-26T23:43:08.419113+0000","last_undegraded":"2024-04-26T23:43:08.419113+0000","last_fullsized":"2024-04-26T23:43:08.419113+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:24:39.693721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:28.186458+0000","last_change":"2024-04-26T23:43:02.439919+0000","last_active":"2024-04-26T23:43:28.186458+0000","last_peered":"2024-04-26T23:43:28.186458+0000","last_clean":"2024-04-26T23:43:28.186458+0000","last_became_active":"2024-04-26T23:43:02.439561+0000","last_became_peered":"2024-04-26T23:43:02.439561+0000","last_unstale":"2024-04-26T23:43:28.186458+0000","last_undegraded":"2024-04-26T23:43:28.186458+0000","last_fullsized":"2024-04-26T23:43:28.186458+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:30:22.905464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.165717+0000","last_change":"2024-04-26T23:43:03.294088+0000","last_active":"2024-04-26T23:43:08.165717+0000","last_peered":"2024-04-26T23:43:08.165717+0000","last_clean":"2024-04-26T23:43:08.165717+0000","last_became_active":"2024-04-26T23:43:03.293726+0000","last_became_peered":"2024-04-26T23:43:03.293726+0000","last_unstale":"2024-04-26T23:43:08.165717+0000","last_undegraded":"2024-04-26T23:43:08.165717+0000","last_fullsized":"2024-04-26T23:43:08.165717+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:19:42.575193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.416554+0000","last_change":"2024-04-26T23:43:07.543056+0000","last_active":"2024-04-26T23:43:08.416554+0000","last_peered":"2024-04-26T23:43:08.416554+0000","last_clean":"2024-04-26T23:43:08.416554+0000","last_became_active":"2024-04-26T23:43:07.542324+0000","last_became_peered":"2024-04-26T23:43:07.542324+0000","last_unstale":"2024-04-26T23:43:08.416554+0000","last_undegraded":"2024-04-26T23:43:08.416554+0000","last_fullsized":"2024-04-26T23:43:08.416554+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:23:46.415932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.417952+0000","last_change":"2024-04-26T23:43:02.454135+0000","last_active":"2024-04-26T23:43:08.417952+0000","last_peered":"2024-04-26T23:43:08.417952+0000","last_clean":"2024-04-26T23:43:08.417952+0000","last_became_active":"2024-04-26T23:43:02.453798+0000","last_became_peered":"2024-04-26T23:43:02.453798+0000","last_unstale":"2024-04-26T23:43:08.417952+0000","last_undegraded":"2024-04-26T23:43:08.417952+0000","last_fullsized":"2024-04-26T23:43:08.417952+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:16:25.286569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.237924+0000","last_change":"2024-04-26T23:43:03.293520+0000","last_active":"2024-04-26T23:43:08.237924+0000","last_peered":"2024-04-26T23:43:08.237924+0000","last_clean":"2024-04-26T23:43:08.237924+0000","last_became_active":"2024-04-26T23:43:03.292537+0000","last_became_peered":"2024-04-26T23:43:03.292537+0000","last_unstale":"2024-04-26T23:43:08.237924+0000","last_undegraded":"2024-04-26T23:43:08.237924+0000","last_fullsized":"2024-04-26T23:43:08.237924+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:19:28.808217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.419616+0000","last_change":"2024-04-26T23:43:07.528116+0000","last_active":"2024-04-26T23:43:08.419616+0000","last_peered":"2024-04-26T23:43:08.419616+0000","last_clean":"2024-04-26T23:43:08.419616+0000","last_became_active":"2024-04-26T23:43:07.527766+0000","last_became_peered":"2024-04-26T23:43:07.527766+0000","last_unstale":"2024-04-26T23:43:08.419616+0000","last_undegraded":"2024-04-26T23:43:08.419616+0000","last_fullsized":"2024-04-26T23:43:08.419616+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:39:23.949964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'12","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:28.184773+0000","last_change":"2024-04-26T23:43:02.437454+0000","last_active":"2024-04-26T23:43:28.184773+0000","last_peered":"2024-04-26T23:43:28.184773+0000","last_clean":"2024-04-26T23:43:28.184773+0000","last_became_active":"2024-04-26T23:43:02.437142+0000","last_became_peered":"2024-04-26T23:43:02.437142+0000","last_unstale":"2024-04-26T23:43:28.184773+0000","last_undegraded":"2024-04-26T23:43:28.184773+0000","last_fullsized":"2024-04-26T23:43:28.184773+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:06:37.751066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.417458+0000","last_change":"2024-04-26T23:43:03.287831+0000","last_active":"2024-04-26T23:43:08.417458+0000","last_peered":"2024-04-26T23:43:08.417458+0000","last_clean":"2024-04-26T23:43:08.417458+0000","last_became_active":"2024-04-26T23:43:03.287479+0000","last_became_peered":"2024-04-26T23:43:03.287479+0000","last_unstale":"2024-04-26T23:43:08.417458+0000","last_undegraded":"2024-04-26T23:43:08.417458+0000","last_fullsized":"2024-04-26T23:43:08.417458+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:39:41.726893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.418327+0000","last_change":"2024-04-26T23:43:07.543985+0000","last_active":"2024-04-26T23:43:08.418327+0000","last_peered":"2024-04-26T23:43:08.418327+0000","last_clean":"2024-04-26T23:43:08.418327+0000","last_became_active":"2024-04-26T23:43:07.543327+0000","last_became_peered":"2024-04-26T23:43:07.543327+0000","last_unstale":"2024-04-26T23:43:08.418327+0000","last_undegraded":"2024-04-26T23:43:08.418327+0000","last_fullsized":"2024-04-26T23:43:08.418327+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:24:10.577822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:29.202829+0000","last_change":"2024-04-26T23:43:02.447627+0000","last_active":"2024-04-26T23:43:29.202829+0000","last_peered":"2024-04-26T23:43:29.202829+0000","last_clean":"2024-04-26T23:43:29.202829+0000","last_became_active":"2024-04-26T23:43:02.447325+0000","last_became_peered":"2024-04-26T23:43:02.447325+0000","last_unstale":"2024-04-26T23:43:29.202829+0000","last_undegraded":"2024-04-26T23:43:29.202829+0000","last_fullsized":"2024-04-26T23:43:29.202829+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:19:03.757202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.421374+0000","last_change":"2024-04-26T23:43:03.313388+0000","last_active":"2024-04-26T23:43:08.421374+0000","last_peered":"2024-04-26T23:43:08.421374+0000","last_clean":"2024-04-26T23:43:08.421374+0000","last_became_active":"2024-04-26T23:43:03.312899+0000","last_became_peered":"2024-04-26T23:43:03.312899+0000","last_unstale":"2024-04-26T23:43:08.421374+0000","last_undegraded":"2024-04-26T23:43:08.421374+0000","last_fullsized":"2024-04-26T23:43:08.421374+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:58:37.126485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.416704+0000","last_change":"2024-04-26T23:43:07.536706+0000","last_active":"2024-04-26T23:43:08.416704+0000","last_peered":"2024-04-26T23:43:08.416704+0000","last_clean":"2024-04-26T23:43:08.416704+0000","last_became_active":"2024-04-26T23:43:07.536316+0000","last_became_peered":"2024-04-26T23:43:07.536316+0000","last_unstale":"2024-04-26T23:43:08.416704+0000","last_undegraded":"2024-04-26T23:43:08.416704+0000","last_fullsized":"2024-04-26T23:43:08.416704+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:36:46.397725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:28.188765+0000","last_change":"2024-04-26T23:43:02.448647+0000","last_active":"2024-04-26T23:43:28.188765+0000","last_peered":"2024-04-26T23:43:28.188765+0000","last_clean":"2024-04-26T23:43:28.188765+0000","last_became_active":"2024-04-26T23:43:02.448392+0000","last_became_peered":"2024-04-26T23:43:02.448392+0000","last_unstale":"2024-04-26T23:43:28.188765+0000","last_undegraded":"2024-04-26T23:43:28.188765+0000","last_fullsized":"2024-04-26T23:43:28.188765+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:19:01.367361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.162983+0000","last_change":"2024-04-26T23:43:03.302980+0000","last_active":"2024-04-26T23:43:08.162983+0000","last_peered":"2024-04-26T23:43:08.162983+0000","last_clean":"2024-04-26T23:43:08.162983+0000","last_became_active":"2024-04-26T23:43:03.302589+0000","last_became_peered":"2024-04-26T23:43:03.302589+0000","last_unstale":"2024-04-26T23:43:08.162983+0000","last_undegraded":"2024-04-26T23:43:08.162983+0000","last_fullsized":"2024-04-26T23:43:08.162983+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:57:18.641063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.423939+0000","last_change":"2024-04-26T23:43:07.547842+0000","last_active":"2024-04-26T23:43:08.423939+0000","last_peered":"2024-04-26T23:43:08.423939+0000","last_clean":"2024-04-26T23:43:08.423939+0000","last_became_active":"2024-04-26T23:43:07.541774+0000","last_became_peered":"2024-04-26T23:43:07.541774+0000","last_unstale":"2024-04-26T23:43:08.423939+0000","last_undegraded":"2024-04-26T23:43:08.423939+0000","last_fullsized":"2024-04-26T23:43:08.423939+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:25:49.999465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.416117+0000","last_change":"2024-04-26T23:43:02.437210+0000","last_active":"2024-04-26T23:43:08.416117+0000","last_peered":"2024-04-26T23:43:08.416117+0000","last_clean":"2024-04-26T23:43:08.416117+0000","last_became_active":"2024-04-26T23:43:02.436886+0000","last_became_peered":"2024-04-26T23:43:02.436886+0000","last_unstale":"2024-04-26T23:43:08.416117+0000","last_undegraded":"2024-04-26T23:43:08.416117+0000","last_fullsized":"2024-04-26T23:43:08.416117+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:07:03.663757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.162627+0000","last_change":"2024-04-26T23:43:03.293770+0000","last_active":"2024-04-26T23:43:08.162627+0000","last_peered":"2024-04-26T23:43:08.162627+0000","last_clean":"2024-04-26T23:43:08.162627+0000","last_became_active":"2024-04-26T23:43:03.293297+0000","last_became_peered":"2024-04-26T23:43:03.293297+0000","last_unstale":"2024-04-26T23:43:08.162627+0000","last_undegraded":"2024-04-26T23:43:08.162627+0000","last_fullsized":"2024-04-26T23:43:08.162627+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:01:42.339059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.423658+0000","last_change":"2024-04-26T23:43:07.545626+0000","last_active":"2024-04-26T23:43:08.423658+0000","last_peered":"2024-04-26T23:43:08.423658+0000","last_clean":"2024-04-26T23:43:08.423658+0000","last_became_active":"2024-04-26T23:43:07.545171+0000","last_became_peered":"2024-04-26T23:43:07.545171+0000","last_unstale":"2024-04-26T23:43:08.423658+0000","last_undegraded":"2024-04-26T23:43:08.423658+0000","last_fullsized":"2024-04-26T23:43:08.423658+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:23:33.829834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:29.204830+0000","last_change":"2024-04-26T23:43:02.457426+0000","last_active":"2024-04-26T23:43:29.204830+0000","last_peered":"2024-04-26T23:43:29.204830+0000","last_clean":"2024-04-26T23:43:29.204830+0000","last_became_active":"2024-04-26T23:43:02.457008+0000","last_became_peered":"2024-04-26T23:43:02.457008+0000","last_unstale":"2024-04-26T23:43:29.204830+0000","last_undegraded":"2024-04-26T23:43:29.204830+0000","last_fullsized":"2024-04-26T23:43:29.204830+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:32:45.324729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.417741+0000","last_change":"2024-04-26T23:43:03.290567+0000","last_active":"2024-04-26T23:43:08.417741+0000","last_peered":"2024-04-26T23:43:08.417741+0000","last_clean":"2024-04-26T23:43:08.417741+0000","last_became_active":"2024-04-26T23:43:03.287807+0000","last_became_peered":"2024-04-26T23:43:03.287807+0000","last_unstale":"2024-04-26T23:43:08.417741+0000","last_undegraded":"2024-04-26T23:43:08.417741+0000","last_fullsized":"2024-04-26T23:43:08.417741+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:25:24.393978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.418176+0000","last_change":"2024-04-26T23:43:07.535402+0000","last_active":"2024-04-26T23:43:08.418176+0000","last_peered":"2024-04-26T23:43:08.418176+0000","last_clean":"2024-04-26T23:43:08.418176+0000","last_became_active":"2024-04-26T23:43:07.534870+0000","last_became_peered":"2024-04-26T23:43:07.534870+0000","last_unstale":"2024-04-26T23:43:08.418176+0000","last_undegraded":"2024-04-26T23:43:08.418176+0000","last_fullsized":"2024-04-26T23:43:08.418176+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:26:37.703831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:30.193275+0000","last_change":"2024-04-26T23:43:02.440638+0000","last_active":"2024-04-26T23:43:30.193275+0000","last_peered":"2024-04-26T23:43:30.193275+0000","last_clean":"2024-04-26T23:43:30.193275+0000","last_became_active":"2024-04-26T23:43:02.440294+0000","last_became_peered":"2024-04-26T23:43:02.440294+0000","last_unstale":"2024-04-26T23:43:30.193275+0000","last_undegraded":"2024-04-26T23:43:30.193275+0000","last_fullsized":"2024-04-26T23:43:30.193275+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:21:40.083946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.416976+0000","last_change":"2024-04-26T23:43:03.286679+0000","last_active":"2024-04-26T23:43:08.416976+0000","last_peered":"2024-04-26T23:43:08.416976+0000","last_clean":"2024-04-26T23:43:08.416976+0000","last_became_active":"2024-04-26T23:43:03.286227+0000","last_became_peered":"2024-04-26T23:43:03.286227+0000","last_unstale":"2024-04-26T23:43:08.416976+0000","last_undegraded":"2024-04-26T23:43:08.416976+0000","last_fullsized":"2024-04-26T23:43:08.416976+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:18:26.934987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.417036+0000","last_change":"2024-04-26T23:43:07.855931+0000","last_active":"2024-04-26T23:43:08.417036+0000","last_peered":"2024-04-26T23:43:08.417036+0000","last_clean":"2024-04-26T23:43:08.417036+0000","last_became_active":"2024-04-26T23:43:07.855407+0000","last_became_peered":"2024-04-26T23:43:07.855407+0000","last_unstale":"2024-04-26T23:43:08.417036+0000","last_undegraded":"2024-04-26T23:43:08.417036+0000","last_fullsized":"2024-04-26T23:43:08.417036+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:39:25.707837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.423544+0000","last_change":"2024-04-26T23:43:02.457971+0000","last_active":"2024-04-26T23:43:08.423544+0000","last_peered":"2024-04-26T23:43:08.423544+0000","last_clean":"2024-04-26T23:43:08.423544+0000","last_became_active":"2024-04-26T23:43:02.457627+0000","last_became_peered":"2024-04-26T23:43:02.457627+0000","last_unstale":"2024-04-26T23:43:08.423544+0000","last_undegraded":"2024-04-26T23:43:08.423544+0000","last_fullsized":"2024-04-26T23:43:08.423544+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:25:51.749774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.417114+0000","last_change":"2024-04-26T23:43:03.305569+0000","last_active":"2024-04-26T23:43:08.417114+0000","last_peered":"2024-04-26T23:43:08.417114+0000","last_clean":"2024-04-26T23:43:08.417114+0000","last_became_active":"2024-04-26T23:43:03.305211+0000","last_became_peered":"2024-04-26T23:43:03.305211+0000","last_unstale":"2024-04-26T23:43:08.417114+0000","last_undegraded":"2024-04-26T23:43:08.417114+0000","last_fullsized":"2024-04-26T23:43:08.417114+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:54:55.847680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.421124+0000","last_change":"2024-04-26T23:43:07.535408+0000","last_active":"2024-04-26T23:43:08.421124+0000","last_peered":"2024-04-26T23:43:08.421124+0000","last_clean":"2024-04-26T23:43:08.421124+0000","last_became_active":"2024-04-26T23:43:07.534957+0000","last_became_peered":"2024-04-26T23:43:07.534957+0000","last_unstale":"2024-04-26T23:43:08.421124+0000","last_undegraded":"2024-04-26T23:43:08.421124+0000","last_fullsized":"2024-04-26T23:43:08.421124+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:35:31.103532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.162656+0000","last_change":"2024-04-26T23:43:02.453636+0000","last_active":"2024-04-26T23:43:08.162656+0000","last_peered":"2024-04-26T23:43:08.162656+0000","last_clean":"2024-04-26T23:43:08.162656+0000","last_became_active":"2024-04-26T23:43:02.453310+0000","last_became_peered":"2024-04-26T23:43:02.453310+0000","last_unstale":"2024-04-26T23:43:08.162656+0000","last_undegraded":"2024-04-26T23:43:08.162656+0000","last_fullsized":"2024-04-26T23:43:08.162656+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:42:29.295303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.416769+0000","last_change":"2024-04-26T23:43:03.283184+0000","last_active":"2024-04-26T23:43:08.416769+0000","last_peered":"2024-04-26T23:43:08.416769+0000","last_clean":"2024-04-26T23:43:08.416769+0000","last_became_active":"2024-04-26T23:43:03.282780+0000","last_became_peered":"2024-04-26T23:43:03.282780+0000","last_unstale":"2024-04-26T23:43:08.416769+0000","last_undegraded":"2024-04-26T23:43:08.416769+0000","last_fullsized":"2024-04-26T23:43:08.416769+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:00:13.907101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.422981+0000","last_change":"2024-04-26T23:43:07.859107+0000","last_active":"2024-04-26T23:43:08.422981+0000","last_peered":"2024-04-26T23:43:08.422981+0000","last_clean":"2024-04-26T23:43:08.422981+0000","last_became_active":"2024-04-26T23:43:07.856492+0000","last_became_peered":"2024-04-26T23:43:07.856492+0000","last_unstale":"2024-04-26T23:43:08.422981+0000","last_undegraded":"2024-04-26T23:43:08.422981+0000","last_fullsized":"2024-04-26T23:43:08.422981+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:13:36.570398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.417094+0000","last_change":"2024-04-26T23:43:03.289078+0000","last_active":"2024-04-26T23:43:08.417094+0000","last_peered":"2024-04-26T23:43:08.417094+0000","last_clean":"2024-04-26T23:43:08.417094+0000","last_became_active":"2024-04-26T23:43:03.288729+0000","last_became_peered":"2024-04-26T23:43:03.288729+0000","last_unstale":"2024-04-26T23:43:08.417094+0000","last_undegraded":"2024-04-26T23:43:08.417094+0000","last_fullsized":"2024-04-26T23:43:08.417094+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:32:36.346472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.419577+0000","last_change":"2024-04-26T23:43:02.463355+0000","last_active":"2024-04-26T23:43:08.419577+0000","last_peered":"2024-04-26T23:43:08.419577+0000","last_clean":"2024-04-26T23:43:08.419577+0000","last_became_active":"2024-04-26T23:43:02.455832+0000","last_became_peered":"2024-04-26T23:43:02.455832+0000","last_unstale":"2024-04-26T23:43:08.419577+0000","last_undegraded":"2024-04-26T23:43:08.419577+0000","last_fullsized":"2024-04-26T23:43:08.419577+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:33:33.837999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.163277+0000","last_change":"2024-04-26T23:43:07.540777+0000","last_active":"2024-04-26T23:43:08.163277+0000","last_peered":"2024-04-26T23:43:08.163277+0000","last_clean":"2024-04-26T23:43:08.163277+0000","last_became_active":"2024-04-26T23:43:07.540389+0000","last_became_peered":"2024-04-26T23:43:07.540389+0000","last_unstale":"2024-04-26T23:43:08.163277+0000","last_undegraded":"2024-04-26T23:43:08.163277+0000","last_fullsized":"2024-04-26T23:43:08.163277+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:57:28.674051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.418640+0000","last_change":"2024-04-26T23:43:03.308391+0000","last_active":"2024-04-26T23:43:08.418640+0000","last_peered":"2024-04-26T23:43:08.418640+0000","last_clean":"2024-04-26T23:43:08.418640+0000","last_became_active":"2024-04-26T23:43:03.306762+0000","last_became_peered":"2024-04-26T23:43:03.306762+0000","last_unstale":"2024-04-26T23:43:08.418640+0000","last_undegraded":"2024-04-26T23:43:08.418640+0000","last_fullsized":"2024-04-26T23:43:08.418640+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:34:01.083036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:28.192474+0000","last_change":"2024-04-26T23:43:02.448037+0000","last_active":"2024-04-26T23:43:28.192474+0000","last_peered":"2024-04-26T23:43:28.192474+0000","last_clean":"2024-04-26T23:43:28.192474+0000","last_became_active":"2024-04-26T23:43:02.447706+0000","last_became_peered":"2024-04-26T23:43:02.447706+0000","last_unstale":"2024-04-26T23:43:28.192474+0000","last_undegraded":"2024-04-26T23:43:28.192474+0000","last_fullsized":"2024-04-26T23:43:28.192474+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:43:56.295191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":343,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.162465+0000","last_change":"2024-04-26T23:41:42.266448+0000","last_active":"2024-04-26T23:43:08.162465+0000","last_peered":"2024-04-26T23:43:08.162465+0000","last_clean":"2024-04-26T23:43:08.162465+0000","last_became_active":"2024-04-26T23:41:42.265898+0000","last_became_peered":"2024-04-26T23:41:42.265898+0000","last_unstale":"2024-04-26T23:43:08.162465+0000","last_undegraded":"2024-04-26T23:43:08.162465+0000","last_fullsized":"2024-04-26T23:43:08.162465+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:37:44.460409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:37:44.460409+0000","last_clean_scrub_stamp":"2024-04-26T23:37:44.460409+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:40:48.177191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.417876+0000","last_change":"2024-04-26T23:43:07.856089+0000","last_active":"2024-04-26T23:43:08.417876+0000","last_peered":"2024-04-26T23:43:08.417876+0000","last_clean":"2024-04-26T23:43:08.417876+0000","last_became_active":"2024-04-26T23:43:07.855542+0000","last_became_peered":"2024-04-26T23:43:07.855542+0000","last_unstale":"2024-04-26T23:43:08.417876+0000","last_undegraded":"2024-04-26T23:43:08.417876+0000","last_fullsized":"2024-04-26T23:43:08.417876+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:48:22.799242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.419122+0000","last_change":"2024-04-26T23:43:03.301989+0000","last_active":"2024-04-26T23:43:08.419122+0000","last_peered":"2024-04-26T23:43:08.419122+0000","last_clean":"2024-04-26T23:43:08.419122+0000","last_became_active":"2024-04-26T23:43:03.301179+0000","last_became_peered":"2024-04-26T23:43:03.301179+0000","last_unstale":"2024-04-26T23:43:08.419122+0000","last_undegraded":"2024-04-26T23:43:08.419122+0000","last_fullsized":"2024-04-26T23:43:08.419122+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:29:33.012009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:29.193899+0000","last_change":"2024-04-26T23:43:02.439648+0000","last_active":"2024-04-26T23:43:29.193899+0000","last_peered":"2024-04-26T23:43:29.193899+0000","last_clean":"2024-04-26T23:43:29.193899+0000","last_became_active":"2024-04-26T23:43:02.439340+0000","last_became_peered":"2024-04-26T23:43:02.439340+0000","last_unstale":"2024-04-26T23:43:29.193899+0000","last_undegraded":"2024-04-26T23:43:29.193899+0000","last_fullsized":"2024-04-26T23:43:29.193899+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:10:01.096657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.416648+0000","last_change":"2024-04-26T23:43:07.534317+0000","last_active":"2024-04-26T23:43:08.416648+0000","last_peered":"2024-04-26T23:43:08.416648+0000","last_clean":"2024-04-26T23:43:08.416648+0000","last_became_active":"2024-04-26T23:43:07.533808+0000","last_became_peered":"2024-04-26T23:43:07.533808+0000","last_unstale":"2024-04-26T23:43:08.416648+0000","last_undegraded":"2024-04-26T23:43:08.416648+0000","last_fullsized":"2024-04-26T23:43:08.416648+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:54:29.952089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:28.187645+0000","last_change":"2024-04-26T23:43:02.457501+0000","last_active":"2024-04-26T23:43:28.187645+0000","last_peered":"2024-04-26T23:43:28.187645+0000","last_clean":"2024-04-26T23:43:28.187645+0000","last_became_active":"2024-04-26T23:43:02.457024+0000","last_became_peered":"2024-04-26T23:43:02.457024+0000","last_unstale":"2024-04-26T23:43:28.187645+0000","last_undegraded":"2024-04-26T23:43:28.187645+0000","last_fullsized":"2024-04-26T23:43:28.187645+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:21:58.542186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.423324+0000","last_change":"2024-04-26T23:43:03.303434+0000","last_active":"2024-04-26T23:43:08.423324+0000","last_peered":"2024-04-26T23:43:08.423324+0000","last_clean":"2024-04-26T23:43:08.423324+0000","last_became_active":"2024-04-26T23:43:03.302455+0000","last_became_peered":"2024-04-26T23:43:03.302455+0000","last_unstale":"2024-04-26T23:43:08.423324+0000","last_undegraded":"2024-04-26T23:43:08.423324+0000","last_fullsized":"2024-04-26T23:43:08.423324+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:57:05.530635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.418642+0000","last_change":"2024-04-26T23:43:07.856932+0000","last_active":"2024-04-26T23:43:08.418642+0000","last_peered":"2024-04-26T23:43:08.418642+0000","last_clean":"2024-04-26T23:43:08.418642+0000","last_became_active":"2024-04-26T23:43:07.855699+0000","last_became_peered":"2024-04-26T23:43:07.855699+0000","last_unstale":"2024-04-26T23:43:08.418642+0000","last_undegraded":"2024-04-26T23:43:08.418642+0000","last_fullsized":"2024-04-26T23:43:08.418642+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:20:28.994301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:29.201401+0000","last_change":"2024-04-26T23:43:02.449852+0000","last_active":"2024-04-26T23:43:29.201401+0000","last_peered":"2024-04-26T23:43:29.201401+0000","last_clean":"2024-04-26T23:43:29.201401+0000","last_became_active":"2024-04-26T23:43:02.449422+0000","last_became_peered":"2024-04-26T23:43:02.449422+0000","last_unstale":"2024-04-26T23:43:29.201401+0000","last_undegraded":"2024-04-26T23:43:29.201401+0000","last_fullsized":"2024-04-26T23:43:29.201401+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:07:25.453872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.421490+0000","last_change":"2024-04-26T23:43:03.304733+0000","last_active":"2024-04-26T23:43:08.421490+0000","last_peered":"2024-04-26T23:43:08.421490+0000","last_clean":"2024-04-26T23:43:08.421490+0000","last_became_active":"2024-04-26T23:43:03.304383+0000","last_became_peered":"2024-04-26T23:43:03.304383+0000","last_unstale":"2024-04-26T23:43:08.421490+0000","last_undegraded":"2024-04-26T23:43:08.421490+0000","last_fullsized":"2024-04-26T23:43:08.421490+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:49:28.679255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.162339+0000","last_change":"2024-04-26T23:43:07.525310+0000","last_active":"2024-04-26T23:43:08.162339+0000","last_peered":"2024-04-26T23:43:08.162339+0000","last_clean":"2024-04-26T23:43:08.162339+0000","last_became_active":"2024-04-26T23:43:07.524502+0000","last_became_peered":"2024-04-26T23:43:07.524502+0000","last_unstale":"2024-04-26T23:43:08.162339+0000","last_undegraded":"2024-04-26T23:43:08.162339+0000","last_fullsized":"2024-04-26T23:43:08.162339+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:07:45.927837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:28.192326+0000","last_change":"2024-04-26T23:43:02.446238+0000","last_active":"2024-04-26T23:43:28.192326+0000","last_peered":"2024-04-26T23:43:28.192326+0000","last_clean":"2024-04-26T23:43:28.192326+0000","last_became_active":"2024-04-26T23:43:02.445959+0000","last_became_peered":"2024-04-26T23:43:02.445959+0000","last_unstale":"2024-04-26T23:43:28.192326+0000","last_undegraded":"2024-04-26T23:43:28.192326+0000","last_fullsized":"2024-04-26T23:43:28.192326+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:25:10.007481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.165423+0000","last_change":"2024-04-26T23:43:03.306035+0000","last_active":"2024-04-26T23:43:08.165423+0000","last_peered":"2024-04-26T23:43:08.165423+0000","last_clean":"2024-04-26T23:43:08.165423+0000","last_became_active":"2024-04-26T23:43:03.305604+0000","last_became_peered":"2024-04-26T23:43:03.305604+0000","last_unstale":"2024-04-26T23:43:08.165423+0000","last_undegraded":"2024-04-26T23:43:08.165423+0000","last_fullsized":"2024-04-26T23:43:08.165423+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:02:38.567754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.416327+0000","last_change":"2024-04-26T23:43:07.855808+0000","last_active":"2024-04-26T23:43:08.416327+0000","last_peered":"2024-04-26T23:43:08.416327+0000","last_clean":"2024-04-26T23:43:08.416327+0000","last_became_active":"2024-04-26T23:43:07.855266+0000","last_became_peered":"2024-04-26T23:43:07.855266+0000","last_unstale":"2024-04-26T23:43:08.416327+0000","last_undegraded":"2024-04-26T23:43:08.416327+0000","last_fullsized":"2024-04-26T23:43:08.416327+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:06:03.016504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:28.193197+0000","last_change":"2024-04-26T23:43:02.443464+0000","last_active":"2024-04-26T23:43:28.193197+0000","last_peered":"2024-04-26T23:43:28.193197+0000","last_clean":"2024-04-26T23:43:28.193197+0000","last_became_active":"2024-04-26T23:43:02.443170+0000","last_became_peered":"2024-04-26T23:43:02.443170+0000","last_unstale":"2024-04-26T23:43:28.193197+0000","last_undegraded":"2024-04-26T23:43:28.193197+0000","last_fullsized":"2024-04-26T23:43:28.193197+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:45:51.431082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.416419+0000","last_change":"2024-04-26T23:43:03.282714+0000","last_active":"2024-04-26T23:43:08.416419+0000","last_peered":"2024-04-26T23:43:08.416419+0000","last_clean":"2024-04-26T23:43:08.416419+0000","last_became_active":"2024-04-26T23:43:03.282353+0000","last_became_peered":"2024-04-26T23:43:03.282353+0000","last_unstale":"2024-04-26T23:43:08.416419+0000","last_undegraded":"2024-04-26T23:43:08.416419+0000","last_fullsized":"2024-04-26T23:43:08.416419+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:18:24.082946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.424304+0000","last_change":"2024-04-26T23:43:07.543301+0000","last_active":"2024-04-26T23:43:08.424304+0000","last_peered":"2024-04-26T23:43:08.424304+0000","last_clean":"2024-04-26T23:43:08.424304+0000","last_became_active":"2024-04-26T23:43:07.540434+0000","last_became_peered":"2024-04-26T23:43:07.540434+0000","last_unstale":"2024-04-26T23:43:08.424304+0000","last_undegraded":"2024-04-26T23:43:08.424304+0000","last_fullsized":"2024-04-26T23:43:08.424304+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:15:18.542595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:28.189484+0000","last_change":"2024-04-26T23:43:02.451391+0000","last_active":"2024-04-26T23:43:28.189484+0000","last_peered":"2024-04-26T23:43:28.189484+0000","last_clean":"2024-04-26T23:43:28.189484+0000","last_became_active":"2024-04-26T23:43:02.450735+0000","last_became_peered":"2024-04-26T23:43:02.450735+0000","last_unstale":"2024-04-26T23:43:28.189484+0000","last_undegraded":"2024-04-26T23:43:28.189484+0000","last_fullsized":"2024-04-26T23:43:28.189484+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:57:48.658671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.416029+0000","last_change":"2024-04-26T23:43:03.301858+0000","last_active":"2024-04-26T23:43:08.416029+0000","last_peered":"2024-04-26T23:43:08.416029+0000","last_clean":"2024-04-26T23:43:08.416029+0000","last_became_active":"2024-04-26T23:43:03.301474+0000","last_became_peered":"2024-04-26T23:43:03.301474+0000","last_unstale":"2024-04-26T23:43:08.416029+0000","last_undegraded":"2024-04-26T23:43:08.416029+0000","last_fullsized":"2024-04-26T23:43:08.416029+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:24:56.386127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.416583+0000","last_change":"2024-04-26T23:43:07.544512+0000","last_active":"2024-04-26T23:43:08.416583+0000","last_peered":"2024-04-26T23:43:08.416583+0000","last_clean":"2024-04-26T23:43:08.416583+0000","last_became_active":"2024-04-26T23:43:07.544029+0000","last_became_peered":"2024-04-26T23:43:07.544029+0000","last_unstale":"2024-04-26T23:43:08.416583+0000","last_undegraded":"2024-04-26T23:43:08.416583+0000","last_fullsized":"2024-04-26T23:43:08.416583+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:38:08.149123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.162473+0000","last_change":"2024-04-26T23:43:02.447594+0000","last_active":"2024-04-26T23:43:08.162473+0000","last_peered":"2024-04-26T23:43:08.162473+0000","last_clean":"2024-04-26T23:43:08.162473+0000","last_became_active":"2024-04-26T23:43:02.447267+0000","last_became_peered":"2024-04-26T23:43:02.447267+0000","last_unstale":"2024-04-26T23:43:08.162473+0000","last_undegraded":"2024-04-26T23:43:08.162473+0000","last_fullsized":"2024-04-26T23:43:08.162473+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:42:18.555585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.420337+0000","last_change":"2024-04-26T23:43:03.307895+0000","last_active":"2024-04-26T23:43:08.420337+0000","last_peered":"2024-04-26T23:43:08.420337+0000","last_clean":"2024-04-26T23:43:08.420337+0000","last_became_active":"2024-04-26T23:43:03.307478+0000","last_became_peered":"2024-04-26T23:43:03.307478+0000","last_unstale":"2024-04-26T23:43:08.420337+0000","last_undegraded":"2024-04-26T23:43:08.420337+0000","last_fullsized":"2024-04-26T23:43:08.420337+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:57:14.356234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.416544+0000","last_change":"2024-04-26T23:43:07.855803+0000","last_active":"2024-04-26T23:43:08.416544+0000","last_peered":"2024-04-26T23:43:08.416544+0000","last_clean":"2024-04-26T23:43:08.416544+0000","last_became_active":"2024-04-26T23:43:07.855200+0000","last_became_peered":"2024-04-26T23:43:07.855200+0000","last_unstale":"2024-04-26T23:43:08.416544+0000","last_undegraded":"2024-04-26T23:43:08.416544+0000","last_fullsized":"2024-04-26T23:43:08.416544+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:02:46.223330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:28.184840+0000","last_change":"2024-04-26T23:43:02.437513+0000","last_active":"2024-04-26T23:43:28.184840+0000","last_peered":"2024-04-26T23:43:28.184840+0000","last_clean":"2024-04-26T23:43:28.184840+0000","last_became_active":"2024-04-26T23:43:02.437153+0000","last_became_peered":"2024-04-26T23:43:02.437153+0000","last_unstale":"2024-04-26T23:43:28.184840+0000","last_undegraded":"2024-04-26T23:43:28.184840+0000","last_fullsized":"2024-04-26T23:43:28.184840+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:11:45.477579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.162800+0000","last_change":"2024-04-26T23:43:03.297835+0000","last_active":"2024-04-26T23:43:08.162800+0000","last_peered":"2024-04-26T23:43:08.162800+0000","last_clean":"2024-04-26T23:43:08.162800+0000","last_became_active":"2024-04-26T23:43:03.297481+0000","last_became_peered":"2024-04-26T23:43:03.297481+0000","last_unstale":"2024-04-26T23:43:08.162800+0000","last_undegraded":"2024-04-26T23:43:08.162800+0000","last_fullsized":"2024-04-26T23:43:08.162800+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:52:54.757461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.237891+0000","last_change":"2024-04-26T23:43:07.536849+0000","last_active":"2024-04-26T23:43:08.237891+0000","last_peered":"2024-04-26T23:43:08.237891+0000","last_clean":"2024-04-26T23:43:08.237891+0000","last_became_active":"2024-04-26T23:43:07.536214+0000","last_became_peered":"2024-04-26T23:43:07.536214+0000","last_unstale":"2024-04-26T23:43:08.237891+0000","last_undegraded":"2024-04-26T23:43:08.237891+0000","last_fullsized":"2024-04-26T23:43:08.237891+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:33:16.782368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:29.203867+0000","last_change":"2024-04-26T23:43:02.464181+0000","last_active":"2024-04-26T23:43:29.203867+0000","last_peered":"2024-04-26T23:43:29.203867+0000","last_clean":"2024-04-26T23:43:29.203867+0000","last_became_active":"2024-04-26T23:43:02.463830+0000","last_became_peered":"2024-04-26T23:43:02.463830+0000","last_unstale":"2024-04-26T23:43:29.203867+0000","last_undegraded":"2024-04-26T23:43:29.203867+0000","last_fullsized":"2024-04-26T23:43:29.203867+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:14:52.831302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.163327+0000","last_change":"2024-04-26T23:43:03.298497+0000","last_active":"2024-04-26T23:43:08.163327+0000","last_peered":"2024-04-26T23:43:08.163327+0000","last_clean":"2024-04-26T23:43:08.163327+0000","last_became_active":"2024-04-26T23:43:03.298194+0000","last_became_peered":"2024-04-26T23:43:03.298194+0000","last_unstale":"2024-04-26T23:43:08.163327+0000","last_undegraded":"2024-04-26T23:43:08.163327+0000","last_fullsized":"2024-04-26T23:43:08.163327+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:39:19.007294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.165664+0000","last_change":"2024-04-26T23:43:07.527962+0000","last_active":"2024-04-26T23:43:08.165664+0000","last_peered":"2024-04-26T23:43:08.165664+0000","last_clean":"2024-04-26T23:43:08.165664+0000","last_became_active":"2024-04-26T23:43:07.527386+0000","last_became_peered":"2024-04-26T23:43:07.527386+0000","last_unstale":"2024-04-26T23:43:08.165664+0000","last_undegraded":"2024-04-26T23:43:08.165664+0000","last_fullsized":"2024-04-26T23:43:08.165664+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:57:21.711670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.418106+0000","last_change":"2024-04-26T23:43:02.453988+0000","last_active":"2024-04-26T23:43:08.418106+0000","last_peered":"2024-04-26T23:43:08.418106+0000","last_clean":"2024-04-26T23:43:08.418106+0000","last_became_active":"2024-04-26T23:43:02.453654+0000","last_became_peered":"2024-04-26T23:43:02.453654+0000","last_unstale":"2024-04-26T23:43:08.418106+0000","last_undegraded":"2024-04-26T23:43:08.418106+0000","last_fullsized":"2024-04-26T23:43:08.418106+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:46:08.718705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.162832+0000","last_change":"2024-04-26T23:43:03.305861+0000","last_active":"2024-04-26T23:43:08.162832+0000","last_peered":"2024-04-26T23:43:08.162832+0000","last_clean":"2024-04-26T23:43:08.162832+0000","last_became_active":"2024-04-26T23:43:03.305301+0000","last_became_peered":"2024-04-26T23:43:03.305301+0000","last_unstale":"2024-04-26T23:43:08.162832+0000","last_undegraded":"2024-04-26T23:43:08.162832+0000","last_fullsized":"2024-04-26T23:43:08.162832+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:08:24.174443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.416825+0000","last_change":"2024-04-26T23:43:07.529914+0000","last_active":"2024-04-26T23:43:08.416825+0000","last_peered":"2024-04-26T23:43:08.416825+0000","last_clean":"2024-04-26T23:43:08.416825+0000","last_became_active":"2024-04-26T23:43:07.529509+0000","last_became_peered":"2024-04-26T23:43:07.529509+0000","last_unstale":"2024-04-26T23:43:08.416825+0000","last_undegraded":"2024-04-26T23:43:08.416825+0000","last_fullsized":"2024-04-26T23:43:08.416825+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:50:23.293672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.422976+0000","last_change":"2024-04-26T23:43:02.452096+0000","last_active":"2024-04-26T23:43:08.422976+0000","last_peered":"2024-04-26T23:43:08.422976+0000","last_clean":"2024-04-26T23:43:08.422976+0000","last_became_active":"2024-04-26T23:43:02.450039+0000","last_became_peered":"2024-04-26T23:43:02.450039+0000","last_unstale":"2024-04-26T23:43:08.422976+0000","last_undegraded":"2024-04-26T23:43:08.422976+0000","last_fullsized":"2024-04-26T23:43:08.422976+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:47:03.062575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.163191+0000","last_change":"2024-04-26T23:43:03.294790+0000","last_active":"2024-04-26T23:43:08.163191+0000","last_peered":"2024-04-26T23:43:08.163191+0000","last_clean":"2024-04-26T23:43:08.163191+0000","last_became_active":"2024-04-26T23:43:03.294460+0000","last_became_peered":"2024-04-26T23:43:03.294460+0000","last_unstale":"2024-04-26T23:43:08.163191+0000","last_undegraded":"2024-04-26T23:43:08.163191+0000","last_fullsized":"2024-04-26T23:43:08.163191+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:57:40.879315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.416114+0000","last_change":"2024-04-26T23:43:03.284344+0000","last_active":"2024-04-26T23:43:08.416114+0000","last_peered":"2024-04-26T23:43:08.416114+0000","last_clean":"2024-04-26T23:43:08.416114+0000","last_became_active":"2024-04-26T23:43:03.283954+0000","last_became_peered":"2024-04-26T23:43:03.283954+0000","last_unstale":"2024-04-26T23:43:08.416114+0000","last_undegraded":"2024-04-26T23:43:08.416114+0000","last_fullsized":"2024-04-26T23:43:08.416114+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:12:56.516067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.162510+0000","last_change":"2024-04-26T23:43:03.291959+0000","last_active":"2024-04-26T23:43:08.162510+0000","last_peered":"2024-04-26T23:43:08.162510+0000","last_clean":"2024-04-26T23:43:08.162510+0000","last_became_active":"2024-04-26T23:43:03.291597+0000","last_became_peered":"2024-04-26T23:43:03.291597+0000","last_unstale":"2024-04-26T23:43:08.162510+0000","last_undegraded":"2024-04-26T23:43:08.162510+0000","last_fullsized":"2024-04-26T23:43:08.162510+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:36:27.324979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.162268+0000","last_change":"2024-04-26T23:43:03.309548+0000","last_active":"2024-04-26T23:43:08.162268+0000","last_peered":"2024-04-26T23:43:08.162268+0000","last_clean":"2024-04-26T23:43:08.162268+0000","last_became_active":"2024-04-26T23:43:03.309113+0000","last_became_peered":"2024-04-26T23:43:03.309113+0000","last_unstale":"2024-04-26T23:43:08.162268+0000","last_undegraded":"2024-04-26T23:43:08.162268+0000","last_fullsized":"2024-04-26T23:43:08.162268+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:49:15.720320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.164961+0000","last_change":"2024-04-26T23:43:03.304580+0000","last_active":"2024-04-26T23:43:08.164961+0000","last_peered":"2024-04-26T23:43:08.164961+0000","last_clean":"2024-04-26T23:43:08.164961+0000","last_became_active":"2024-04-26T23:43:03.303128+0000","last_became_peered":"2024-04-26T23:43:03.303128+0000","last_unstale":"2024-04-26T23:43:08.164961+0000","last_undegraded":"2024-04-26T23:43:08.164961+0000","last_fullsized":"2024-04-26T23:43:08.164961+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:11:36.868298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.423531+0000","last_change":"2024-04-26T23:43:03.306441+0000","last_active":"2024-04-26T23:43:08.423531+0000","last_peered":"2024-04-26T23:43:08.423531+0000","last_clean":"2024-04-26T23:43:08.423531+0000","last_became_active":"2024-04-26T23:43:03.305209+0000","last_became_peered":"2024-04-26T23:43:03.305209+0000","last_unstale":"2024-04-26T23:43:08.423531+0000","last_undegraded":"2024-04-26T23:43:08.423531+0000","last_fullsized":"2024-04-26T23:43:08.423531+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:41:39.757001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.416988+0000","last_change":"2024-04-26T23:43:03.284291+0000","last_active":"2024-04-26T23:43:08.416988+0000","last_peered":"2024-04-26T23:43:08.416988+0000","last_clean":"2024-04-26T23:43:08.416988+0000","last_became_active":"2024-04-26T23:43:03.284006+0000","last_became_peered":"2024-04-26T23:43:03.284006+0000","last_unstale":"2024-04-26T23:43:08.416988+0000","last_undegraded":"2024-04-26T23:43:08.416988+0000","last_fullsized":"2024-04-26T23:43:08.416988+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:24:01.386322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.163092+0000","last_change":"2024-04-26T23:43:03.286478+0000","last_active":"2024-04-26T23:43:08.163092+0000","last_peered":"2024-04-26T23:43:08.163092+0000","last_clean":"2024-04-26T23:43:08.163092+0000","last_became_active":"2024-04-26T23:43:03.286115+0000","last_became_peered":"2024-04-26T23:43:03.286115+0000","last_unstale":"2024-04-26T23:43:08.163092+0000","last_undegraded":"2024-04-26T23:43:08.163092+0000","last_fullsized":"2024-04-26T23:43:08.163092+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:34:57.534124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.421801+0000","last_change":"2024-04-26T23:43:03.310549+0000","last_active":"2024-04-26T23:43:08.421801+0000","last_peered":"2024-04-26T23:43:08.421801+0000","last_clean":"2024-04-26T23:43:08.421801+0000","last_became_active":"2024-04-26T23:43:03.309786+0000","last_became_peered":"2024-04-26T23:43:03.309786+0000","last_unstale":"2024-04-26T23:43:08.421801+0000","last_undegraded":"2024-04-26T23:43:08.421801+0000","last_fullsized":"2024-04-26T23:43:08.421801+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:03:33.243368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.416711+0000","last_change":"2024-04-26T23:43:03.300602+0000","last_active":"2024-04-26T23:43:08.416711+0000","last_peered":"2024-04-26T23:43:08.416711+0000","last_clean":"2024-04-26T23:43:08.416711+0000","last_became_active":"2024-04-26T23:43:03.300225+0000","last_became_peered":"2024-04-26T23:43:03.300225+0000","last_unstale":"2024-04-26T23:43:08.416711+0000","last_undegraded":"2024-04-26T23:43:08.416711+0000","last_fullsized":"2024-04-26T23:43:08.416711+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:52:36.638153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.420392+0000","last_change":"2024-04-26T23:43:03.305631+0000","last_active":"2024-04-26T23:43:08.420392+0000","last_peered":"2024-04-26T23:43:08.420392+0000","last_clean":"2024-04-26T23:43:08.420392+0000","last_became_active":"2024-04-26T23:43:03.300193+0000","last_became_peered":"2024-04-26T23:43:03.300193+0000","last_unstale":"2024-04-26T23:43:08.420392+0000","last_undegraded":"2024-04-26T23:43:08.420392+0000","last_fullsized":"2024-04-26T23:43:08.420392+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:27:08.355753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.237844+0000","last_change":"2024-04-26T23:43:03.309514+0000","last_active":"2024-04-26T23:43:08.237844+0000","last_peered":"2024-04-26T23:43:08.237844+0000","last_clean":"2024-04-26T23:43:08.237844+0000","last_became_active":"2024-04-26T23:43:03.309085+0000","last_became_peered":"2024-04-26T23:43:03.309085+0000","last_unstale":"2024-04-26T23:43:08.237844+0000","last_undegraded":"2024-04-26T23:43:08.237844+0000","last_fullsized":"2024-04-26T23:43:08.237844+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:08:45.649513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.422068+0000","last_change":"2024-04-26T23:43:03.303553+0000","last_active":"2024-04-26T23:43:08.422068+0000","last_peered":"2024-04-26T23:43:08.422068+0000","last_clean":"2024-04-26T23:43:08.422068+0000","last_became_active":"2024-04-26T23:43:03.302979+0000","last_became_peered":"2024-04-26T23:43:03.302979+0000","last_unstale":"2024-04-26T23:43:08.422068+0000","last_undegraded":"2024-04-26T23:43:08.422068+0000","last_fullsized":"2024-04-26T23:43:08.422068+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:44:25.809138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.416432+0000","last_change":"2024-04-26T23:43:03.288433+0000","last_active":"2024-04-26T23:43:08.416432+0000","last_peered":"2024-04-26T23:43:08.416432+0000","last_clean":"2024-04-26T23:43:08.416432+0000","last_became_active":"2024-04-26T23:43:03.288125+0000","last_became_peered":"2024-04-26T23:43:03.288125+0000","last_unstale":"2024-04-26T23:43:08.416432+0000","last_undegraded":"2024-04-26T23:43:08.416432+0000","last_fullsized":"2024-04-26T23:43:08.416432+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:53:21.673755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":282161,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6740,"num_objects":53,"num_object_clones":0,"num_object_copies":159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":116,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":20238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102,"ondisk_log_size":102,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383700,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":477797,"data_compressed":14636,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645336,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1036288,"data_stored":1071619,"data_compressed":50523,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808862,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":478033,"data_compressed":14636,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070500,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":477953,"data_compressed":14636,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332138,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1028096,"data_stored":1069375,"data_compressed":50523,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495663,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":962560,"data_stored":477215,"data_compressed":14636,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574924,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":481057,"data_compressed":14636,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738448,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":966656,"data_stored":477305,"data_compressed":14636,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411399,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":481713,"data_compressed":14636,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182464,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":978944,"data_stored":477499,"data_compressed":14636,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920827,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":966656,"data_stored":477237,"data_compressed":14636,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757302,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1040384,"data_stored":1069059,"data_compressed":50523,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34242,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34242,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34242,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35887,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35887,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35887,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35887,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35887,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":4798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T23:43:34.028 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T23:43:34.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.264+0000 7f4bc9a00640 1 -- 172.21.15.72:0/2215691265 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4bc410a940 msgr2=0x7f4bc41111d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:34.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.264+0000 7f4bc9a00640 1 --2- 172.21.15.72:0/2215691265 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4bc410a940 0x7f4bc41111d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc009900 tx=0x7f4bbc02a3f0 comp rx=0 tx=0).stop 2024-04-26T23:43:34.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.265+0000 7f4bc9a00640 1 -- 172.21.15.72:0/2215691265 shutdown_connections 2024-04-26T23:43:34.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.265+0000 7f4bc9a00640 1 --2- 172.21.15.72:0/2215691265 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4bc410a940 0x7f4bc41111d0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:34.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.265+0000 7f4bc9a00640 1 --2- 172.21.15.72:0/2215691265 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4bc4109f80 0x7f4bc410a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:34.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.265+0000 7f4bc9a00640 1 --2- 172.21.15.72:0/2215691265 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4bc4108d90 0x7f4bc41091b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:34.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.265+0000 7f4bc9a00640 1 -- 172.21.15.72:0/2215691265 >> 172.21.15.72:0/2215691265 conn(0x7f4bc4077d30 msgr2=0x7f4bc407a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:34.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.265+0000 7f4bc9a00640 1 -- 172.21.15.72:0/2215691265 shutdown_connections 2024-04-26T23:43:34.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.265+0000 7f4bc9a00640 1 -- 172.21.15.72:0/2215691265 wait complete. 2024-04-26T23:43:34.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.266+0000 7f4bc9a00640 1 Processor -- start 2024-04-26T23:43:34.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.266+0000 7f4bc9a00640 1 -- start start 2024-04-26T23:43:34.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.266+0000 7f4bc9a00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4bc4108d90 0x7f4bc419eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:34.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.266+0000 7f4bc9a00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4bc4109f80 0x7f4bc419f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:34.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.266+0000 7f4bc9a00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4bc410a940 0x7f4bc41a64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:34.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.266+0000 7f4bc9a00640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f4bc4114910 con 0x7f4bc4108d90 2024-04-26T23:43:34.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.266+0000 7f4bc9a00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f4bc4114790 con 0x7f4bc410a940 2024-04-26T23:43:34.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.266+0000 7f4bc9a00640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f4bc4114a90 con 0x7f4bc4109f80 2024-04-26T23:43:34.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.266+0000 7f4bc3e00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4bc4108d90 0x7f4bc419eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:34.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.266+0000 7f4bc3e00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4bc4108d90 0x7f4bc419eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42574/0 (socket says 172.21.15.72:42574) 2024-04-26T23:43:34.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.266+0000 7f4bc3e00640 1 -- 172.21.15.72:0/1805960565 learned_addr learned my addr 172.21.15.72:0/1805960565 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:43:34.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.267+0000 7f4bc9000640 1 --2- 172.21.15.72:0/1805960565 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4bc410a940 0x7f4bc41a64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:34.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.266+0000 7f4bbb400640 1 --2- 172.21.15.72:0/1805960565 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4bc4109f80 0x7f4bc419f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:34.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.267+0000 7f4bc3e00640 1 -- 172.21.15.72:0/1805960565 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4bc4109f80 msgr2=0x7f4bc419f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:34.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.267+0000 7f4bc3e00640 1 --2- 172.21.15.72:0/1805960565 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4bc4109f80 0x7f4bc419f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:34.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.267+0000 7f4bc3e00640 1 -- 172.21.15.72:0/1805960565 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4bc410a940 msgr2=0x7f4bc41a64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:34.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.267+0000 7f4bc3e00640 1 --2- 172.21.15.72:0/1805960565 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4bc410a940 0x7f4bc41a64a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:34.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.267+0000 7f4bc3e00640 1 -- 172.21.15.72:0/1805960565 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bc41a6b10 con 0x7f4bc4108d90 2024-04-26T23:43:34.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.267+0000 7f4bbb400640 1 --2- 172.21.15.72:0/1805960565 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4bc4109f80 0x7f4bc419f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:43:34.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.267+0000 7f4bc3e00640 1 --2- 172.21.15.72:0/1805960565 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4bc4108d90 0x7f4bc419eef0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4bb400e9c0 tx=0x7f4bb400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:34.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.267+0000 7f4bc1600640 1 -- 172.21.15.72:0/1805960565 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bb400cd80 con 0x7f4bc4108d90 2024-04-26T23:43:34.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.267+0000 7f4bc9a00640 1 -- 172.21.15.72:0/1805960565 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bc41a6e00 con 0x7f4bc4108d90 2024-04-26T23:43:34.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.267+0000 7f4bc1600640 1 -- 172.21.15.72:0/1805960565 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4bb40035f0 con 0x7f4bc4108d90 2024-04-26T23:43:34.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.267+0000 7f4bc1600640 1 -- 172.21.15.72:0/1805960565 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bb4010690 con 0x7f4bc4108d90 2024-04-26T23:43:34.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.268+0000 7f4bc9a00640 1 -- 172.21.15.72:0/1805960565 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bc41a7210 con 0x7f4bc4108d90 2024-04-26T23:43:34.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.269+0000 7f4bc9a00640 1 -- 172.21.15.72:0/1805960565 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bc41100b0 con 0x7f4bc4108d90 2024-04-26T23:43:34.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.270+0000 7f4bc1600640 1 -- 172.21.15.72:0/1805960565 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4bb4004030 con 0x7f4bc4108d90 2024-04-26T23:43:34.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.272+0000 7f4bc1600640 1 --2- 172.21.15.72:0/1805960565 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f4ba80a3c40 0x7f4ba80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:34.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.272+0000 7f4bc1600640 1 -- 172.21.15.72:0/1805960565 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f4bb40d0d80 con 0x7f4bc4108d90 2024-04-26T23:43:34.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.272+0000 7f4bbb400640 1 --2- 172.21.15.72:0/1805960565 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f4ba80a3c40 0x7f4ba80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:34.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.273+0000 7f4bbb400640 1 --2- 172.21.15.72:0/1805960565 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f4ba80a3c40 0x7f4ba80a6100 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4bc41a0400 tx=0x7f4bac009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:34.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.277+0000 7f4bc1600640 1 -- 172.21.15.72:0/1805960565 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4bb409d3c0 con 0x7f4bc4108d90 2024-04-26T23:43:34.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:34 smithi072 ceph-mon[28184]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:43:34.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:34 smithi072 ceph-mon[28184]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-04-26T23:43:34.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:34 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/1673951662' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:43:34.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.479+0000 7f4bc9a00640 1 -- 172.21.15.72:0/1805960565 --> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4bc4062d70 con 0x7f4ba80a3c40 2024-04-26T23:43:34.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.493+0000 7f4bc1600640 1 -- 172.21.15.72:0/1805960565 <== mgr.14152 v2:172.21.15.72:6800/4281177709 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340630 (secure 0 0 0) 0x7f4bc4062d70 con 0x7f4ba80a3c40 2024-04-26T23:43:34.500 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:43:34.504 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-04-26T23:43:34.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.501+0000 7f4bc9a00640 1 -- 172.21.15.72:0/1805960565 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f4ba80a3c40 msgr2=0x7f4ba80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:34.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.501+0000 7f4bc9a00640 1 --2- 172.21.15.72:0/1805960565 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f4ba80a3c40 0x7f4ba80a6100 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4bc41a0400 tx=0x7f4bac009500 comp rx=0 tx=0).stop 2024-04-26T23:43:34.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.501+0000 7f4bc9a00640 1 -- 172.21.15.72:0/1805960565 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4bc4108d90 msgr2=0x7f4bc419eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:34.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.501+0000 7f4bc9a00640 1 --2- 172.21.15.72:0/1805960565 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4bc4108d90 0x7f4bc419eef0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4bb400e9c0 tx=0x7f4bb400ee80 comp rx=0 tx=0).stop 2024-04-26T23:43:34.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.501+0000 7f4bc9a00640 1 -- 172.21.15.72:0/1805960565 shutdown_connections 2024-04-26T23:43:34.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.501+0000 7f4bc9a00640 1 --2- 172.21.15.72:0/1805960565 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f4ba80a3c40 0x7f4ba80a6100 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:34.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.501+0000 7f4bc9a00640 1 --2- 172.21.15.72:0/1805960565 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4bc410a940 0x7f4bc41a64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:34.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.501+0000 7f4bc9a00640 1 --2- 172.21.15.72:0/1805960565 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4bc4109f80 0x7f4bc419f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:34.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.501+0000 7f4bc9a00640 1 --2- 172.21.15.72:0/1805960565 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4bc4108d90 0x7f4bc419eef0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:34.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.501+0000 7f4bc9a00640 1 -- 172.21.15.72:0/1805960565 >> 172.21.15.72:0/1805960565 conn(0x7f4bc4077d30 msgr2=0x7f4bc410d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:34.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.501+0000 7f4bc9a00640 1 -- 172.21.15.72:0/1805960565 shutdown_connections 2024-04-26T23:43:34.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:34.501+0000 7f4bc9a00640 1 -- 172.21.15.72:0/1805960565 wait complete. 2024-04-26T23:43:34.527 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":337,"stamp":"2024-04-26T23:43:34.417538+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":635348,"num_objects":63,"num_object_clones":0,"num_object_copies":190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":172,"num_read_kb":180,"num_write":325,"num_write_kb":3025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":265,"ondisk_log_size":265,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":356688,"kb_used_data":11696,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124585136,"statfs":{"total":1151940427776,"available":1151575179264,"internally_reserved":0,"allocated":11976704,"data_stored":7565340,"data_compressed":283293,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19139,"internal_metadata":353154365},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":42121,"num_objects":39,"num_object_clones":0,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":46,"num_read_kb":71,"num_write":98,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004687"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.417345+0000","last_change":"2024-04-26T23:43:03.301992+0000","last_active":"2024-04-26T23:43:08.417345+0000","last_peered":"2024-04-26T23:43:08.417345+0000","last_clean":"2024-04-26T23:43:08.417345+0000","last_became_active":"2024-04-26T23:43:03.301498+0000","last_became_peered":"2024-04-26T23:43:03.301498+0000","last_unstale":"2024-04-26T23:43:08.417345+0000","last_undegraded":"2024-04-26T23:43:08.417345+0000","last_fullsized":"2024-04-26T23:43:08.417345+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:40:53.567805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.165116+0000","last_change":"2024-04-26T23:43:03.284964+0000","last_active":"2024-04-26T23:43:08.165116+0000","last_peered":"2024-04-26T23:43:08.165116+0000","last_clean":"2024-04-26T23:43:08.165116+0000","last_became_active":"2024-04-26T23:43:03.284185+0000","last_became_peered":"2024-04-26T23:43:03.284185+0000","last_unstale":"2024-04-26T23:43:08.165116+0000","last_undegraded":"2024-04-26T23:43:08.165116+0000","last_fullsized":"2024-04-26T23:43:08.165116+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:55:35.325775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.165383+0000","last_change":"2024-04-26T23:43:03.300333+0000","last_active":"2024-04-26T23:43:08.165383+0000","last_peered":"2024-04-26T23:43:08.165383+0000","last_clean":"2024-04-26T23:43:08.165383+0000","last_became_active":"2024-04-26T23:43:03.295779+0000","last_became_peered":"2024-04-26T23:43:03.295779+0000","last_unstale":"2024-04-26T23:43:08.165383+0000","last_undegraded":"2024-04-26T23:43:08.165383+0000","last_fullsized":"2024-04-26T23:43:08.165383+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:45:06.598862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.416873+0000","last_change":"2024-04-26T23:43:03.290211+0000","last_active":"2024-04-26T23:43:08.416873+0000","last_peered":"2024-04-26T23:43:08.416873+0000","last_clean":"2024-04-26T23:43:08.416873+0000","last_became_active":"2024-04-26T23:43:03.289853+0000","last_became_peered":"2024-04-26T23:43:03.289853+0000","last_unstale":"2024-04-26T23:43:08.416873+0000","last_undegraded":"2024-04-26T23:43:08.416873+0000","last_fullsized":"2024-04-26T23:43:08.416873+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:14:45.032300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.416540+0000","last_change":"2024-04-26T23:43:03.309667+0000","last_active":"2024-04-26T23:43:08.416540+0000","last_peered":"2024-04-26T23:43:08.416540+0000","last_clean":"2024-04-26T23:43:08.416540+0000","last_became_active":"2024-04-26T23:43:03.309305+0000","last_became_peered":"2024-04-26T23:43:03.309305+0000","last_unstale":"2024-04-26T23:43:08.416540+0000","last_undegraded":"2024-04-26T23:43:08.416540+0000","last_fullsized":"2024-04-26T23:43:08.416540+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:31:56.739866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.416669+0000","last_change":"2024-04-26T23:43:03.292107+0000","last_active":"2024-04-26T23:43:08.416669+0000","last_peered":"2024-04-26T23:43:08.416669+0000","last_clean":"2024-04-26T23:43:08.416669+0000","last_became_active":"2024-04-26T23:43:03.291760+0000","last_became_peered":"2024-04-26T23:43:03.291760+0000","last_unstale":"2024-04-26T23:43:08.416669+0000","last_undegraded":"2024-04-26T23:43:08.416669+0000","last_fullsized":"2024-04-26T23:43:08.416669+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:31:54.654379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.162471+0000","last_change":"2024-04-26T23:43:03.294923+0000","last_active":"2024-04-26T23:43:08.162471+0000","last_peered":"2024-04-26T23:43:08.162471+0000","last_clean":"2024-04-26T23:43:08.162471+0000","last_became_active":"2024-04-26T23:43:03.294523+0000","last_became_peered":"2024-04-26T23:43:03.294523+0000","last_unstale":"2024-04-26T23:43:08.162471+0000","last_undegraded":"2024-04-26T23:43:08.162471+0000","last_fullsized":"2024-04-26T23:43:08.162471+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:36:19.419773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.419625+0000","last_change":"2024-04-26T23:43:03.301335+0000","last_active":"2024-04-26T23:43:08.419625+0000","last_peered":"2024-04-26T23:43:08.419625+0000","last_clean":"2024-04-26T23:43:08.419625+0000","last_became_active":"2024-04-26T23:43:03.300918+0000","last_became_peered":"2024-04-26T23:43:03.300918+0000","last_unstale":"2024-04-26T23:43:08.419625+0000","last_undegraded":"2024-04-26T23:43:08.419625+0000","last_fullsized":"2024-04-26T23:43:08.419625+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:15:05.719452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.238112+0000","last_change":"2024-04-26T23:43:03.285114+0000","last_active":"2024-04-26T23:43:08.238112+0000","last_peered":"2024-04-26T23:43:08.238112+0000","last_clean":"2024-04-26T23:43:08.238112+0000","last_became_active":"2024-04-26T23:43:03.284744+0000","last_became_peered":"2024-04-26T23:43:03.284744+0000","last_unstale":"2024-04-26T23:43:08.238112+0000","last_undegraded":"2024-04-26T23:43:08.238112+0000","last_fullsized":"2024-04-26T23:43:08.238112+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:48:52.936951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.417313+0000","last_change":"2024-04-26T23:43:03.287643+0000","last_active":"2024-04-26T23:43:08.417313+0000","last_peered":"2024-04-26T23:43:08.417313+0000","last_clean":"2024-04-26T23:43:08.417313+0000","last_became_active":"2024-04-26T23:43:03.287324+0000","last_became_peered":"2024-04-26T23:43:03.287324+0000","last_unstale":"2024-04-26T23:43:08.417313+0000","last_undegraded":"2024-04-26T23:43:08.417313+0000","last_fullsized":"2024-04-26T23:43:08.417313+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:50:14.816548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.163323+0000","last_change":"2024-04-26T23:43:03.296421+0000","last_active":"2024-04-26T23:43:08.163323+0000","last_peered":"2024-04-26T23:43:08.163323+0000","last_clean":"2024-04-26T23:43:08.163323+0000","last_became_active":"2024-04-26T23:43:03.296060+0000","last_became_peered":"2024-04-26T23:43:03.296060+0000","last_unstale":"2024-04-26T23:43:08.163323+0000","last_undegraded":"2024-04-26T23:43:08.163323+0000","last_fullsized":"2024-04-26T23:43:08.163323+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:50:44.393446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.416764+0000","last_change":"2024-04-26T23:43:03.288562+0000","last_active":"2024-04-26T23:43:08.416764+0000","last_peered":"2024-04-26T23:43:08.416764+0000","last_clean":"2024-04-26T23:43:08.416764+0000","last_became_active":"2024-04-26T23:43:03.288219+0000","last_became_peered":"2024-04-26T23:43:03.288219+0000","last_unstale":"2024-04-26T23:43:08.416764+0000","last_undegraded":"2024-04-26T23:43:08.416764+0000","last_fullsized":"2024-04-26T23:43:08.416764+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:54:32.107146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.422359+0000","last_change":"2024-04-26T23:43:03.296840+0000","last_active":"2024-04-26T23:43:08.422359+0000","last_peered":"2024-04-26T23:43:08.422359+0000","last_clean":"2024-04-26T23:43:08.422359+0000","last_became_active":"2024-04-26T23:43:03.296501+0000","last_became_peered":"2024-04-26T23:43:03.296501+0000","last_unstale":"2024-04-26T23:43:08.422359+0000","last_undegraded":"2024-04-26T23:43:08.422359+0000","last_fullsized":"2024-04-26T23:43:08.422359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:07:05.293388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.164489+0000","last_change":"2024-04-26T23:43:03.291739+0000","last_active":"2024-04-26T23:43:08.164489+0000","last_peered":"2024-04-26T23:43:08.164489+0000","last_clean":"2024-04-26T23:43:08.164489+0000","last_became_active":"2024-04-26T23:43:03.287839+0000","last_became_peered":"2024-04-26T23:43:03.287839+0000","last_unstale":"2024-04-26T23:43:08.164489+0000","last_undegraded":"2024-04-26T23:43:08.164489+0000","last_fullsized":"2024-04-26T23:43:08.164489+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:10:56.698324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.164219+0000","last_change":"2024-04-26T23:43:03.295767+0000","last_active":"2024-04-26T23:43:08.164219+0000","last_peered":"2024-04-26T23:43:08.164219+0000","last_clean":"2024-04-26T23:43:08.164219+0000","last_became_active":"2024-04-26T23:43:03.295339+0000","last_became_peered":"2024-04-26T23:43:03.295339+0000","last_unstale":"2024-04-26T23:43:08.164219+0000","last_undegraded":"2024-04-26T23:43:08.164219+0000","last_fullsized":"2024-04-26T23:43:08.164219+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:21:44.410953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.237529+0000","last_change":"2024-04-26T23:43:03.301687+0000","last_active":"2024-04-26T23:43:08.237529+0000","last_peered":"2024-04-26T23:43:08.237529+0000","last_clean":"2024-04-26T23:43:08.237529+0000","last_became_active":"2024-04-26T23:43:03.301355+0000","last_became_peered":"2024-04-26T23:43:03.301355+0000","last_unstale":"2024-04-26T23:43:08.237529+0000","last_undegraded":"2024-04-26T23:43:08.237529+0000","last_fullsized":"2024-04-26T23:43:08.237529+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:00:57.991022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.416993+0000","last_change":"2024-04-26T23:43:03.302109+0000","last_active":"2024-04-26T23:43:08.416993+0000","last_peered":"2024-04-26T23:43:08.416993+0000","last_clean":"2024-04-26T23:43:08.416993+0000","last_became_active":"2024-04-26T23:43:03.301725+0000","last_became_peered":"2024-04-26T23:43:03.301725+0000","last_unstale":"2024-04-26T23:43:08.416993+0000","last_undegraded":"2024-04-26T23:43:08.416993+0000","last_fullsized":"2024-04-26T23:43:08.416993+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:36:44.630125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.423155+0000","last_change":"2024-04-26T23:43:03.298880+0000","last_active":"2024-04-26T23:43:08.423155+0000","last_peered":"2024-04-26T23:43:08.423155+0000","last_clean":"2024-04-26T23:43:08.423155+0000","last_became_active":"2024-04-26T23:43:03.298464+0000","last_became_peered":"2024-04-26T23:43:03.298464+0000","last_unstale":"2024-04-26T23:43:08.423155+0000","last_undegraded":"2024-04-26T23:43:08.423155+0000","last_fullsized":"2024-04-26T23:43:08.423155+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:42:54.745002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.163034+0000","last_change":"2024-04-26T23:43:03.309454+0000","last_active":"2024-04-26T23:43:08.163034+0000","last_peered":"2024-04-26T23:43:08.163034+0000","last_clean":"2024-04-26T23:43:08.163034+0000","last_became_active":"2024-04-26T23:43:03.309091+0000","last_became_peered":"2024-04-26T23:43:03.309091+0000","last_unstale":"2024-04-26T23:43:08.163034+0000","last_undegraded":"2024-04-26T23:43:08.163034+0000","last_fullsized":"2024-04-26T23:43:08.163034+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:46:12.333410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.162389+0000","last_change":"2024-04-26T23:43:07.541317+0000","last_active":"2024-04-26T23:43:08.162389+0000","last_peered":"2024-04-26T23:43:08.162389+0000","last_clean":"2024-04-26T23:43:08.162389+0000","last_became_active":"2024-04-26T23:43:07.540760+0000","last_became_peered":"2024-04-26T23:43:07.540760+0000","last_unstale":"2024-04-26T23:43:08.162389+0000","last_undegraded":"2024-04-26T23:43:08.162389+0000","last_fullsized":"2024-04-26T23:43:08.162389+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:28:55.545416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:29.204260+0000","last_change":"2024-04-26T23:43:02.450956+0000","last_active":"2024-04-26T23:43:29.204260+0000","last_peered":"2024-04-26T23:43:29.204260+0000","last_clean":"2024-04-26T23:43:29.204260+0000","last_became_active":"2024-04-26T23:43:02.446733+0000","last_became_peered":"2024-04-26T23:43:02.446733+0000","last_unstale":"2024-04-26T23:43:29.204260+0000","last_undegraded":"2024-04-26T23:43:29.204260+0000","last_fullsized":"2024-04-26T23:43:29.204260+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:58:47.613575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.420791+0000","last_change":"2024-04-26T23:43:03.311681+0000","last_active":"2024-04-26T23:43:08.420791+0000","last_peered":"2024-04-26T23:43:08.420791+0000","last_clean":"2024-04-26T23:43:08.420791+0000","last_became_active":"2024-04-26T23:43:03.311087+0000","last_became_peered":"2024-04-26T23:43:03.311087+0000","last_unstale":"2024-04-26T23:43:08.420791+0000","last_undegraded":"2024-04-26T23:43:08.420791+0000","last_fullsized":"2024-04-26T23:43:08.420791+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:13:00.744062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.417017+0000","last_change":"2024-04-26T23:43:07.543547+0000","last_active":"2024-04-26T23:43:08.417017+0000","last_peered":"2024-04-26T23:43:08.417017+0000","last_clean":"2024-04-26T23:43:08.417017+0000","last_became_active":"2024-04-26T23:43:07.543073+0000","last_became_peered":"2024-04-26T23:43:07.543073+0000","last_unstale":"2024-04-26T23:43:08.417017+0000","last_undegraded":"2024-04-26T23:43:08.417017+0000","last_fullsized":"2024-04-26T23:43:08.417017+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:30:05.418875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.421015+0000","last_change":"2024-04-26T23:43:02.460459+0000","last_active":"2024-04-26T23:43:08.421015+0000","last_peered":"2024-04-26T23:43:08.421015+0000","last_clean":"2024-04-26T23:43:08.421015+0000","last_became_active":"2024-04-26T23:43:02.459652+0000","last_became_peered":"2024-04-26T23:43:02.459652+0000","last_unstale":"2024-04-26T23:43:08.421015+0000","last_undegraded":"2024-04-26T23:43:08.421015+0000","last_fullsized":"2024-04-26T23:43:08.421015+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:27:19.166748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.165811+0000","last_change":"2024-04-26T23:43:03.300078+0000","last_active":"2024-04-26T23:43:08.165811+0000","last_peered":"2024-04-26T23:43:08.165811+0000","last_clean":"2024-04-26T23:43:08.165811+0000","last_became_active":"2024-04-26T23:43:03.296319+0000","last_became_peered":"2024-04-26T23:43:03.296319+0000","last_unstale":"2024-04-26T23:43:08.165811+0000","last_undegraded":"2024-04-26T23:43:08.165811+0000","last_fullsized":"2024-04-26T23:43:08.165811+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:33:51.859539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.416408+0000","last_change":"2024-04-26T23:43:07.543494+0000","last_active":"2024-04-26T23:43:08.416408+0000","last_peered":"2024-04-26T23:43:08.416408+0000","last_clean":"2024-04-26T23:43:08.416408+0000","last_became_active":"2024-04-26T23:43:07.542806+0000","last_became_peered":"2024-04-26T23:43:07.542806+0000","last_unstale":"2024-04-26T23:43:08.416408+0000","last_undegraded":"2024-04-26T23:43:08.416408+0000","last_fullsized":"2024-04-26T23:43:08.416408+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:54:40.522615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:29.201942+0000","last_change":"2024-04-26T23:43:02.438943+0000","last_active":"2024-04-26T23:43:29.201942+0000","last_peered":"2024-04-26T23:43:29.201942+0000","last_clean":"2024-04-26T23:43:29.201942+0000","last_became_active":"2024-04-26T23:43:02.438604+0000","last_became_peered":"2024-04-26T23:43:02.438604+0000","last_unstale":"2024-04-26T23:43:29.201942+0000","last_undegraded":"2024-04-26T23:43:29.201942+0000","last_fullsized":"2024-04-26T23:43:29.201942+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:06:51.054342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.237792+0000","last_change":"2024-04-26T23:43:03.293856+0000","last_active":"2024-04-26T23:43:08.237792+0000","last_peered":"2024-04-26T23:43:08.237792+0000","last_clean":"2024-04-26T23:43:08.237792+0000","last_became_active":"2024-04-26T23:43:03.293189+0000","last_became_peered":"2024-04-26T23:43:03.293189+0000","last_unstale":"2024-04-26T23:43:08.237792+0000","last_undegraded":"2024-04-26T23:43:08.237792+0000","last_fullsized":"2024-04-26T23:43:08.237792+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:50:27.545567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.416144+0000","last_change":"2024-04-26T23:43:07.535396+0000","last_active":"2024-04-26T23:43:08.416144+0000","last_peered":"2024-04-26T23:43:08.416144+0000","last_clean":"2024-04-26T23:43:08.416144+0000","last_became_active":"2024-04-26T23:43:07.534914+0000","last_became_peered":"2024-04-26T23:43:07.534914+0000","last_unstale":"2024-04-26T23:43:08.416144+0000","last_undegraded":"2024-04-26T23:43:08.416144+0000","last_fullsized":"2024-04-26T23:43:08.416144+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:27:31.199079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.416306+0000","last_change":"2024-04-26T23:43:02.442935+0000","last_active":"2024-04-26T23:43:08.416306+0000","last_peered":"2024-04-26T23:43:08.416306+0000","last_clean":"2024-04-26T23:43:08.416306+0000","last_became_active":"2024-04-26T23:43:02.442588+0000","last_became_peered":"2024-04-26T23:43:02.442588+0000","last_unstale":"2024-04-26T23:43:08.416306+0000","last_undegraded":"2024-04-26T23:43:08.416306+0000","last_fullsized":"2024-04-26T23:43:08.416306+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:05:26.442001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.418012+0000","last_change":"2024-04-26T23:43:03.289407+0000","last_active":"2024-04-26T23:43:08.418012+0000","last_peered":"2024-04-26T23:43:08.418012+0000","last_clean":"2024-04-26T23:43:08.418012+0000","last_became_active":"2024-04-26T23:43:03.287951+0000","last_became_peered":"2024-04-26T23:43:03.287951+0000","last_unstale":"2024-04-26T23:43:08.418012+0000","last_undegraded":"2024-04-26T23:43:08.418012+0000","last_fullsized":"2024-04-26T23:43:08.418012+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:24:42.687653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.162911+0000","last_change":"2024-04-26T23:43:07.527019+0000","last_active":"2024-04-26T23:43:08.162911+0000","last_peered":"2024-04-26T23:43:08.162911+0000","last_clean":"2024-04-26T23:43:08.162911+0000","last_became_active":"2024-04-26T23:43:07.526516+0000","last_became_peered":"2024-04-26T23:43:07.526516+0000","last_unstale":"2024-04-26T23:43:08.162911+0000","last_undegraded":"2024-04-26T23:43:08.162911+0000","last_fullsized":"2024-04-26T23:43:08.162911+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:44:41.267144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'6","reported_seq":36,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:32.204506+0000","last_change":"2024-04-26T23:43:02.445159+0000","last_active":"2024-04-26T23:43:32.204506+0000","last_peered":"2024-04-26T23:43:32.204506+0000","last_clean":"2024-04-26T23:43:32.204506+0000","last_became_active":"2024-04-26T23:43:02.444781+0000","last_became_peered":"2024-04-26T23:43:02.444781+0000","last_unstale":"2024-04-26T23:43:32.204506+0000","last_undegraded":"2024-04-26T23:43:32.204506+0000","last_fullsized":"2024-04-26T23:43:32.204506+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:45:16.509790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.417133+0000","last_change":"2024-04-26T23:43:03.292717+0000","last_active":"2024-04-26T23:43:08.417133+0000","last_peered":"2024-04-26T23:43:08.417133+0000","last_clean":"2024-04-26T23:43:08.417133+0000","last_became_active":"2024-04-26T23:43:03.292355+0000","last_became_peered":"2024-04-26T23:43:03.292355+0000","last_unstale":"2024-04-26T23:43:08.417133+0000","last_undegraded":"2024-04-26T23:43:08.417133+0000","last_fullsized":"2024-04-26T23:43:08.417133+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:06:02.063165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.162988+0000","last_change":"2024-04-26T23:43:07.540758+0000","last_active":"2024-04-26T23:43:08.162988+0000","last_peered":"2024-04-26T23:43:08.162988+0000","last_clean":"2024-04-26T23:43:08.162988+0000","last_became_active":"2024-04-26T23:43:07.540305+0000","last_became_peered":"2024-04-26T23:43:07.540305+0000","last_unstale":"2024-04-26T23:43:08.162988+0000","last_undegraded":"2024-04-26T23:43:08.162988+0000","last_fullsized":"2024-04-26T23:43:08.162988+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:08:04.491544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:29.197680+0000","last_change":"2024-04-26T23:43:02.453650+0000","last_active":"2024-04-26T23:43:29.197680+0000","last_peered":"2024-04-26T23:43:29.197680+0000","last_clean":"2024-04-26T23:43:29.197680+0000","last_became_active":"2024-04-26T23:43:02.453319+0000","last_became_peered":"2024-04-26T23:43:02.453319+0000","last_unstale":"2024-04-26T23:43:29.197680+0000","last_undegraded":"2024-04-26T23:43:29.197680+0000","last_fullsized":"2024-04-26T23:43:29.197680+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:23:27.570603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.417670+0000","last_change":"2024-04-26T23:43:03.300758+0000","last_active":"2024-04-26T23:43:08.417670+0000","last_peered":"2024-04-26T23:43:08.417670+0000","last_clean":"2024-04-26T23:43:08.417670+0000","last_became_active":"2024-04-26T23:43:03.300193+0000","last_became_peered":"2024-04-26T23:43:03.300193+0000","last_unstale":"2024-04-26T23:43:08.417670+0000","last_undegraded":"2024-04-26T23:43:08.417670+0000","last_fullsized":"2024-04-26T23:43:08.417670+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:37:43.289361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.417265+0000","last_change":"2024-04-26T23:43:07.546026+0000","last_active":"2024-04-26T23:43:08.417265+0000","last_peered":"2024-04-26T23:43:08.417265+0000","last_clean":"2024-04-26T23:43:08.417265+0000","last_became_active":"2024-04-26T23:43:07.545668+0000","last_became_peered":"2024-04-26T23:43:07.545668+0000","last_unstale":"2024-04-26T23:43:08.417265+0000","last_undegraded":"2024-04-26T23:43:08.417265+0000","last_fullsized":"2024-04-26T23:43:08.417265+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:54:20.580267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:29.203088+0000","last_change":"2024-04-26T23:43:02.454937+0000","last_active":"2024-04-26T23:43:29.203088+0000","last_peered":"2024-04-26T23:43:29.203088+0000","last_clean":"2024-04-26T23:43:29.203088+0000","last_became_active":"2024-04-26T23:43:02.454585+0000","last_became_peered":"2024-04-26T23:43:02.454585+0000","last_unstale":"2024-04-26T23:43:29.203088+0000","last_undegraded":"2024-04-26T23:43:29.203088+0000","last_fullsized":"2024-04-26T23:43:29.203088+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:55:54.999411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.419333+0000","last_change":"2024-04-26T23:43:03.312622+0000","last_active":"2024-04-26T23:43:08.419333+0000","last_peered":"2024-04-26T23:43:08.419333+0000","last_clean":"2024-04-26T23:43:08.419333+0000","last_became_active":"2024-04-26T23:43:03.310774+0000","last_became_peered":"2024-04-26T23:43:03.310774+0000","last_unstale":"2024-04-26T23:43:08.419333+0000","last_undegraded":"2024-04-26T23:43:08.419333+0000","last_fullsized":"2024-04-26T23:43:08.419333+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:31:11.627552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.165398+0000","last_change":"2024-04-26T23:43:07.535682+0000","last_active":"2024-04-26T23:43:08.165398+0000","last_peered":"2024-04-26T23:43:08.165398+0000","last_clean":"2024-04-26T23:43:08.165398+0000","last_became_active":"2024-04-26T23:43:07.535221+0000","last_became_peered":"2024-04-26T23:43:07.535221+0000","last_unstale":"2024-04-26T23:43:08.165398+0000","last_undegraded":"2024-04-26T23:43:08.165398+0000","last_fullsized":"2024-04-26T23:43:08.165398+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:33:54.105329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.417216+0000","last_change":"2024-04-26T23:43:02.443541+0000","last_active":"2024-04-26T23:43:08.417216+0000","last_peered":"2024-04-26T23:43:08.417216+0000","last_clean":"2024-04-26T23:43:08.417216+0000","last_became_active":"2024-04-26T23:43:02.442948+0000","last_became_peered":"2024-04-26T23:43:02.442948+0000","last_unstale":"2024-04-26T23:43:08.417216+0000","last_undegraded":"2024-04-26T23:43:08.417216+0000","last_fullsized":"2024-04-26T23:43:08.417216+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:58:44.595575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.422169+0000","last_change":"2024-04-26T23:43:03.311029+0000","last_active":"2024-04-26T23:43:08.422169+0000","last_peered":"2024-04-26T23:43:08.422169+0000","last_clean":"2024-04-26T23:43:08.422169+0000","last_became_active":"2024-04-26T23:43:03.310722+0000","last_became_peered":"2024-04-26T23:43:03.310722+0000","last_unstale":"2024-04-26T23:43:08.422169+0000","last_undegraded":"2024-04-26T23:43:08.422169+0000","last_fullsized":"2024-04-26T23:43:08.422169+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:07:54.449492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.421647+0000","last_change":"2024-04-26T23:43:07.537391+0000","last_active":"2024-04-26T23:43:08.421647+0000","last_peered":"2024-04-26T23:43:08.421647+0000","last_clean":"2024-04-26T23:43:08.421647+0000","last_became_active":"2024-04-26T23:43:07.536216+0000","last_became_peered":"2024-04-26T23:43:07.536216+0000","last_unstale":"2024-04-26T23:43:08.421647+0000","last_undegraded":"2024-04-26T23:43:08.421647+0000","last_fullsized":"2024-04-26T23:43:08.421647+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:45:55.326283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:32.203594+0000","last_change":"2024-04-26T23:43:02.451929+0000","last_active":"2024-04-26T23:43:32.203594+0000","last_peered":"2024-04-26T23:43:32.203594+0000","last_clean":"2024-04-26T23:43:32.203594+0000","last_became_active":"2024-04-26T23:43:02.451700+0000","last_became_peered":"2024-04-26T23:43:02.451700+0000","last_unstale":"2024-04-26T23:43:32.203594+0000","last_undegraded":"2024-04-26T23:43:32.203594+0000","last_fullsized":"2024-04-26T23:43:32.203594+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:21:31.279681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.238028+0000","last_change":"2024-04-26T23:43:03.295462+0000","last_active":"2024-04-26T23:43:08.238028+0000","last_peered":"2024-04-26T23:43:08.238028+0000","last_clean":"2024-04-26T23:43:08.238028+0000","last_became_active":"2024-04-26T23:43:03.295044+0000","last_became_peered":"2024-04-26T23:43:03.295044+0000","last_unstale":"2024-04-26T23:43:08.238028+0000","last_undegraded":"2024-04-26T23:43:08.238028+0000","last_fullsized":"2024-04-26T23:43:08.238028+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:56:19.445166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.419113+0000","last_change":"2024-04-26T23:43:07.539397+0000","last_active":"2024-04-26T23:43:08.419113+0000","last_peered":"2024-04-26T23:43:08.419113+0000","last_clean":"2024-04-26T23:43:08.419113+0000","last_became_active":"2024-04-26T23:43:07.533081+0000","last_became_peered":"2024-04-26T23:43:07.533081+0000","last_unstale":"2024-04-26T23:43:08.419113+0000","last_undegraded":"2024-04-26T23:43:08.419113+0000","last_fullsized":"2024-04-26T23:43:08.419113+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:24:39.693721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:28.186458+0000","last_change":"2024-04-26T23:43:02.439919+0000","last_active":"2024-04-26T23:43:28.186458+0000","last_peered":"2024-04-26T23:43:28.186458+0000","last_clean":"2024-04-26T23:43:28.186458+0000","last_became_active":"2024-04-26T23:43:02.439561+0000","last_became_peered":"2024-04-26T23:43:02.439561+0000","last_unstale":"2024-04-26T23:43:28.186458+0000","last_undegraded":"2024-04-26T23:43:28.186458+0000","last_fullsized":"2024-04-26T23:43:28.186458+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:30:22.905464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.165717+0000","last_change":"2024-04-26T23:43:03.294088+0000","last_active":"2024-04-26T23:43:08.165717+0000","last_peered":"2024-04-26T23:43:08.165717+0000","last_clean":"2024-04-26T23:43:08.165717+0000","last_became_active":"2024-04-26T23:43:03.293726+0000","last_became_peered":"2024-04-26T23:43:03.293726+0000","last_unstale":"2024-04-26T23:43:08.165717+0000","last_undegraded":"2024-04-26T23:43:08.165717+0000","last_fullsized":"2024-04-26T23:43:08.165717+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:19:42.575193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.416554+0000","last_change":"2024-04-26T23:43:07.543056+0000","last_active":"2024-04-26T23:43:08.416554+0000","last_peered":"2024-04-26T23:43:08.416554+0000","last_clean":"2024-04-26T23:43:08.416554+0000","last_became_active":"2024-04-26T23:43:07.542324+0000","last_became_peered":"2024-04-26T23:43:07.542324+0000","last_unstale":"2024-04-26T23:43:08.416554+0000","last_undegraded":"2024-04-26T23:43:08.416554+0000","last_fullsized":"2024-04-26T23:43:08.416554+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:23:46.415932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:28.188110+0000","last_change":"2024-04-26T23:43:02.454135+0000","last_active":"2024-04-26T23:43:28.188110+0000","last_peered":"2024-04-26T23:43:28.188110+0000","last_clean":"2024-04-26T23:43:28.188110+0000","last_became_active":"2024-04-26T23:43:02.453798+0000","last_became_peered":"2024-04-26T23:43:02.453798+0000","last_unstale":"2024-04-26T23:43:28.188110+0000","last_undegraded":"2024-04-26T23:43:28.188110+0000","last_fullsized":"2024-04-26T23:43:28.188110+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:16:25.286569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.237924+0000","last_change":"2024-04-26T23:43:03.293520+0000","last_active":"2024-04-26T23:43:08.237924+0000","last_peered":"2024-04-26T23:43:08.237924+0000","last_clean":"2024-04-26T23:43:08.237924+0000","last_became_active":"2024-04-26T23:43:03.292537+0000","last_became_peered":"2024-04-26T23:43:03.292537+0000","last_unstale":"2024-04-26T23:43:08.237924+0000","last_undegraded":"2024-04-26T23:43:08.237924+0000","last_fullsized":"2024-04-26T23:43:08.237924+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:19:28.808217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.419616+0000","last_change":"2024-04-26T23:43:07.528116+0000","last_active":"2024-04-26T23:43:08.419616+0000","last_peered":"2024-04-26T23:43:08.419616+0000","last_clean":"2024-04-26T23:43:08.419616+0000","last_became_active":"2024-04-26T23:43:07.527766+0000","last_became_peered":"2024-04-26T23:43:07.527766+0000","last_unstale":"2024-04-26T23:43:08.419616+0000","last_undegraded":"2024-04-26T23:43:08.419616+0000","last_fullsized":"2024-04-26T23:43:08.419616+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:39:23.949964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'24","reported_seq":58,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:33.210690+0000","last_change":"2024-04-26T23:43:02.437454+0000","last_active":"2024-04-26T23:43:33.210690+0000","last_peered":"2024-04-26T23:43:33.210690+0000","last_clean":"2024-04-26T23:43:33.210690+0000","last_became_active":"2024-04-26T23:43:02.437142+0000","last_became_peered":"2024-04-26T23:43:02.437142+0000","last_unstale":"2024-04-26T23:43:33.210690+0000","last_undegraded":"2024-04-26T23:43:33.210690+0000","last_fullsized":"2024-04-26T23:43:33.210690+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:06:37.751066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40471,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":12,"num_read_kb":45,"num_write":25,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.417458+0000","last_change":"2024-04-26T23:43:03.287831+0000","last_active":"2024-04-26T23:43:08.417458+0000","last_peered":"2024-04-26T23:43:08.417458+0000","last_clean":"2024-04-26T23:43:08.417458+0000","last_became_active":"2024-04-26T23:43:03.287479+0000","last_became_peered":"2024-04-26T23:43:03.287479+0000","last_unstale":"2024-04-26T23:43:08.417458+0000","last_undegraded":"2024-04-26T23:43:08.417458+0000","last_fullsized":"2024-04-26T23:43:08.417458+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:39:41.726893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.418327+0000","last_change":"2024-04-26T23:43:07.543985+0000","last_active":"2024-04-26T23:43:08.418327+0000","last_peered":"2024-04-26T23:43:08.418327+0000","last_clean":"2024-04-26T23:43:08.418327+0000","last_became_active":"2024-04-26T23:43:07.543327+0000","last_became_peered":"2024-04-26T23:43:07.543327+0000","last_unstale":"2024-04-26T23:43:08.418327+0000","last_undegraded":"2024-04-26T23:43:08.418327+0000","last_fullsized":"2024-04-26T23:43:08.418327+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:24:10.577822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:29.202829+0000","last_change":"2024-04-26T23:43:02.447627+0000","last_active":"2024-04-26T23:43:29.202829+0000","last_peered":"2024-04-26T23:43:29.202829+0000","last_clean":"2024-04-26T23:43:29.202829+0000","last_became_active":"2024-04-26T23:43:02.447325+0000","last_became_peered":"2024-04-26T23:43:02.447325+0000","last_unstale":"2024-04-26T23:43:29.202829+0000","last_undegraded":"2024-04-26T23:43:29.202829+0000","last_fullsized":"2024-04-26T23:43:29.202829+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:19:03.757202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.421374+0000","last_change":"2024-04-26T23:43:03.313388+0000","last_active":"2024-04-26T23:43:08.421374+0000","last_peered":"2024-04-26T23:43:08.421374+0000","last_clean":"2024-04-26T23:43:08.421374+0000","last_became_active":"2024-04-26T23:43:03.312899+0000","last_became_peered":"2024-04-26T23:43:03.312899+0000","last_unstale":"2024-04-26T23:43:08.421374+0000","last_undegraded":"2024-04-26T23:43:08.421374+0000","last_fullsized":"2024-04-26T23:43:08.421374+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:58:37.126485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.416704+0000","last_change":"2024-04-26T23:43:07.536706+0000","last_active":"2024-04-26T23:43:08.416704+0000","last_peered":"2024-04-26T23:43:08.416704+0000","last_clean":"2024-04-26T23:43:08.416704+0000","last_became_active":"2024-04-26T23:43:07.536316+0000","last_became_peered":"2024-04-26T23:43:07.536316+0000","last_unstale":"2024-04-26T23:43:08.416704+0000","last_undegraded":"2024-04-26T23:43:08.416704+0000","last_fullsized":"2024-04-26T23:43:08.416704+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:36:46.397725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:28.188765+0000","last_change":"2024-04-26T23:43:02.448647+0000","last_active":"2024-04-26T23:43:28.188765+0000","last_peered":"2024-04-26T23:43:28.188765+0000","last_clean":"2024-04-26T23:43:28.188765+0000","last_became_active":"2024-04-26T23:43:02.448392+0000","last_became_peered":"2024-04-26T23:43:02.448392+0000","last_unstale":"2024-04-26T23:43:28.188765+0000","last_undegraded":"2024-04-26T23:43:28.188765+0000","last_fullsized":"2024-04-26T23:43:28.188765+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:19:01.367361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.162983+0000","last_change":"2024-04-26T23:43:03.302980+0000","last_active":"2024-04-26T23:43:08.162983+0000","last_peered":"2024-04-26T23:43:08.162983+0000","last_clean":"2024-04-26T23:43:08.162983+0000","last_became_active":"2024-04-26T23:43:03.302589+0000","last_became_peered":"2024-04-26T23:43:03.302589+0000","last_unstale":"2024-04-26T23:43:08.162983+0000","last_undegraded":"2024-04-26T23:43:08.162983+0000","last_fullsized":"2024-04-26T23:43:08.162983+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:57:18.641063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.423939+0000","last_change":"2024-04-26T23:43:07.547842+0000","last_active":"2024-04-26T23:43:08.423939+0000","last_peered":"2024-04-26T23:43:08.423939+0000","last_clean":"2024-04-26T23:43:08.423939+0000","last_became_active":"2024-04-26T23:43:07.541774+0000","last_became_peered":"2024-04-26T23:43:07.541774+0000","last_unstale":"2024-04-26T23:43:08.423939+0000","last_undegraded":"2024-04-26T23:43:08.423939+0000","last_fullsized":"2024-04-26T23:43:08.423939+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:25:49.999465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'4","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:32.208055+0000","last_change":"2024-04-26T23:43:02.437210+0000","last_active":"2024-04-26T23:43:32.208055+0000","last_peered":"2024-04-26T23:43:32.208055+0000","last_clean":"2024-04-26T23:43:32.208055+0000","last_became_active":"2024-04-26T23:43:02.436886+0000","last_became_peered":"2024-04-26T23:43:02.436886+0000","last_unstale":"2024-04-26T23:43:32.208055+0000","last_undegraded":"2024-04-26T23:43:32.208055+0000","last_fullsized":"2024-04-26T23:43:32.208055+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:07:03.663757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.162627+0000","last_change":"2024-04-26T23:43:03.293770+0000","last_active":"2024-04-26T23:43:08.162627+0000","last_peered":"2024-04-26T23:43:08.162627+0000","last_clean":"2024-04-26T23:43:08.162627+0000","last_became_active":"2024-04-26T23:43:03.293297+0000","last_became_peered":"2024-04-26T23:43:03.293297+0000","last_unstale":"2024-04-26T23:43:08.162627+0000","last_undegraded":"2024-04-26T23:43:08.162627+0000","last_fullsized":"2024-04-26T23:43:08.162627+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:01:42.339059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.423658+0000","last_change":"2024-04-26T23:43:07.545626+0000","last_active":"2024-04-26T23:43:08.423658+0000","last_peered":"2024-04-26T23:43:08.423658+0000","last_clean":"2024-04-26T23:43:08.423658+0000","last_became_active":"2024-04-26T23:43:07.545171+0000","last_became_peered":"2024-04-26T23:43:07.545171+0000","last_unstale":"2024-04-26T23:43:08.423658+0000","last_undegraded":"2024-04-26T23:43:08.423658+0000","last_fullsized":"2024-04-26T23:43:08.423658+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:23:33.829834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:29.204830+0000","last_change":"2024-04-26T23:43:02.457426+0000","last_active":"2024-04-26T23:43:29.204830+0000","last_peered":"2024-04-26T23:43:29.204830+0000","last_clean":"2024-04-26T23:43:29.204830+0000","last_became_active":"2024-04-26T23:43:02.457008+0000","last_became_peered":"2024-04-26T23:43:02.457008+0000","last_unstale":"2024-04-26T23:43:29.204830+0000","last_undegraded":"2024-04-26T23:43:29.204830+0000","last_fullsized":"2024-04-26T23:43:29.204830+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:32:45.324729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.417741+0000","last_change":"2024-04-26T23:43:03.290567+0000","last_active":"2024-04-26T23:43:08.417741+0000","last_peered":"2024-04-26T23:43:08.417741+0000","last_clean":"2024-04-26T23:43:08.417741+0000","last_became_active":"2024-04-26T23:43:03.287807+0000","last_became_peered":"2024-04-26T23:43:03.287807+0000","last_unstale":"2024-04-26T23:43:08.417741+0000","last_undegraded":"2024-04-26T23:43:08.417741+0000","last_fullsized":"2024-04-26T23:43:08.417741+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:25:24.393978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.418176+0000","last_change":"2024-04-26T23:43:07.535402+0000","last_active":"2024-04-26T23:43:08.418176+0000","last_peered":"2024-04-26T23:43:08.418176+0000","last_clean":"2024-04-26T23:43:08.418176+0000","last_became_active":"2024-04-26T23:43:07.534870+0000","last_became_peered":"2024-04-26T23:43:07.534870+0000","last_unstale":"2024-04-26T23:43:08.418176+0000","last_undegraded":"2024-04-26T23:43:08.418176+0000","last_fullsized":"2024-04-26T23:43:08.418176+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:26:37.703831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:30.193275+0000","last_change":"2024-04-26T23:43:02.440638+0000","last_active":"2024-04-26T23:43:30.193275+0000","last_peered":"2024-04-26T23:43:30.193275+0000","last_clean":"2024-04-26T23:43:30.193275+0000","last_became_active":"2024-04-26T23:43:02.440294+0000","last_became_peered":"2024-04-26T23:43:02.440294+0000","last_unstale":"2024-04-26T23:43:30.193275+0000","last_undegraded":"2024-04-26T23:43:30.193275+0000","last_fullsized":"2024-04-26T23:43:30.193275+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:21:40.083946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.416976+0000","last_change":"2024-04-26T23:43:03.286679+0000","last_active":"2024-04-26T23:43:08.416976+0000","last_peered":"2024-04-26T23:43:08.416976+0000","last_clean":"2024-04-26T23:43:08.416976+0000","last_became_active":"2024-04-26T23:43:03.286227+0000","last_became_peered":"2024-04-26T23:43:03.286227+0000","last_unstale":"2024-04-26T23:43:08.416976+0000","last_undegraded":"2024-04-26T23:43:08.416976+0000","last_fullsized":"2024-04-26T23:43:08.416976+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:18:26.934987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.417036+0000","last_change":"2024-04-26T23:43:07.855931+0000","last_active":"2024-04-26T23:43:08.417036+0000","last_peered":"2024-04-26T23:43:08.417036+0000","last_clean":"2024-04-26T23:43:08.417036+0000","last_became_active":"2024-04-26T23:43:07.855407+0000","last_became_peered":"2024-04-26T23:43:07.855407+0000","last_unstale":"2024-04-26T23:43:08.417036+0000","last_undegraded":"2024-04-26T23:43:08.417036+0000","last_fullsized":"2024-04-26T23:43:08.417036+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:39:25.707837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.423544+0000","last_change":"2024-04-26T23:43:02.457971+0000","last_active":"2024-04-26T23:43:08.423544+0000","last_peered":"2024-04-26T23:43:08.423544+0000","last_clean":"2024-04-26T23:43:08.423544+0000","last_became_active":"2024-04-26T23:43:02.457627+0000","last_became_peered":"2024-04-26T23:43:02.457627+0000","last_unstale":"2024-04-26T23:43:08.423544+0000","last_undegraded":"2024-04-26T23:43:08.423544+0000","last_fullsized":"2024-04-26T23:43:08.423544+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:25:51.749774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.417114+0000","last_change":"2024-04-26T23:43:03.305569+0000","last_active":"2024-04-26T23:43:08.417114+0000","last_peered":"2024-04-26T23:43:08.417114+0000","last_clean":"2024-04-26T23:43:08.417114+0000","last_became_active":"2024-04-26T23:43:03.305211+0000","last_became_peered":"2024-04-26T23:43:03.305211+0000","last_unstale":"2024-04-26T23:43:08.417114+0000","last_undegraded":"2024-04-26T23:43:08.417114+0000","last_fullsized":"2024-04-26T23:43:08.417114+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:54:55.847680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.421124+0000","last_change":"2024-04-26T23:43:07.535408+0000","last_active":"2024-04-26T23:43:08.421124+0000","last_peered":"2024-04-26T23:43:08.421124+0000","last_clean":"2024-04-26T23:43:08.421124+0000","last_became_active":"2024-04-26T23:43:07.534957+0000","last_became_peered":"2024-04-26T23:43:07.534957+0000","last_unstale":"2024-04-26T23:43:08.421124+0000","last_undegraded":"2024-04-26T23:43:08.421124+0000","last_fullsized":"2024-04-26T23:43:08.421124+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:35:31.103532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.162656+0000","last_change":"2024-04-26T23:43:02.453636+0000","last_active":"2024-04-26T23:43:08.162656+0000","last_peered":"2024-04-26T23:43:08.162656+0000","last_clean":"2024-04-26T23:43:08.162656+0000","last_became_active":"2024-04-26T23:43:02.453310+0000","last_became_peered":"2024-04-26T23:43:02.453310+0000","last_unstale":"2024-04-26T23:43:08.162656+0000","last_undegraded":"2024-04-26T23:43:08.162656+0000","last_fullsized":"2024-04-26T23:43:08.162656+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:42:29.295303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.416769+0000","last_change":"2024-04-26T23:43:03.283184+0000","last_active":"2024-04-26T23:43:08.416769+0000","last_peered":"2024-04-26T23:43:08.416769+0000","last_clean":"2024-04-26T23:43:08.416769+0000","last_became_active":"2024-04-26T23:43:03.282780+0000","last_became_peered":"2024-04-26T23:43:03.282780+0000","last_unstale":"2024-04-26T23:43:08.416769+0000","last_undegraded":"2024-04-26T23:43:08.416769+0000","last_fullsized":"2024-04-26T23:43:08.416769+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:00:13.907101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.422981+0000","last_change":"2024-04-26T23:43:07.859107+0000","last_active":"2024-04-26T23:43:08.422981+0000","last_peered":"2024-04-26T23:43:08.422981+0000","last_clean":"2024-04-26T23:43:08.422981+0000","last_became_active":"2024-04-26T23:43:07.856492+0000","last_became_peered":"2024-04-26T23:43:07.856492+0000","last_unstale":"2024-04-26T23:43:08.422981+0000","last_undegraded":"2024-04-26T23:43:08.422981+0000","last_fullsized":"2024-04-26T23:43:08.422981+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:13:36.570398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.417094+0000","last_change":"2024-04-26T23:43:03.289078+0000","last_active":"2024-04-26T23:43:08.417094+0000","last_peered":"2024-04-26T23:43:08.417094+0000","last_clean":"2024-04-26T23:43:08.417094+0000","last_became_active":"2024-04-26T23:43:03.288729+0000","last_became_peered":"2024-04-26T23:43:03.288729+0000","last_unstale":"2024-04-26T23:43:08.417094+0000","last_undegraded":"2024-04-26T23:43:08.417094+0000","last_fullsized":"2024-04-26T23:43:08.417094+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:32:36.346472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:32.207574+0000","last_change":"2024-04-26T23:43:02.463355+0000","last_active":"2024-04-26T23:43:32.207574+0000","last_peered":"2024-04-26T23:43:32.207574+0000","last_clean":"2024-04-26T23:43:32.207574+0000","last_became_active":"2024-04-26T23:43:02.455832+0000","last_became_peered":"2024-04-26T23:43:02.455832+0000","last_unstale":"2024-04-26T23:43:32.207574+0000","last_undegraded":"2024-04-26T23:43:32.207574+0000","last_fullsized":"2024-04-26T23:43:32.207574+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:33:33.837999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.163277+0000","last_change":"2024-04-26T23:43:07.540777+0000","last_active":"2024-04-26T23:43:08.163277+0000","last_peered":"2024-04-26T23:43:08.163277+0000","last_clean":"2024-04-26T23:43:08.163277+0000","last_became_active":"2024-04-26T23:43:07.540389+0000","last_became_peered":"2024-04-26T23:43:07.540389+0000","last_unstale":"2024-04-26T23:43:08.163277+0000","last_undegraded":"2024-04-26T23:43:08.163277+0000","last_fullsized":"2024-04-26T23:43:08.163277+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:57:28.674051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.418640+0000","last_change":"2024-04-26T23:43:03.308391+0000","last_active":"2024-04-26T23:43:08.418640+0000","last_peered":"2024-04-26T23:43:08.418640+0000","last_clean":"2024-04-26T23:43:08.418640+0000","last_became_active":"2024-04-26T23:43:03.306762+0000","last_became_peered":"2024-04-26T23:43:03.306762+0000","last_unstale":"2024-04-26T23:43:08.418640+0000","last_undegraded":"2024-04-26T23:43:08.418640+0000","last_fullsized":"2024-04-26T23:43:08.418640+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:34:01.083036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:30.197513+0000","last_change":"2024-04-26T23:43:02.448037+0000","last_active":"2024-04-26T23:43:30.197513+0000","last_peered":"2024-04-26T23:43:30.197513+0000","last_clean":"2024-04-26T23:43:30.197513+0000","last_became_active":"2024-04-26T23:43:02.447706+0000","last_became_peered":"2024-04-26T23:43:02.447706+0000","last_unstale":"2024-04-26T23:43:30.197513+0000","last_undegraded":"2024-04-26T23:43:30.197513+0000","last_fullsized":"2024-04-26T23:43:30.197513+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:43:56.295191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":343,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.162465+0000","last_change":"2024-04-26T23:41:42.266448+0000","last_active":"2024-04-26T23:43:08.162465+0000","last_peered":"2024-04-26T23:43:08.162465+0000","last_clean":"2024-04-26T23:43:08.162465+0000","last_became_active":"2024-04-26T23:41:42.265898+0000","last_became_peered":"2024-04-26T23:41:42.265898+0000","last_unstale":"2024-04-26T23:43:08.162465+0000","last_undegraded":"2024-04-26T23:43:08.162465+0000","last_fullsized":"2024-04-26T23:43:08.162465+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:37:44.460409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:37:44.460409+0000","last_clean_scrub_stamp":"2024-04-26T23:37:44.460409+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:40:48.177191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.417876+0000","last_change":"2024-04-26T23:43:07.856089+0000","last_active":"2024-04-26T23:43:08.417876+0000","last_peered":"2024-04-26T23:43:08.417876+0000","last_clean":"2024-04-26T23:43:08.417876+0000","last_became_active":"2024-04-26T23:43:07.855542+0000","last_became_peered":"2024-04-26T23:43:07.855542+0000","last_unstale":"2024-04-26T23:43:08.417876+0000","last_undegraded":"2024-04-26T23:43:08.417876+0000","last_fullsized":"2024-04-26T23:43:08.417876+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:48:22.799242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.419122+0000","last_change":"2024-04-26T23:43:03.301989+0000","last_active":"2024-04-26T23:43:08.419122+0000","last_peered":"2024-04-26T23:43:08.419122+0000","last_clean":"2024-04-26T23:43:08.419122+0000","last_became_active":"2024-04-26T23:43:03.301179+0000","last_became_peered":"2024-04-26T23:43:03.301179+0000","last_unstale":"2024-04-26T23:43:08.419122+0000","last_undegraded":"2024-04-26T23:43:08.419122+0000","last_fullsized":"2024-04-26T23:43:08.419122+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:29:33.012009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:29.193899+0000","last_change":"2024-04-26T23:43:02.439648+0000","last_active":"2024-04-26T23:43:29.193899+0000","last_peered":"2024-04-26T23:43:29.193899+0000","last_clean":"2024-04-26T23:43:29.193899+0000","last_became_active":"2024-04-26T23:43:02.439340+0000","last_became_peered":"2024-04-26T23:43:02.439340+0000","last_unstale":"2024-04-26T23:43:29.193899+0000","last_undegraded":"2024-04-26T23:43:29.193899+0000","last_fullsized":"2024-04-26T23:43:29.193899+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:10:01.096657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.416648+0000","last_change":"2024-04-26T23:43:07.534317+0000","last_active":"2024-04-26T23:43:08.416648+0000","last_peered":"2024-04-26T23:43:08.416648+0000","last_clean":"2024-04-26T23:43:08.416648+0000","last_became_active":"2024-04-26T23:43:07.533808+0000","last_became_peered":"2024-04-26T23:43:07.533808+0000","last_unstale":"2024-04-26T23:43:08.416648+0000","last_undegraded":"2024-04-26T23:43:08.416648+0000","last_fullsized":"2024-04-26T23:43:08.416648+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:54:29.952089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:28.187645+0000","last_change":"2024-04-26T23:43:02.457501+0000","last_active":"2024-04-26T23:43:28.187645+0000","last_peered":"2024-04-26T23:43:28.187645+0000","last_clean":"2024-04-26T23:43:28.187645+0000","last_became_active":"2024-04-26T23:43:02.457024+0000","last_became_peered":"2024-04-26T23:43:02.457024+0000","last_unstale":"2024-04-26T23:43:28.187645+0000","last_undegraded":"2024-04-26T23:43:28.187645+0000","last_fullsized":"2024-04-26T23:43:28.187645+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:21:58.542186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.423324+0000","last_change":"2024-04-26T23:43:03.303434+0000","last_active":"2024-04-26T23:43:08.423324+0000","last_peered":"2024-04-26T23:43:08.423324+0000","last_clean":"2024-04-26T23:43:08.423324+0000","last_became_active":"2024-04-26T23:43:03.302455+0000","last_became_peered":"2024-04-26T23:43:03.302455+0000","last_unstale":"2024-04-26T23:43:08.423324+0000","last_undegraded":"2024-04-26T23:43:08.423324+0000","last_fullsized":"2024-04-26T23:43:08.423324+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:57:05.530635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.418642+0000","last_change":"2024-04-26T23:43:07.856932+0000","last_active":"2024-04-26T23:43:08.418642+0000","last_peered":"2024-04-26T23:43:08.418642+0000","last_clean":"2024-04-26T23:43:08.418642+0000","last_became_active":"2024-04-26T23:43:07.855699+0000","last_became_peered":"2024-04-26T23:43:07.855699+0000","last_unstale":"2024-04-26T23:43:08.418642+0000","last_undegraded":"2024-04-26T23:43:08.418642+0000","last_fullsized":"2024-04-26T23:43:08.418642+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:20:28.994301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:29.201401+0000","last_change":"2024-04-26T23:43:02.449852+0000","last_active":"2024-04-26T23:43:29.201401+0000","last_peered":"2024-04-26T23:43:29.201401+0000","last_clean":"2024-04-26T23:43:29.201401+0000","last_became_active":"2024-04-26T23:43:02.449422+0000","last_became_peered":"2024-04-26T23:43:02.449422+0000","last_unstale":"2024-04-26T23:43:29.201401+0000","last_undegraded":"2024-04-26T23:43:29.201401+0000","last_fullsized":"2024-04-26T23:43:29.201401+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:07:25.453872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.421490+0000","last_change":"2024-04-26T23:43:03.304733+0000","last_active":"2024-04-26T23:43:08.421490+0000","last_peered":"2024-04-26T23:43:08.421490+0000","last_clean":"2024-04-26T23:43:08.421490+0000","last_became_active":"2024-04-26T23:43:03.304383+0000","last_became_peered":"2024-04-26T23:43:03.304383+0000","last_unstale":"2024-04-26T23:43:08.421490+0000","last_undegraded":"2024-04-26T23:43:08.421490+0000","last_fullsized":"2024-04-26T23:43:08.421490+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:49:28.679255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.162339+0000","last_change":"2024-04-26T23:43:07.525310+0000","last_active":"2024-04-26T23:43:08.162339+0000","last_peered":"2024-04-26T23:43:08.162339+0000","last_clean":"2024-04-26T23:43:08.162339+0000","last_became_active":"2024-04-26T23:43:07.524502+0000","last_became_peered":"2024-04-26T23:43:07.524502+0000","last_unstale":"2024-04-26T23:43:08.162339+0000","last_undegraded":"2024-04-26T23:43:08.162339+0000","last_fullsized":"2024-04-26T23:43:08.162339+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:07:45.927837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:28.192326+0000","last_change":"2024-04-26T23:43:02.446238+0000","last_active":"2024-04-26T23:43:28.192326+0000","last_peered":"2024-04-26T23:43:28.192326+0000","last_clean":"2024-04-26T23:43:28.192326+0000","last_became_active":"2024-04-26T23:43:02.445959+0000","last_became_peered":"2024-04-26T23:43:02.445959+0000","last_unstale":"2024-04-26T23:43:28.192326+0000","last_undegraded":"2024-04-26T23:43:28.192326+0000","last_fullsized":"2024-04-26T23:43:28.192326+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:25:10.007481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.165423+0000","last_change":"2024-04-26T23:43:03.306035+0000","last_active":"2024-04-26T23:43:08.165423+0000","last_peered":"2024-04-26T23:43:08.165423+0000","last_clean":"2024-04-26T23:43:08.165423+0000","last_became_active":"2024-04-26T23:43:03.305604+0000","last_became_peered":"2024-04-26T23:43:03.305604+0000","last_unstale":"2024-04-26T23:43:08.165423+0000","last_undegraded":"2024-04-26T23:43:08.165423+0000","last_fullsized":"2024-04-26T23:43:08.165423+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:02:38.567754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.416327+0000","last_change":"2024-04-26T23:43:07.855808+0000","last_active":"2024-04-26T23:43:08.416327+0000","last_peered":"2024-04-26T23:43:08.416327+0000","last_clean":"2024-04-26T23:43:08.416327+0000","last_became_active":"2024-04-26T23:43:07.855266+0000","last_became_peered":"2024-04-26T23:43:07.855266+0000","last_unstale":"2024-04-26T23:43:08.416327+0000","last_undegraded":"2024-04-26T23:43:08.416327+0000","last_fullsized":"2024-04-26T23:43:08.416327+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:06:03.016504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:28.193197+0000","last_change":"2024-04-26T23:43:02.443464+0000","last_active":"2024-04-26T23:43:28.193197+0000","last_peered":"2024-04-26T23:43:28.193197+0000","last_clean":"2024-04-26T23:43:28.193197+0000","last_became_active":"2024-04-26T23:43:02.443170+0000","last_became_peered":"2024-04-26T23:43:02.443170+0000","last_unstale":"2024-04-26T23:43:28.193197+0000","last_undegraded":"2024-04-26T23:43:28.193197+0000","last_fullsized":"2024-04-26T23:43:28.193197+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:45:51.431082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.416419+0000","last_change":"2024-04-26T23:43:03.282714+0000","last_active":"2024-04-26T23:43:08.416419+0000","last_peered":"2024-04-26T23:43:08.416419+0000","last_clean":"2024-04-26T23:43:08.416419+0000","last_became_active":"2024-04-26T23:43:03.282353+0000","last_became_peered":"2024-04-26T23:43:03.282353+0000","last_unstale":"2024-04-26T23:43:08.416419+0000","last_undegraded":"2024-04-26T23:43:08.416419+0000","last_fullsized":"2024-04-26T23:43:08.416419+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:18:24.082946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.424304+0000","last_change":"2024-04-26T23:43:07.543301+0000","last_active":"2024-04-26T23:43:08.424304+0000","last_peered":"2024-04-26T23:43:08.424304+0000","last_clean":"2024-04-26T23:43:08.424304+0000","last_became_active":"2024-04-26T23:43:07.540434+0000","last_became_peered":"2024-04-26T23:43:07.540434+0000","last_unstale":"2024-04-26T23:43:08.424304+0000","last_undegraded":"2024-04-26T23:43:08.424304+0000","last_fullsized":"2024-04-26T23:43:08.424304+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:15:18.542595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:28.189484+0000","last_change":"2024-04-26T23:43:02.451391+0000","last_active":"2024-04-26T23:43:28.189484+0000","last_peered":"2024-04-26T23:43:28.189484+0000","last_clean":"2024-04-26T23:43:28.189484+0000","last_became_active":"2024-04-26T23:43:02.450735+0000","last_became_peered":"2024-04-26T23:43:02.450735+0000","last_unstale":"2024-04-26T23:43:28.189484+0000","last_undegraded":"2024-04-26T23:43:28.189484+0000","last_fullsized":"2024-04-26T23:43:28.189484+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:57:48.658671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.416029+0000","last_change":"2024-04-26T23:43:03.301858+0000","last_active":"2024-04-26T23:43:08.416029+0000","last_peered":"2024-04-26T23:43:08.416029+0000","last_clean":"2024-04-26T23:43:08.416029+0000","last_became_active":"2024-04-26T23:43:03.301474+0000","last_became_peered":"2024-04-26T23:43:03.301474+0000","last_unstale":"2024-04-26T23:43:08.416029+0000","last_undegraded":"2024-04-26T23:43:08.416029+0000","last_fullsized":"2024-04-26T23:43:08.416029+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:24:56.386127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.540133+0000","last_change":"2024-04-26T23:43:07.544512+0000","last_active":"2024-04-26T23:43:08.540133+0000","last_peered":"2024-04-26T23:43:08.540133+0000","last_clean":"2024-04-26T23:43:08.540133+0000","last_became_active":"2024-04-26T23:43:07.544029+0000","last_became_peered":"2024-04-26T23:43:07.544029+0000","last_unstale":"2024-04-26T23:43:08.540133+0000","last_undegraded":"2024-04-26T23:43:08.540133+0000","last_fullsized":"2024-04-26T23:43:08.540133+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:38:08.149123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.162473+0000","last_change":"2024-04-26T23:43:02.447594+0000","last_active":"2024-04-26T23:43:08.162473+0000","last_peered":"2024-04-26T23:43:08.162473+0000","last_clean":"2024-04-26T23:43:08.162473+0000","last_became_active":"2024-04-26T23:43:02.447267+0000","last_became_peered":"2024-04-26T23:43:02.447267+0000","last_unstale":"2024-04-26T23:43:08.162473+0000","last_undegraded":"2024-04-26T23:43:08.162473+0000","last_fullsized":"2024-04-26T23:43:08.162473+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:42:18.555585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.420337+0000","last_change":"2024-04-26T23:43:03.307895+0000","last_active":"2024-04-26T23:43:08.420337+0000","last_peered":"2024-04-26T23:43:08.420337+0000","last_clean":"2024-04-26T23:43:08.420337+0000","last_became_active":"2024-04-26T23:43:03.307478+0000","last_became_peered":"2024-04-26T23:43:03.307478+0000","last_unstale":"2024-04-26T23:43:08.420337+0000","last_undegraded":"2024-04-26T23:43:08.420337+0000","last_fullsized":"2024-04-26T23:43:08.420337+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:57:14.356234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.416544+0000","last_change":"2024-04-26T23:43:07.855803+0000","last_active":"2024-04-26T23:43:08.416544+0000","last_peered":"2024-04-26T23:43:08.416544+0000","last_clean":"2024-04-26T23:43:08.416544+0000","last_became_active":"2024-04-26T23:43:07.855200+0000","last_became_peered":"2024-04-26T23:43:07.855200+0000","last_unstale":"2024-04-26T23:43:08.416544+0000","last_undegraded":"2024-04-26T23:43:08.416544+0000","last_fullsized":"2024-04-26T23:43:08.416544+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:02:46.223330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:28.184840+0000","last_change":"2024-04-26T23:43:02.437513+0000","last_active":"2024-04-26T23:43:28.184840+0000","last_peered":"2024-04-26T23:43:28.184840+0000","last_clean":"2024-04-26T23:43:28.184840+0000","last_became_active":"2024-04-26T23:43:02.437153+0000","last_became_peered":"2024-04-26T23:43:02.437153+0000","last_unstale":"2024-04-26T23:43:28.184840+0000","last_undegraded":"2024-04-26T23:43:28.184840+0000","last_fullsized":"2024-04-26T23:43:28.184840+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:11:45.477579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.162800+0000","last_change":"2024-04-26T23:43:03.297835+0000","last_active":"2024-04-26T23:43:08.162800+0000","last_peered":"2024-04-26T23:43:08.162800+0000","last_clean":"2024-04-26T23:43:08.162800+0000","last_became_active":"2024-04-26T23:43:03.297481+0000","last_became_peered":"2024-04-26T23:43:03.297481+0000","last_unstale":"2024-04-26T23:43:08.162800+0000","last_undegraded":"2024-04-26T23:43:08.162800+0000","last_fullsized":"2024-04-26T23:43:08.162800+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:52:54.757461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.237891+0000","last_change":"2024-04-26T23:43:07.536849+0000","last_active":"2024-04-26T23:43:08.237891+0000","last_peered":"2024-04-26T23:43:08.237891+0000","last_clean":"2024-04-26T23:43:08.237891+0000","last_became_active":"2024-04-26T23:43:07.536214+0000","last_became_peered":"2024-04-26T23:43:07.536214+0000","last_unstale":"2024-04-26T23:43:08.237891+0000","last_undegraded":"2024-04-26T23:43:08.237891+0000","last_fullsized":"2024-04-26T23:43:08.237891+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:33:16.782368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:29.203867+0000","last_change":"2024-04-26T23:43:02.464181+0000","last_active":"2024-04-26T23:43:29.203867+0000","last_peered":"2024-04-26T23:43:29.203867+0000","last_clean":"2024-04-26T23:43:29.203867+0000","last_became_active":"2024-04-26T23:43:02.463830+0000","last_became_peered":"2024-04-26T23:43:02.463830+0000","last_unstale":"2024-04-26T23:43:29.203867+0000","last_undegraded":"2024-04-26T23:43:29.203867+0000","last_fullsized":"2024-04-26T23:43:29.203867+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:14:52.831302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.163327+0000","last_change":"2024-04-26T23:43:03.298497+0000","last_active":"2024-04-26T23:43:08.163327+0000","last_peered":"2024-04-26T23:43:08.163327+0000","last_clean":"2024-04-26T23:43:08.163327+0000","last_became_active":"2024-04-26T23:43:03.298194+0000","last_became_peered":"2024-04-26T23:43:03.298194+0000","last_unstale":"2024-04-26T23:43:08.163327+0000","last_undegraded":"2024-04-26T23:43:08.163327+0000","last_fullsized":"2024-04-26T23:43:08.163327+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:39:19.007294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.165664+0000","last_change":"2024-04-26T23:43:07.527962+0000","last_active":"2024-04-26T23:43:08.165664+0000","last_peered":"2024-04-26T23:43:08.165664+0000","last_clean":"2024-04-26T23:43:08.165664+0000","last_became_active":"2024-04-26T23:43:07.527386+0000","last_became_peered":"2024-04-26T23:43:07.527386+0000","last_unstale":"2024-04-26T23:43:08.165664+0000","last_undegraded":"2024-04-26T23:43:08.165664+0000","last_fullsized":"2024-04-26T23:43:08.165664+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:57:21.711670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.418151+0000","last_change":"2024-04-26T23:43:02.453988+0000","last_active":"2024-04-26T23:43:08.418151+0000","last_peered":"2024-04-26T23:43:08.418151+0000","last_clean":"2024-04-26T23:43:08.418151+0000","last_became_active":"2024-04-26T23:43:02.453654+0000","last_became_peered":"2024-04-26T23:43:02.453654+0000","last_unstale":"2024-04-26T23:43:08.418151+0000","last_undegraded":"2024-04-26T23:43:08.418151+0000","last_fullsized":"2024-04-26T23:43:08.418151+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:46:08.718705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.162832+0000","last_change":"2024-04-26T23:43:03.305861+0000","last_active":"2024-04-26T23:43:08.162832+0000","last_peered":"2024-04-26T23:43:08.162832+0000","last_clean":"2024-04-26T23:43:08.162832+0000","last_became_active":"2024-04-26T23:43:03.305301+0000","last_became_peered":"2024-04-26T23:43:03.305301+0000","last_unstale":"2024-04-26T23:43:08.162832+0000","last_undegraded":"2024-04-26T23:43:08.162832+0000","last_fullsized":"2024-04-26T23:43:08.162832+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:08:24.174443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.416825+0000","last_change":"2024-04-26T23:43:07.529914+0000","last_active":"2024-04-26T23:43:08.416825+0000","last_peered":"2024-04-26T23:43:08.416825+0000","last_clean":"2024-04-26T23:43:08.416825+0000","last_became_active":"2024-04-26T23:43:07.529509+0000","last_became_peered":"2024-04-26T23:43:07.529509+0000","last_unstale":"2024-04-26T23:43:08.416825+0000","last_undegraded":"2024-04-26T23:43:08.416825+0000","last_fullsized":"2024-04-26T23:43:08.416825+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:50:23.293672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.422976+0000","last_change":"2024-04-26T23:43:02.452096+0000","last_active":"2024-04-26T23:43:08.422976+0000","last_peered":"2024-04-26T23:43:08.422976+0000","last_clean":"2024-04-26T23:43:08.422976+0000","last_became_active":"2024-04-26T23:43:02.450039+0000","last_became_peered":"2024-04-26T23:43:02.450039+0000","last_unstale":"2024-04-26T23:43:08.422976+0000","last_undegraded":"2024-04-26T23:43:08.422976+0000","last_fullsized":"2024-04-26T23:43:08.422976+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:47:03.062575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.163191+0000","last_change":"2024-04-26T23:43:03.294790+0000","last_active":"2024-04-26T23:43:08.163191+0000","last_peered":"2024-04-26T23:43:08.163191+0000","last_clean":"2024-04-26T23:43:08.163191+0000","last_became_active":"2024-04-26T23:43:03.294460+0000","last_became_peered":"2024-04-26T23:43:03.294460+0000","last_unstale":"2024-04-26T23:43:08.163191+0000","last_undegraded":"2024-04-26T23:43:08.163191+0000","last_fullsized":"2024-04-26T23:43:08.163191+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:57:40.879315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.416114+0000","last_change":"2024-04-26T23:43:03.284344+0000","last_active":"2024-04-26T23:43:08.416114+0000","last_peered":"2024-04-26T23:43:08.416114+0000","last_clean":"2024-04-26T23:43:08.416114+0000","last_became_active":"2024-04-26T23:43:03.283954+0000","last_became_peered":"2024-04-26T23:43:03.283954+0000","last_unstale":"2024-04-26T23:43:08.416114+0000","last_undegraded":"2024-04-26T23:43:08.416114+0000","last_fullsized":"2024-04-26T23:43:08.416114+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:12:56.516067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.162510+0000","last_change":"2024-04-26T23:43:03.291959+0000","last_active":"2024-04-26T23:43:08.162510+0000","last_peered":"2024-04-26T23:43:08.162510+0000","last_clean":"2024-04-26T23:43:08.162510+0000","last_became_active":"2024-04-26T23:43:03.291597+0000","last_became_peered":"2024-04-26T23:43:03.291597+0000","last_unstale":"2024-04-26T23:43:08.162510+0000","last_undegraded":"2024-04-26T23:43:08.162510+0000","last_fullsized":"2024-04-26T23:43:08.162510+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:36:27.324979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.162268+0000","last_change":"2024-04-26T23:43:03.309548+0000","last_active":"2024-04-26T23:43:08.162268+0000","last_peered":"2024-04-26T23:43:08.162268+0000","last_clean":"2024-04-26T23:43:08.162268+0000","last_became_active":"2024-04-26T23:43:03.309113+0000","last_became_peered":"2024-04-26T23:43:03.309113+0000","last_unstale":"2024-04-26T23:43:08.162268+0000","last_undegraded":"2024-04-26T23:43:08.162268+0000","last_fullsized":"2024-04-26T23:43:08.162268+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:49:15.720320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.164961+0000","last_change":"2024-04-26T23:43:03.304580+0000","last_active":"2024-04-26T23:43:08.164961+0000","last_peered":"2024-04-26T23:43:08.164961+0000","last_clean":"2024-04-26T23:43:08.164961+0000","last_became_active":"2024-04-26T23:43:03.303128+0000","last_became_peered":"2024-04-26T23:43:03.303128+0000","last_unstale":"2024-04-26T23:43:08.164961+0000","last_undegraded":"2024-04-26T23:43:08.164961+0000","last_fullsized":"2024-04-26T23:43:08.164961+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:11:36.868298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.423531+0000","last_change":"2024-04-26T23:43:03.306441+0000","last_active":"2024-04-26T23:43:08.423531+0000","last_peered":"2024-04-26T23:43:08.423531+0000","last_clean":"2024-04-26T23:43:08.423531+0000","last_became_active":"2024-04-26T23:43:03.305209+0000","last_became_peered":"2024-04-26T23:43:03.305209+0000","last_unstale":"2024-04-26T23:43:08.423531+0000","last_undegraded":"2024-04-26T23:43:08.423531+0000","last_fullsized":"2024-04-26T23:43:08.423531+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:41:39.757001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.416988+0000","last_change":"2024-04-26T23:43:03.284291+0000","last_active":"2024-04-26T23:43:08.416988+0000","last_peered":"2024-04-26T23:43:08.416988+0000","last_clean":"2024-04-26T23:43:08.416988+0000","last_became_active":"2024-04-26T23:43:03.284006+0000","last_became_peered":"2024-04-26T23:43:03.284006+0000","last_unstale":"2024-04-26T23:43:08.416988+0000","last_undegraded":"2024-04-26T23:43:08.416988+0000","last_fullsized":"2024-04-26T23:43:08.416988+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:24:01.386322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.163092+0000","last_change":"2024-04-26T23:43:03.286478+0000","last_active":"2024-04-26T23:43:08.163092+0000","last_peered":"2024-04-26T23:43:08.163092+0000","last_clean":"2024-04-26T23:43:08.163092+0000","last_became_active":"2024-04-26T23:43:03.286115+0000","last_became_peered":"2024-04-26T23:43:03.286115+0000","last_unstale":"2024-04-26T23:43:08.163092+0000","last_undegraded":"2024-04-26T23:43:08.163092+0000","last_fullsized":"2024-04-26T23:43:08.163092+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:34:57.534124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.421801+0000","last_change":"2024-04-26T23:43:03.310549+0000","last_active":"2024-04-26T23:43:08.421801+0000","last_peered":"2024-04-26T23:43:08.421801+0000","last_clean":"2024-04-26T23:43:08.421801+0000","last_became_active":"2024-04-26T23:43:03.309786+0000","last_became_peered":"2024-04-26T23:43:03.309786+0000","last_unstale":"2024-04-26T23:43:08.421801+0000","last_undegraded":"2024-04-26T23:43:08.421801+0000","last_fullsized":"2024-04-26T23:43:08.421801+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:03:33.243368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.416711+0000","last_change":"2024-04-26T23:43:03.300602+0000","last_active":"2024-04-26T23:43:08.416711+0000","last_peered":"2024-04-26T23:43:08.416711+0000","last_clean":"2024-04-26T23:43:08.416711+0000","last_became_active":"2024-04-26T23:43:03.300225+0000","last_became_peered":"2024-04-26T23:43:03.300225+0000","last_unstale":"2024-04-26T23:43:08.416711+0000","last_undegraded":"2024-04-26T23:43:08.416711+0000","last_fullsized":"2024-04-26T23:43:08.416711+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:52:36.638153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.420392+0000","last_change":"2024-04-26T23:43:03.305631+0000","last_active":"2024-04-26T23:43:08.420392+0000","last_peered":"2024-04-26T23:43:08.420392+0000","last_clean":"2024-04-26T23:43:08.420392+0000","last_became_active":"2024-04-26T23:43:03.300193+0000","last_became_peered":"2024-04-26T23:43:03.300193+0000","last_unstale":"2024-04-26T23:43:08.420392+0000","last_undegraded":"2024-04-26T23:43:08.420392+0000","last_fullsized":"2024-04-26T23:43:08.420392+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:27:08.355753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.237844+0000","last_change":"2024-04-26T23:43:03.309514+0000","last_active":"2024-04-26T23:43:08.237844+0000","last_peered":"2024-04-26T23:43:08.237844+0000","last_clean":"2024-04-26T23:43:08.237844+0000","last_became_active":"2024-04-26T23:43:03.309085+0000","last_became_peered":"2024-04-26T23:43:03.309085+0000","last_unstale":"2024-04-26T23:43:08.237844+0000","last_undegraded":"2024-04-26T23:43:08.237844+0000","last_fullsized":"2024-04-26T23:43:08.237844+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:08:45.649513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.422068+0000","last_change":"2024-04-26T23:43:03.303553+0000","last_active":"2024-04-26T23:43:08.422068+0000","last_peered":"2024-04-26T23:43:08.422068+0000","last_clean":"2024-04-26T23:43:08.422068+0000","last_became_active":"2024-04-26T23:43:03.302979+0000","last_became_peered":"2024-04-26T23:43:03.302979+0000","last_unstale":"2024-04-26T23:43:08.422068+0000","last_undegraded":"2024-04-26T23:43:08.422068+0000","last_fullsized":"2024-04-26T23:43:08.422068+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:44:25.809138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-26T23:43:08.416432+0000","last_change":"2024-04-26T23:43:03.288433+0000","last_active":"2024-04-26T23:43:08.416432+0000","last_peered":"2024-04-26T23:43:08.416432+0000","last_clean":"2024-04-26T23:43:08.416432+0000","last_became_active":"2024-04-26T23:43:03.288125+0000","last_became_peered":"2024-04-26T23:43:03.288125+0000","last_unstale":"2024-04-26T23:43:08.416432+0000","last_undegraded":"2024-04-26T23:43:08.416432+0000","last_fullsized":"2024-04-26T23:43:08.416432+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:53:21.673755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":282161,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":44863,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":46,"num_read_kb":71,"num_write":143,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":60158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383700,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":477797,"data_compressed":14636,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645337,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1093632,"data_stored":1110023,"data_compressed":50523,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808862,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":478033,"data_compressed":14636,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070500,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":477953,"data_compressed":14636,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332138,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1028096,"data_stored":1069375,"data_compressed":50523,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495664,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":966656,"data_stored":477305,"data_compressed":14636,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574924,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":481057,"data_compressed":14636,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738449,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":478693,"data_compressed":14636,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411399,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":481713,"data_compressed":14636,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182464,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":978944,"data_stored":477499,"data_compressed":14636,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920828,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":482346,"data_compressed":14636,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757303,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1044480,"data_stored":1073546,"data_compressed":50523,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34242,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34242,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34242,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35887,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35887,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35887,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35887,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35887,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":42440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T23:43:34.532 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-26T23:43:34.533 INFO:tasks.cephadm:Setting up client nodes... 2024-04-26T23:43:34.533 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-26T23:43:34.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:34 smithi136 ceph-mon[33050]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:43:34.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:34 smithi136 ceph-mon[33050]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-04-26T23:43:34.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:34 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/1673951662' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:43:34.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:34 smithi076 ceph-mon[32774]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:43:34.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:34 smithi076 ceph-mon[32774]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-04-26T23:43:34.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:34 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/1673951662' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:43:35.009 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:43:35.496 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:35 smithi072 ceph-mon[28184]: from='client.24853 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T23:43:35.496 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:35 smithi072 ceph-mon[28184]: pgmap v337: 129 pgs: 129 active+clean; 620 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 6.0 KiB/s wr, 11 op/s 2024-04-26T23:43:35.496 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:35 smithi072 ceph-mon[28184]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T23:43:35.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:35 smithi136 ceph-mon[33050]: from='client.24853 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T23:43:35.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:35 smithi136 ceph-mon[33050]: pgmap v337: 129 pgs: 129 active+clean; 620 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 6.0 KiB/s wr, 11 op/s 2024-04-26T23:43:35.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:35 smithi136 ceph-mon[33050]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T23:43:35.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:35 smithi076 ceph-mon[32774]: from='client.24853 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T23:43:35.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:35 smithi076 ceph-mon[32774]: pgmap v337: 129 pgs: 129 active+clean; 620 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 6.0 KiB/s wr, 11 op/s 2024-04-26T23:43:35.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:35 smithi076 ceph-mon[32774]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T23:43:37.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.387+0000 7f80b9400700 1 -- 172.21.15.72:0/477798745 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f80b410b660 msgr2=0x7f80b4111f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:37.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.387+0000 7f80b9400700 1 --2- 172.21.15.72:0/477798745 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f80b410b660 0x7f80b4111f10 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f80ac00b0d0 tx=0x7f80ac00b3e0 comp rx=0 tx=0).stop 2024-04-26T23:43:37.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.387+0000 7f80b9400700 1 -- 172.21.15.72:0/477798745 shutdown_connections 2024-04-26T23:43:37.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.387+0000 7f80b9400700 1 --2- 172.21.15.72:0/477798745 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f80b410b660 0x7f80b4111f10 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:37.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.387+0000 7f80b9400700 1 --2- 172.21.15.72:0/477798745 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f80b4109ac0 0x7f80b4109ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:37.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.387+0000 7f80b9400700 1 --2- 172.21.15.72:0/477798745 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f80b410acc0 0x7f80b410b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:37.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.387+0000 7f80b9400700 1 -- 172.21.15.72:0/477798745 >> 172.21.15.72:0/477798745 conn(0x7f80b4078a90 msgr2=0x7f80b407aed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:37.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.388+0000 7f80b9400700 1 -- 172.21.15.72:0/477798745 shutdown_connections 2024-04-26T23:43:37.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.388+0000 7f80b9400700 1 -- 172.21.15.72:0/477798745 wait complete. 2024-04-26T23:43:37.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.389+0000 7f80b9400700 1 Processor -- start 2024-04-26T23:43:37.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.389+0000 7f80b9400700 1 -- start start 2024-04-26T23:43:37.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.390+0000 7f80b9400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f80b4109ac0 0x7f80b419fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:37.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.390+0000 7f80b9400700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f80b410acc0 0x7f80b41a0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:37.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.390+0000 7f80b9400700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f80b410b660 0x7f80b41a70a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:37.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.390+0000 7f80b9400700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f80b41155b0 con 0x7f80b410b660 2024-04-26T23:43:37.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.390+0000 7f80b9400700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f80b4115430 con 0x7f80b4109ac0 2024-04-26T23:43:37.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.390+0000 7f80b9400700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f80b4115730 con 0x7f80b410acc0 2024-04-26T23:43:37.395 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.390+0000 7f80b3200700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f80b410b660 0x7f80b41a70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:37.395 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.390+0000 7f80b2800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f80b4109ac0 0x7f80b419fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:37.395 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.390+0000 7f80b3200700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f80b410b660 0x7f80b41a70a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:39604/0 (socket says 172.21.15.72:39604) 2024-04-26T23:43:37.395 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.390+0000 7f80b3200700 1 -- 172.21.15.72:0/3945439279 learned_addr learned my addr 172.21.15.72:0/3945439279 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:43:37.395 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.390+0000 7f80a9e00700 1 --2- 172.21.15.72:0/3945439279 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f80b410acc0 0x7f80b41a0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:37.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.391+0000 7f80b3200700 1 -- 172.21.15.72:0/3945439279 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f80b410acc0 msgr2=0x7f80b41a0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:37.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.391+0000 7f80b3200700 1 --2- 172.21.15.72:0/3945439279 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f80b410acc0 0x7f80b41a0020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:37.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.391+0000 7f80b3200700 1 -- 172.21.15.72:0/3945439279 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f80b4109ac0 msgr2=0x7f80b419fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:37.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.391+0000 7f80b3200700 1 --2- 172.21.15.72:0/3945439279 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f80b4109ac0 0x7f80b419fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:37.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.391+0000 7f80b3200700 1 -- 172.21.15.72:0/3945439279 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80b41a7740 con 0x7f80b410b660 2024-04-26T23:43:37.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.391+0000 7f80b2800700 1 --2- 172.21.15.72:0/3945439279 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f80b4109ac0 0x7f80b419fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:43:37.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.391+0000 7f80b3200700 1 --2- 172.21.15.72:0/3945439279 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f80b410b660 0x7f80b41a70a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f80ac004cb0 tx=0x7f80ac004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:37.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.391+0000 7f80abe00700 1 -- 172.21.15.72:0/3945439279 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80ac01f030 con 0x7f80b410b660 2024-04-26T23:43:37.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.391+0000 7f80b9400700 1 -- 172.21.15.72:0/3945439279 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80b41a79d0 con 0x7f80b410b660 2024-04-26T23:43:37.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.391+0000 7f80abe00700 1 -- 172.21.15.72:0/3945439279 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80ac005040 con 0x7f80b410b660 2024-04-26T23:43:37.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.391+0000 7f80abe00700 1 -- 172.21.15.72:0/3945439279 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80ac01b690 con 0x7f80b410b660 2024-04-26T23:43:37.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.391+0000 7f80b9400700 1 -- 172.21.15.72:0/3945439279 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80b41a7ed0 con 0x7f80b410b660 2024-04-26T23:43:37.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.393+0000 7f80b9400700 1 -- 172.21.15.72:0/3945439279 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80b4051960 con 0x7f80b410b660 2024-04-26T23:43:37.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.395+0000 7f80abe00700 1 -- 172.21.15.72:0/3945439279 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f80ac005480 con 0x7f80b410b660 2024-04-26T23:43:37.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.396+0000 7f80abe00700 1 --2- 172.21.15.72:0/3945439279 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f80980a3d80 0x7f80980a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:37.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.396+0000 7f80abe00700 1 -- 172.21.15.72:0/3945439279 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f80ac00f020 con 0x7f80b410b660 2024-04-26T23:43:37.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.396+0000 7f80b2800700 1 --2- 172.21.15.72:0/3945439279 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f80980a3d80 0x7f80980a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:37.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.397+0000 7f80b2800700 1 --2- 172.21.15.72:0/3945439279 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f80980a3d80 0x7f80980a6240 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f80b410ab20 tx=0x7f80a4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:37.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.401+0000 7f80abe00700 1 -- 172.21.15.72:0/3945439279 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f80ac029020 con 0x7f80b410b660 2024-04-26T23:43:37.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.714+0000 7f80b9400700 1 -- 172.21.15.72:0/3945439279 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f80b4064030 con 0x7f80b410b660 2024-04-26T23:43:37.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.718+0000 7f80abe00700 1 -- 172.21.15.72:0/3945439279 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f80ac09df60 con 0x7f80b410b660 2024-04-26T23:43:37.722 INFO:teuthology.orchestra.run.smithi072.stdout:[client.0] 2024-04-26T23:43:37.722 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQApPCxmaMm0KhAAZl4JFDvcuI+3Usj8Jei+8w== 2024-04-26T23:43:37.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.720+0000 7f80b9400700 1 -- 172.21.15.72:0/3945439279 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f80980a3d80 msgr2=0x7f80980a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:37.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.720+0000 7f80b9400700 1 --2- 172.21.15.72:0/3945439279 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f80980a3d80 0x7f80980a6240 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f80b410ab20 tx=0x7f80a4009260 comp rx=0 tx=0).stop 2024-04-26T23:43:37.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.720+0000 7f80b9400700 1 -- 172.21.15.72:0/3945439279 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f80b410b660 msgr2=0x7f80b41a70a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:37.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.720+0000 7f80b9400700 1 --2- 172.21.15.72:0/3945439279 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f80b410b660 0x7f80b41a70a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f80ac004cb0 tx=0x7f80ac004ce0 comp rx=0 tx=0).stop 2024-04-26T23:43:37.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.720+0000 7f80b9400700 1 -- 172.21.15.72:0/3945439279 shutdown_connections 2024-04-26T23:43:37.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.720+0000 7f80b9400700 1 --2- 172.21.15.72:0/3945439279 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f80980a3d80 0x7f80980a6240 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:37.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.721+0000 7f80b9400700 1 --2- 172.21.15.72:0/3945439279 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f80b410b660 0x7f80b41a70a0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:37.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.721+0000 7f80b9400700 1 --2- 172.21.15.72:0/3945439279 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f80b4109ac0 0x7f80b419fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:37.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.721+0000 7f80b9400700 1 --2- 172.21.15.72:0/3945439279 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f80b410acc0 0x7f80b41a0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:37.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.721+0000 7f80b9400700 1 -- 172.21.15.72:0/3945439279 >> 172.21.15.72:0/3945439279 conn(0x7f80b4078a90 msgr2=0x7f80b410df60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:37.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.721+0000 7f80b9400700 1 -- 172.21.15.72:0/3945439279 shutdown_connections 2024-04-26T23:43:37.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:37.721+0000 7f80b9400700 1 -- 172.21.15.72:0/3945439279 wait complete. 2024-04-26T23:43:37.742 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:37 smithi072 ceph-mon[28184]: pgmap v338: 129 pgs: 129 active+clean; 621 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 6.0 KiB/s wr, 12 op/s 2024-04-26T23:43:37.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:37 smithi136 ceph-mon[33050]: pgmap v338: 129 pgs: 129 active+clean; 621 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 6.0 KiB/s wr, 12 op/s 2024-04-26T23:43:37.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:37 smithi076 ceph-mon[32774]: pgmap v338: 129 pgs: 129 active+clean; 621 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 6.0 KiB/s wr, 12 op/s 2024-04-26T23:43:38.523 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-26T23:43:38.523 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-26T23:43:38.523 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-26T23:43:38.653 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-26T23:43:38.653 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-26T23:43:38.654 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-26T23:43:38.771 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:38 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/3945439279' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T23:43:38.771 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:38 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/3945439279' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-26T23:43:38.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:38 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/3945439279' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T23:43:38.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:38 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/3945439279' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-26T23:43:38.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:38 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/3945439279' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T23:43:38.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:38 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/3945439279' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-26T23:43:39.131 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:43:39.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:39 smithi072 ceph-mon[28184]: pgmap v339: 129 pgs: 129 active+clean; 621 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 6.0 KiB/s wr, 12 op/s 2024-04-26T23:43:39.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:39 smithi136 ceph-mon[33050]: pgmap v339: 129 pgs: 129 active+clean; 621 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 6.0 KiB/s wr, 12 op/s 2024-04-26T23:43:39.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:39 smithi076 ceph-mon[32774]: pgmap v339: 129 pgs: 129 active+clean; 621 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 6.0 KiB/s wr, 12 op/s 2024-04-26T23:43:41.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.368+0000 7faf69000700 1 -- 172.21.15.72:0/2662186398 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faf64075650 msgr2=0x7faf64075a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:41.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.368+0000 7faf69000700 1 --2- 172.21.15.72:0/2662186398 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faf64075650 0x7faf64075a30 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7faf54009990 tx=0x7faf54009ca0 comp rx=0 tx=0).stop 2024-04-26T23:43:41.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.368+0000 7faf69000700 1 -- 172.21.15.72:0/2662186398 shutdown_connections 2024-04-26T23:43:41.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.368+0000 7faf69000700 1 --2- 172.21.15.72:0/2662186398 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faf641145f0 0x7faf641169e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:41.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.368+0000 7faf69000700 1 --2- 172.21.15.72:0/2662186398 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faf64075650 0x7faf64075a30 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:41.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.368+0000 7faf69000700 1 --2- 172.21.15.72:0/2662186398 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf64075f70 0x7faf641140b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:41.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.368+0000 7faf69000700 1 -- 172.21.15.72:0/2662186398 >> 172.21.15.72:0/2662186398 conn(0x7faf6407a580 msgr2=0x7faf6407a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:41.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.368+0000 7faf69000700 1 -- 172.21.15.72:0/2662186398 shutdown_connections 2024-04-26T23:43:41.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.368+0000 7faf69000700 1 -- 172.21.15.72:0/2662186398 wait complete. 2024-04-26T23:43:41.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.369+0000 7faf69000700 1 Processor -- start 2024-04-26T23:43:41.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.369+0000 7faf69000700 1 -- start start 2024-04-26T23:43:41.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.370+0000 7faf69000700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf64075650 0x7faf641a01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:41.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.370+0000 7faf69000700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faf64075f70 0x7faf641a0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:41.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.370+0000 7faf69000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faf641145f0 0x7faf641a4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:41.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.370+0000 7faf69000700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7faf64119c40 con 0x7faf64075f70 2024-04-26T23:43:41.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.370+0000 7faf69000700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7faf64119ac0 con 0x7faf641145f0 2024-04-26T23:43:41.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.370+0000 7faf69000700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7faf64119dc0 con 0x7faf64075650 2024-04-26T23:43:41.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.370+0000 7faf61400700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faf64075f70 0x7faf641a0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:41.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.370+0000 7faf61e00700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf64075650 0x7faf641a01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:41.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.370+0000 7faf61400700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faf64075f70 0x7faf641a0710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:39620/0 (socket says 172.21.15.72:39620) 2024-04-26T23:43:41.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.370+0000 7faf61400700 1 -- 172.21.15.72:0/1534994923 learned_addr learned my addr 172.21.15.72:0/1534994923 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:43:41.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.370+0000 7faf62800700 1 --2- 172.21.15.72:0/1534994923 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faf641145f0 0x7faf641a4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:41.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.371+0000 7faf61400700 1 -- 172.21.15.72:0/1534994923 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf64075650 msgr2=0x7faf641a01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:41.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.371+0000 7faf61400700 1 --2- 172.21.15.72:0/1534994923 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf64075650 0x7faf641a01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:41.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.371+0000 7faf61400700 1 -- 172.21.15.72:0/1534994923 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faf641145f0 msgr2=0x7faf641a4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:41.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.371+0000 7faf61400700 1 --2- 172.21.15.72:0/1534994923 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faf641145f0 0x7faf641a4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:41.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.371+0000 7faf61400700 1 -- 172.21.15.72:0/1534994923 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf641a51e0 con 0x7faf64075f70 2024-04-26T23:43:41.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.371+0000 7faf62800700 1 --2- 172.21.15.72:0/1534994923 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faf641145f0 0x7faf641a4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:43:41.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.371+0000 7faf61400700 1 --2- 172.21.15.72:0/1534994923 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faf64075f70 0x7faf641a0710 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7faf5000e9c0 tx=0x7faf5000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:41.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.371+0000 7faf5aa00700 1 -- 172.21.15.72:0/1534994923 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf5000cbd0 con 0x7faf64075f70 2024-04-26T23:43:41.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.371+0000 7faf5aa00700 1 -- 172.21.15.72:0/1534994923 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf5000ee50 con 0x7faf64075f70 2024-04-26T23:43:41.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.371+0000 7faf69000700 1 -- 172.21.15.72:0/1534994923 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf641a54d0 con 0x7faf64075f70 2024-04-26T23:43:41.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.371+0000 7faf5aa00700 1 -- 172.21.15.72:0/1534994923 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf50005550 con 0x7faf64075f70 2024-04-26T23:43:41.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.372+0000 7faf69000700 1 -- 172.21.15.72:0/1534994923 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf641acd50 con 0x7faf64075f70 2024-04-26T23:43:41.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.373+0000 7faf69000700 1 -- 172.21.15.72:0/1534994923 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf6406a110 con 0x7faf64075f70 2024-04-26T23:43:41.379 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.376+0000 7faf5aa00700 1 -- 172.21.15.72:0/1534994923 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7faf50004750 con 0x7faf64075f70 2024-04-26T23:43:41.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.377+0000 7faf5aa00700 1 --2- 172.21.15.72:0/1534994923 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7faf480a3cd0 0x7faf480a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:41.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.377+0000 7faf5aa00700 1 -- 172.21.15.72:0/1534994923 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7faf500d0cf0 con 0x7faf64075f70 2024-04-26T23:43:41.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.377+0000 7faf61e00700 1 --2- 172.21.15.72:0/1534994923 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7faf480a3cd0 0x7faf480a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:41.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.378+0000 7faf61e00700 1 --2- 172.21.15.72:0/1534994923 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7faf480a3cd0 0x7faf480a6190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7faf5400b2e0 tx=0x7faf54005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:41.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.379+0000 7faf5aa00700 1 -- 172.21.15.72:0/1534994923 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7faf50054060 con 0x7faf64075f70 2024-04-26T23:43:41.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.666+0000 7faf69000700 1 -- 172.21.15.72:0/1534994923 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7faf641a1560 con 0x7faf64075f70 2024-04-26T23:43:41.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.674+0000 7faf5aa00700 1 -- 172.21.15.72:0/1534994923 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+264996 (secure 0 0 0) 0x7faf641a1560 con 0x7faf64075f70 2024-04-26T23:43:41.682 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:43:41.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.681+0000 7faf69000700 1 -- 172.21.15.72:0/1534994923 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7faf480a3cd0 msgr2=0x7faf480a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:41.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.681+0000 7faf69000700 1 --2- 172.21.15.72:0/1534994923 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7faf480a3cd0 0x7faf480a6190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7faf5400b2e0 tx=0x7faf54005fd0 comp rx=0 tx=0).stop 2024-04-26T23:43:41.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.681+0000 7faf69000700 1 -- 172.21.15.72:0/1534994923 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faf64075f70 msgr2=0x7faf641a0710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:41.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.681+0000 7faf69000700 1 --2- 172.21.15.72:0/1534994923 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faf64075f70 0x7faf641a0710 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7faf5000e9c0 tx=0x7faf5000ecd0 comp rx=0 tx=0).stop 2024-04-26T23:43:41.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.681+0000 7faf69000700 1 -- 172.21.15.72:0/1534994923 shutdown_connections 2024-04-26T23:43:41.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.681+0000 7faf69000700 1 --2- 172.21.15.72:0/1534994923 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7faf480a3cd0 0x7faf480a6190 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:41.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.681+0000 7faf69000700 1 --2- 172.21.15.72:0/1534994923 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faf641145f0 0x7faf641a4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:41.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.681+0000 7faf69000700 1 --2- 172.21.15.72:0/1534994923 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf64075650 0x7faf641a01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:41.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.681+0000 7faf69000700 1 --2- 172.21.15.72:0/1534994923 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faf64075f70 0x7faf641a0710 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:41.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.681+0000 7faf69000700 1 -- 172.21.15.72:0/1534994923 >> 172.21.15.72:0/1534994923 conn(0x7faf6407a580 msgr2=0x7faf64111480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:41.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.681+0000 7faf69000700 1 -- 172.21.15.72:0/1534994923 shutdown_connections 2024-04-26T23:43:41.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:41.681+0000 7faf69000700 1 -- 172.21.15.72:0/1534994923 wait complete. 2024-04-26T23:43:41.758 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:41 smithi072 ceph-mon[28184]: pgmap v340: 129 pgs: 129 active+clean; 629 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 7.1 KiB/s wr, 15 op/s 2024-04-26T23:43:41.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:41 smithi136 ceph-mon[33050]: pgmap v340: 129 pgs: 129 active+clean; 629 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 7.1 KiB/s wr, 15 op/s 2024-04-26T23:43:41.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:41 smithi076 ceph-mon[32774]: pgmap v340: 129 pgs: 129 active+clean; 629 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 7.1 KiB/s wr, 15 op/s 2024-04-26T23:43:42.556 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":4281177709},{"type":"v1","addr":"172.21.15.72:6801","nonce":4281177709}]},"active_addr":"172.21.15.72:6801/4281177709","active_change":"2024-04-26T23:34:58.266583+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24110,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.72:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.72:0","nonce":967783728}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.72:0","nonce":2012029988}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.72:0","nonce":1431674288}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.72:0","nonce":3756828755}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.72:0","nonce":847912719}]}]} 2024-04-26T23:43:42.561 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-26T23:43:42.561 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-26T23:43:42.561 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T23:43:42.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:42 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/1534994923' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-26T23:43:42.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:42 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/1534994923' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-26T23:43:42.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:42 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/1534994923' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-26T23:43:43.047 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:43:43.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:43 smithi072 ceph-mon[28184]: pgmap v341: 129 pgs: 129 active+clean; 629 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 6.2 KiB/s wr, 12 op/s 2024-04-26T23:43:43.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:43 smithi136 ceph-mon[33050]: pgmap v341: 129 pgs: 129 active+clean; 629 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 6.2 KiB/s wr, 12 op/s 2024-04-26T23:43:43.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:43 smithi076 ceph-mon[32774]: pgmap v341: 129 pgs: 129 active+clean; 629 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 6.2 KiB/s wr, 12 op/s 2024-04-26T23:43:45.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.259+0000 7fa556e00700 1 -- 172.21.15.72:0/714214906 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa550075f70 msgr2=0x7fa55010f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:45.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.259+0000 7fa556e00700 1 --2- 172.21.15.72:0/714214906 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa550075f70 0x7fa55010f8e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa538009990 tx=0x7fa538009ca0 comp rx=0 tx=0).stop 2024-04-26T23:43:45.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.259+0000 7fa556e00700 1 -- 172.21.15.72:0/714214906 shutdown_connections 2024-04-26T23:43:45.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.259+0000 7fa556e00700 1 --2- 172.21.15.72:0/714214906 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa55010fe20 0x7fa550112210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:45.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.259+0000 7fa556e00700 1 --2- 172.21.15.72:0/714214906 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa550075650 0x7fa550075a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:45.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.259+0000 7fa556e00700 1 --2- 172.21.15.72:0/714214906 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa550075f70 0x7fa55010f8e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:45.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.259+0000 7fa556e00700 1 -- 172.21.15.72:0/714214906 >> 172.21.15.72:0/714214906 conn(0x7fa55007a580 msgr2=0x7fa55007a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:45.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.259+0000 7fa556e00700 1 -- 172.21.15.72:0/714214906 shutdown_connections 2024-04-26T23:43:45.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.259+0000 7fa556e00700 1 -- 172.21.15.72:0/714214906 wait complete. 2024-04-26T23:43:45.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.260+0000 7fa556e00700 1 Processor -- start 2024-04-26T23:43:45.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.260+0000 7fa556e00700 1 -- start start 2024-04-26T23:43:45.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.261+0000 7fa556e00700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa550075650 0x7fa5501a1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:45.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.261+0000 7fa556e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa550075f70 0x7fa5501a1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:45.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.261+0000 7fa556e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa55010fe20 0x7fa55019bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:45.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.261+0000 7fa556e00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fa5501157f0 con 0x7fa55010fe20 2024-04-26T23:43:45.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.261+0000 7fa556e00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa550115670 con 0x7fa550075f70 2024-04-26T23:43:45.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.261+0000 7fa556e00700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fa550115970 con 0x7fa550075650 2024-04-26T23:43:45.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.261+0000 7fa54fe00700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa550075650 0x7fa5501a1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:45.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.261+0000 7fa554e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa55010fe20 0x7fa55019bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:45.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.261+0000 7fa54fe00700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa550075650 0x7fa5501a1a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.72:50214/0 (socket says 172.21.15.72:50214) 2024-04-26T23:43:45.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.261+0000 7fa54fe00700 1 -- 172.21.15.72:0/1403197453 learned_addr learned my addr 172.21.15.72:0/1403197453 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:43:45.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.261+0000 7fa547400700 1 --2- 172.21.15.72:0/1403197453 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa550075f70 0x7fa5501a1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:45.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.261+0000 7fa54fe00700 1 -- 172.21.15.72:0/1403197453 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa550075f70 msgr2=0x7fa5501a1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:45.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.261+0000 7fa54fe00700 1 --2- 172.21.15.72:0/1403197453 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa550075f70 0x7fa5501a1f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:45.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.261+0000 7fa54fe00700 1 -- 172.21.15.72:0/1403197453 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa55010fe20 msgr2=0x7fa55019bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:45.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.261+0000 7fa54fe00700 1 --2- 172.21.15.72:0/1403197453 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa55010fe20 0x7fa55019bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:45.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.262+0000 7fa54fe00700 1 -- 172.21.15.72:0/1403197453 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa55019c450 con 0x7fa550075650 2024-04-26T23:43:45.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.262+0000 7fa54fe00700 1 --2- 172.21.15.72:0/1403197453 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa550075650 0x7fa5501a1a50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa54000e9c0 tx=0x7fa54000ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:45.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.262+0000 7fa54d600700 1 -- 172.21.15.72:0/1403197453 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa54000cbd0 con 0x7fa550075650 2024-04-26T23:43:45.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.262+0000 7fa556e00700 1 -- 172.21.15.72:0/1403197453 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa55019c740 con 0x7fa550075650 2024-04-26T23:43:45.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.262+0000 7fa54d600700 1 -- 172.21.15.72:0/1403197453 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa54000ee60 con 0x7fa550075650 2024-04-26T23:43:45.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.262+0000 7fa54d600700 1 -- 172.21.15.72:0/1403197453 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5400055c0 con 0x7fa550075650 2024-04-26T23:43:45.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.262+0000 7fa556e00700 1 -- 172.21.15.72:0/1403197453 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5501a89d0 con 0x7fa550075650 2024-04-26T23:43:45.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.263+0000 7fa556e00700 1 -- 172.21.15.72:0/1403197453 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa550076d40 con 0x7fa550075650 2024-04-26T23:43:45.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.266+0000 7fa54d600700 1 -- 172.21.15.72:0/1403197453 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fa540004750 con 0x7fa550075650 2024-04-26T23:43:45.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.267+0000 7fa54d600700 1 --2- 172.21.15.72:0/1403197453 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fa5340a3d50 0x7fa5340a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:45.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.267+0000 7fa54d600700 1 -- 172.21.15.72:0/1403197453 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fa5400d0c70 con 0x7fa550075650 2024-04-26T23:43:45.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.267+0000 7fa547400700 1 --2- 172.21.15.72:0/1403197453 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fa5340a3d50 0x7fa5340a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:45.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.268+0000 7fa547400700 1 --2- 172.21.15.72:0/1403197453 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fa5340a3d50 0x7fa5340a6210 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa53800b2e0 tx=0x7fa538005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:45.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.270+0000 7fa54d600700 1 -- 172.21.15.72:0/1403197453 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fa540054060 con 0x7fa550075650 2024-04-26T23:43:45.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.506+0000 7fa556e00700 1 -- 172.21.15.72:0/1403197453 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa55019cfe0 con 0x7fa550075650 2024-04-26T23:43:45.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.513+0000 7fa54d600700 1 -- 172.21.15.72:0/1403197453 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24081 (secure 0 0 0) 0x7fa54009d250 con 0x7fa550075650 2024-04-26T23:43:45.517 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:45 smithi072 ceph-mon[28184]: pgmap v342: 129 pgs: 129 active+clean; 629 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 5.3 KiB/s wr, 12 op/s 2024-04-26T23:43:45.518 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:43:45.518 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"fsid":"3c8453ae-0425-11ef-bc93-c7b262605968","created":"2024-04-26T23:33:48.088300+0000","modified":"2024-04-26T23:43:07.990780+0000","last_up_change":"2024-04-26T23:42:04.482891+0000","last_in_change":"2024-04-26T23:41:44.784415+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:37:44.428727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:43:00.431829+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:43:01.872977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:43:05.512050+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e20c9ebf-56f6-47c1-a11b-e1e615af042b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6803","nonce":1639298875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6805","nonce":1639298875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6809","nonce":1639298875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6807","nonce":1639298875}]},"public_addr":"172.21.15.72:6803/1639298875","cluster_addr":"172.21.15.72:6805/1639298875","heartbeat_back_addr":"172.21.15.72:6809/1639298875","heartbeat_front_addr":"172.21.15.72:6807/1639298875","state":["exists","up"]},{"osd":1,"uuid":"e6cac469-780c-4d66-b560-b686072371fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6801","nonce":4185342193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6803","nonce":4185342193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6807","nonce":4185342193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6805","nonce":4185342193}]},"public_addr":"172.21.15.76:6801/4185342193","cluster_addr":"172.21.15.76:6803/4185342193","heartbeat_back_addr":"172.21.15.76:6807/4185342193","heartbeat_front_addr":"172.21.15.76:6805/4185342193","state":["exists","up"]},{"osd":2,"uuid":"838a49fc-18de-446c-8336-67b3e05da231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6801","nonce":3033860799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6803","nonce":3033860799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6807","nonce":3033860799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6805","nonce":3033860799}]},"public_addr":"172.21.15.136:6801/3033860799","cluster_addr":"172.21.15.136:6803/3033860799","heartbeat_back_addr":"172.21.15.136:6807/3033860799","heartbeat_front_addr":"172.21.15.136:6805/3033860799","state":["exists","up"]},{"osd":3,"uuid":"e6da89be-5ba7-48a4-9306-f75498660e0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6811","nonce":2071107815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6813","nonce":2071107815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6817","nonce":2071107815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6815","nonce":2071107815}]},"public_addr":"172.21.15.72:6811/2071107815","cluster_addr":"172.21.15.72:6813/2071107815","heartbeat_back_addr":"172.21.15.72:6817/2071107815","heartbeat_front_addr":"172.21.15.72:6815/2071107815","state":["exists","up"]},{"osd":4,"uuid":"0b17c491-d798-488f-892d-521ef2fbc51e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6809","nonce":147449238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6811","nonce":147449238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6815","nonce":147449238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6813","nonce":147449238}]},"public_addr":"172.21.15.76:6809/147449238","cluster_addr":"172.21.15.76:6811/147449238","heartbeat_back_addr":"172.21.15.76:6815/147449238","heartbeat_front_addr":"172.21.15.76:6813/147449238","state":["exists","up"]},{"osd":5,"uuid":"c7e887b3-ab38-4608-9bfd-1ed49956e8af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6809","nonce":2320149212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6811","nonce":2320149212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6815","nonce":2320149212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6813","nonce":2320149212}]},"public_addr":"172.21.15.136:6809/2320149212","cluster_addr":"172.21.15.136:6811/2320149212","heartbeat_back_addr":"172.21.15.136:6815/2320149212","heartbeat_front_addr":"172.21.15.136:6813/2320149212","state":["exists","up"]},{"osd":6,"uuid":"f0bd604b-0f57-4a38-9295-eb200264f1ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6819","nonce":389499207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6821","nonce":389499207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6825","nonce":389499207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6823","nonce":389499207}]},"public_addr":"172.21.15.72:6819/389499207","cluster_addr":"172.21.15.72:6821/389499207","heartbeat_back_addr":"172.21.15.72:6825/389499207","heartbeat_front_addr":"172.21.15.72:6823/389499207","state":["exists","up"]},{"osd":7,"uuid":"1879d4e4-038e-4cd6-a3ec-3a940bcc3797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6817","nonce":375888575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6819","nonce":375888575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6823","nonce":375888575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6821","nonce":375888575}]},"public_addr":"172.21.15.76:6817/375888575","cluster_addr":"172.21.15.76:6819/375888575","heartbeat_back_addr":"172.21.15.76:6823/375888575","heartbeat_front_addr":"172.21.15.76:6821/375888575","state":["exists","up"]},{"osd":8,"uuid":"6f404062-97e7-4ccb-a582-76dc1ffae548","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6817","nonce":3446774052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6819","nonce":3446774052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6823","nonce":3446774052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6821","nonce":3446774052}]},"public_addr":"172.21.15.136:6817/3446774052","cluster_addr":"172.21.15.136:6819/3446774052","heartbeat_back_addr":"172.21.15.136:6823/3446774052","heartbeat_front_addr":"172.21.15.136:6821/3446774052","state":["exists","up"]},{"osd":9,"uuid":"2989ca68-3164-4667-9bcf-230dcbb7b8f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6827","nonce":3423789152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6829","nonce":3423789152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6832","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6833","nonce":3423789152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6831","nonce":3423789152}]},"public_addr":"172.21.15.72:6827/3423789152","cluster_addr":"172.21.15.72:6829/3423789152","heartbeat_back_addr":"172.21.15.72:6833/3423789152","heartbeat_front_addr":"172.21.15.72:6831/3423789152","state":["exists","up"]},{"osd":10,"uuid":"9ec6d46a-0913-4208-97b5-167188ee8cb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6825","nonce":1120991574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6827","nonce":1120991574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6831","nonce":1120991574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6829","nonce":1120991574}]},"public_addr":"172.21.15.76:6825/1120991574","cluster_addr":"172.21.15.76:6827/1120991574","heartbeat_back_addr":"172.21.15.76:6831/1120991574","heartbeat_front_addr":"172.21.15.76:6829/1120991574","state":["exists","up"]},{"osd":11,"uuid":"bdd0774f-6e58-4918-b4cc-afd4f1118d3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6825","nonce":1135995193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6827","nonce":1135995193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6831","nonce":1135995193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6829","nonce":1135995193}]},"public_addr":"172.21.15.136:6825/1135995193","cluster_addr":"172.21.15.136:6827/1135995193","heartbeat_back_addr":"172.21.15.136:6831/1135995193","heartbeat_front_addr":"172.21.15.136:6829/1135995193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:36:52.128788+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:37:17.523654+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:37:41.634677+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:38:15.060797+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:38:41.234111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:39:06.783560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:39:40.884386+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:40:08.340465+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:40:34.344858+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:41:08.525309+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:41:36.782624+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:42:02.756888+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:0/3625071330":"2024-04-27T23:34:58.266218+0000","172.21.15.72:6801/2903979134":"2024-04-27T23:34:58.266218+0000","172.21.15.72:6800/2903979134":"2024-04-27T23:34:58.266218+0000","172.21.15.72:0/1519142576":"2024-04-27T23:34:58.266218+0000","172.21.15.72:0/3669626460":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/945623297":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/3479150073":"2024-04-27T23:34:24.233319+0000","172.21.15.72:6800/4175896290":"2024-04-27T23:34:24.233319+0000","172.21.15.72:6801/4175896290":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/3783175012":"2024-04-27T23:34:58.266218+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:43:45.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.516+0000 7fa556e00700 1 -- 172.21.15.72:0/1403197453 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fa5340a3d50 msgr2=0x7fa5340a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:45.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.516+0000 7fa556e00700 1 --2- 172.21.15.72:0/1403197453 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fa5340a3d50 0x7fa5340a6210 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa53800b2e0 tx=0x7fa538005960 comp rx=0 tx=0).stop 2024-04-26T23:43:45.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.516+0000 7fa556e00700 1 -- 172.21.15.72:0/1403197453 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa550075650 msgr2=0x7fa5501a1a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:45.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.516+0000 7fa556e00700 1 --2- 172.21.15.72:0/1403197453 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa550075650 0x7fa5501a1a50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa54000e9c0 tx=0x7fa54000ecd0 comp rx=0 tx=0).stop 2024-04-26T23:43:45.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.516+0000 7fa556e00700 1 -- 172.21.15.72:0/1403197453 shutdown_connections 2024-04-26T23:43:45.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.516+0000 7fa556e00700 1 --2- 172.21.15.72:0/1403197453 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fa5340a3d50 0x7fa5340a6210 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:45.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.516+0000 7fa556e00700 1 --2- 172.21.15.72:0/1403197453 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa55010fe20 0x7fa55019bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:45.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.516+0000 7fa556e00700 1 --2- 172.21.15.72:0/1403197453 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa550075650 0x7fa5501a1a50 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:45.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.516+0000 7fa556e00700 1 --2- 172.21.15.72:0/1403197453 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa550075f70 0x7fa5501a1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:45.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.516+0000 7fa556e00700 1 -- 172.21.15.72:0/1403197453 >> 172.21.15.72:0/1403197453 conn(0x7fa55007a580 msgr2=0x7fa5501006d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:45.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.516+0000 7fa556e00700 1 -- 172.21.15.72:0/1403197453 shutdown_connections 2024-04-26T23:43:45.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:45.516+0000 7fa556e00700 1 -- 172.21.15.72:0/1403197453 wait complete. 2024-04-26T23:43:45.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:45 smithi136 ceph-mon[33050]: pgmap v342: 129 pgs: 129 active+clean; 629 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 5.3 KiB/s wr, 12 op/s 2024-04-26T23:43:45.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:45 smithi076 ceph-mon[32774]: pgmap v342: 129 pgs: 129 active+clean; 629 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 5.3 KiB/s wr, 12 op/s 2024-04-26T23:43:46.337 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-26T23:43:46.337 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T23:43:46.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:46 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/1403197453' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:43:46.793 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:43:46.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:46 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/1403197453' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:43:46.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:46 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/1403197453' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:43:47.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:47 smithi072 ceph-mon[28184]: pgmap v343: 129 pgs: 129 active+clean; 629 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-04-26T23:43:47.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:47 smithi136 ceph-mon[33050]: pgmap v343: 129 pgs: 129 active+clean; 629 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-04-26T23:43:47.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:47 smithi076 ceph-mon[32774]: pgmap v343: 129 pgs: 129 active+clean; 629 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-04-26T23:43:49.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.125+0000 7faa48e00700 1 -- 172.21.15.72:0/1993426248 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa44105660 msgr2=0x7faa44105ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:49.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.125+0000 7faa48e00700 1 --2- 172.21.15.72:0/1993426248 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa44105660 0x7faa44105ae0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7faa300098f0 tx=0x7faa30009c00 comp rx=0 tx=0).stop 2024-04-26T23:43:49.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.126+0000 7faa48e00700 1 -- 172.21.15.72:0/1993426248 shutdown_connections 2024-04-26T23:43:49.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.126+0000 7faa48e00700 1 --2- 172.21.15.72:0/1993426248 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faa44106020 0x7faa44112530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:49.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.126+0000 7faa48e00700 1 --2- 172.21.15.72:0/1993426248 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa4410b660 0x7faa4410ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:49.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.126+0000 7faa48e00700 1 --2- 172.21.15.72:0/1993426248 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa44105660 0x7faa44105ae0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:49.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.126+0000 7faa48e00700 1 -- 172.21.15.72:0/1993426248 >> 172.21.15.72:0/1993426248 conn(0x7faa44101150 msgr2=0x7faa44103570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:49.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.126+0000 7faa48e00700 1 -- 172.21.15.72:0/1993426248 shutdown_connections 2024-04-26T23:43:49.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.126+0000 7faa48e00700 1 -- 172.21.15.72:0/1993426248 wait complete. 2024-04-26T23:43:49.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.127+0000 7faa48e00700 1 Processor -- start 2024-04-26T23:43:49.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.127+0000 7faa48e00700 1 -- start start 2024-04-26T23:43:49.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.128+0000 7faa48e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faa44105660 0x7faa4419be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:49.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.128+0000 7faa48e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa44106020 0x7faa4419c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:49.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.128+0000 7faa48e00700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa4410b660 0x7faa441a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:49.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.128+0000 7faa48e00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7faa44115890 con 0x7faa44106020 2024-04-26T23:43:49.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.128+0000 7faa48e00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7faa44115710 con 0x7faa44105660 2024-04-26T23:43:49.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.128+0000 7faa48e00700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7faa44115a10 con 0x7faa4410b660 2024-04-26T23:43:49.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.128+0000 7faa41400700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa44106020 0x7faa4419c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:49.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.128+0000 7faa41e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faa44105660 0x7faa4419be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:49.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.128+0000 7faa41400700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa44106020 0x7faa4419c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47188/0 (socket says 172.21.15.72:47188) 2024-04-26T23:43:49.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.128+0000 7faa41e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faa44105660 0x7faa4419be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:56800/0 (socket says 172.21.15.72:56800) 2024-04-26T23:43:49.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.128+0000 7faa41400700 1 -- 172.21.15.72:0/2416477974 learned_addr learned my addr 172.21.15.72:0/2416477974 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:43:49.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.128+0000 7faa42800700 1 --2- 172.21.15.72:0/2416477974 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa4410b660 0x7faa441a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:49.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.129+0000 7faa41e00700 1 -- 172.21.15.72:0/2416477974 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa4410b660 msgr2=0x7faa441a0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:49.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.129+0000 7faa41e00700 1 --2- 172.21.15.72:0/2416477974 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa4410b660 0x7faa441a0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:49.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.129+0000 7faa41e00700 1 -- 172.21.15.72:0/2416477974 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa44106020 msgr2=0x7faa4419c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:49.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.129+0000 7faa41e00700 1 --2- 172.21.15.72:0/2416477974 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa44106020 0x7faa4419c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:49.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.129+0000 7faa41e00700 1 -- 172.21.15.72:0/2416477974 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa441a0dc0 con 0x7faa44105660 2024-04-26T23:43:49.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.129+0000 7faa41400700 1 --2- 172.21.15.72:0/2416477974 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa44106020 0x7faa4419c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:43:49.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.129+0000 7faa41e00700 1 --2- 172.21.15.72:0/2416477974 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faa44105660 0x7faa4419be00 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7faa3400e9f0 tx=0x7faa3400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:49.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.129+0000 7faa3aa00700 1 -- 172.21.15.72:0/2416477974 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa3400c9c0 con 0x7faa44105660 2024-04-26T23:43:49.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.129+0000 7faa3aa00700 1 -- 172.21.15.72:0/2416477974 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa3400cb60 con 0x7faa44105660 2024-04-26T23:43:49.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.129+0000 7faa3aa00700 1 -- 172.21.15.72:0/2416477974 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa34018680 con 0x7faa44105660 2024-04-26T23:43:49.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.129+0000 7faa48e00700 1 -- 172.21.15.72:0/2416477974 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa441a10b0 con 0x7faa44105660 2024-04-26T23:43:49.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.129+0000 7faa48e00700 1 -- 172.21.15.72:0/2416477974 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa441a8930 con 0x7faa44105660 2024-04-26T23:43:49.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.130+0000 7faa48e00700 1 -- 172.21.15.72:0/2416477974 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa4406a110 con 0x7faa44105660 2024-04-26T23:43:49.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.133+0000 7faa3aa00700 1 -- 172.21.15.72:0/2416477974 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7faa34010430 con 0x7faa44105660 2024-04-26T23:43:49.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.134+0000 7faa3aa00700 1 --2- 172.21.15.72:0/2416477974 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7faa280a3d50 0x7faa280a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:43:49.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.134+0000 7faa3aa00700 1 -- 172.21.15.72:0/2416477974 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7faa340d09d0 con 0x7faa44105660 2024-04-26T23:43:49.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.134+0000 7faa41400700 1 --2- 172.21.15.72:0/2416477974 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7faa280a3d50 0x7faa280a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:43:49.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.135+0000 7faa41400700 1 --2- 172.21.15.72:0/2416477974 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7faa280a3d50 0x7faa280a6210 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7faa441037c0 tx=0x7faa30019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:43:49.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.136+0000 7faa3aa00700 1 -- 172.21.15.72:0/2416477974 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7faa3409cfb0 con 0x7faa44105660 2024-04-26T23:43:49.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.367+0000 7faa48e00700 1 -- 172.21.15.72:0/2416477974 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7faa4402e5b0 con 0x7faa44105660 2024-04-26T23:43:49.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.369+0000 7faa3aa00700 1 -- 172.21.15.72:0/2416477974 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24081 (secure 0 0 0) 0x7faa3409c8b0 con 0x7faa44105660 2024-04-26T23:43:49.374 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:43:49.374 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"fsid":"3c8453ae-0425-11ef-bc93-c7b262605968","created":"2024-04-26T23:33:48.088300+0000","modified":"2024-04-26T23:43:07.990780+0000","last_up_change":"2024-04-26T23:42:04.482891+0000","last_in_change":"2024-04-26T23:41:44.784415+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:37:44.428727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:43:00.431829+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:43:01.872977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:43:05.512050+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e20c9ebf-56f6-47c1-a11b-e1e615af042b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6803","nonce":1639298875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6805","nonce":1639298875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6809","nonce":1639298875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6807","nonce":1639298875}]},"public_addr":"172.21.15.72:6803/1639298875","cluster_addr":"172.21.15.72:6805/1639298875","heartbeat_back_addr":"172.21.15.72:6809/1639298875","heartbeat_front_addr":"172.21.15.72:6807/1639298875","state":["exists","up"]},{"osd":1,"uuid":"e6cac469-780c-4d66-b560-b686072371fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6801","nonce":4185342193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6803","nonce":4185342193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6807","nonce":4185342193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6805","nonce":4185342193}]},"public_addr":"172.21.15.76:6801/4185342193","cluster_addr":"172.21.15.76:6803/4185342193","heartbeat_back_addr":"172.21.15.76:6807/4185342193","heartbeat_front_addr":"172.21.15.76:6805/4185342193","state":["exists","up"]},{"osd":2,"uuid":"838a49fc-18de-446c-8336-67b3e05da231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6801","nonce":3033860799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6803","nonce":3033860799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6807","nonce":3033860799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6805","nonce":3033860799}]},"public_addr":"172.21.15.136:6801/3033860799","cluster_addr":"172.21.15.136:6803/3033860799","heartbeat_back_addr":"172.21.15.136:6807/3033860799","heartbeat_front_addr":"172.21.15.136:6805/3033860799","state":["exists","up"]},{"osd":3,"uuid":"e6da89be-5ba7-48a4-9306-f75498660e0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6811","nonce":2071107815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6813","nonce":2071107815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6817","nonce":2071107815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6815","nonce":2071107815}]},"public_addr":"172.21.15.72:6811/2071107815","cluster_addr":"172.21.15.72:6813/2071107815","heartbeat_back_addr":"172.21.15.72:6817/2071107815","heartbeat_front_addr":"172.21.15.72:6815/2071107815","state":["exists","up"]},{"osd":4,"uuid":"0b17c491-d798-488f-892d-521ef2fbc51e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6809","nonce":147449238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6811","nonce":147449238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6815","nonce":147449238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6813","nonce":147449238}]},"public_addr":"172.21.15.76:6809/147449238","cluster_addr":"172.21.15.76:6811/147449238","heartbeat_back_addr":"172.21.15.76:6815/147449238","heartbeat_front_addr":"172.21.15.76:6813/147449238","state":["exists","up"]},{"osd":5,"uuid":"c7e887b3-ab38-4608-9bfd-1ed49956e8af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6809","nonce":2320149212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6811","nonce":2320149212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6815","nonce":2320149212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6813","nonce":2320149212}]},"public_addr":"172.21.15.136:6809/2320149212","cluster_addr":"172.21.15.136:6811/2320149212","heartbeat_back_addr":"172.21.15.136:6815/2320149212","heartbeat_front_addr":"172.21.15.136:6813/2320149212","state":["exists","up"]},{"osd":6,"uuid":"f0bd604b-0f57-4a38-9295-eb200264f1ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6819","nonce":389499207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6821","nonce":389499207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6825","nonce":389499207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6823","nonce":389499207}]},"public_addr":"172.21.15.72:6819/389499207","cluster_addr":"172.21.15.72:6821/389499207","heartbeat_back_addr":"172.21.15.72:6825/389499207","heartbeat_front_addr":"172.21.15.72:6823/389499207","state":["exists","up"]},{"osd":7,"uuid":"1879d4e4-038e-4cd6-a3ec-3a940bcc3797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6817","nonce":375888575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6819","nonce":375888575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6823","nonce":375888575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6821","nonce":375888575}]},"public_addr":"172.21.15.76:6817/375888575","cluster_addr":"172.21.15.76:6819/375888575","heartbeat_back_addr":"172.21.15.76:6823/375888575","heartbeat_front_addr":"172.21.15.76:6821/375888575","state":["exists","up"]},{"osd":8,"uuid":"6f404062-97e7-4ccb-a582-76dc1ffae548","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6817","nonce":3446774052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6819","nonce":3446774052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6823","nonce":3446774052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6821","nonce":3446774052}]},"public_addr":"172.21.15.136:6817/3446774052","cluster_addr":"172.21.15.136:6819/3446774052","heartbeat_back_addr":"172.21.15.136:6823/3446774052","heartbeat_front_addr":"172.21.15.136:6821/3446774052","state":["exists","up"]},{"osd":9,"uuid":"2989ca68-3164-4667-9bcf-230dcbb7b8f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6827","nonce":3423789152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6829","nonce":3423789152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6832","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6833","nonce":3423789152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6831","nonce":3423789152}]},"public_addr":"172.21.15.72:6827/3423789152","cluster_addr":"172.21.15.72:6829/3423789152","heartbeat_back_addr":"172.21.15.72:6833/3423789152","heartbeat_front_addr":"172.21.15.72:6831/3423789152","state":["exists","up"]},{"osd":10,"uuid":"9ec6d46a-0913-4208-97b5-167188ee8cb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6825","nonce":1120991574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6827","nonce":1120991574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6831","nonce":1120991574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6829","nonce":1120991574}]},"public_addr":"172.21.15.76:6825/1120991574","cluster_addr":"172.21.15.76:6827/1120991574","heartbeat_back_addr":"172.21.15.76:6831/1120991574","heartbeat_front_addr":"172.21.15.76:6829/1120991574","state":["exists","up"]},{"osd":11,"uuid":"bdd0774f-6e58-4918-b4cc-afd4f1118d3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6825","nonce":1135995193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6827","nonce":1135995193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6831","nonce":1135995193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6829","nonce":1135995193}]},"public_addr":"172.21.15.136:6825/1135995193","cluster_addr":"172.21.15.136:6827/1135995193","heartbeat_back_addr":"172.21.15.136:6831/1135995193","heartbeat_front_addr":"172.21.15.136:6829/1135995193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:36:52.128788+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:37:17.523654+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:37:41.634677+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:38:15.060797+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:38:41.234111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:39:06.783560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:39:40.884386+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:40:08.340465+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:40:34.344858+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:41:08.525309+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:41:36.782624+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:42:02.756888+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:0/3625071330":"2024-04-27T23:34:58.266218+0000","172.21.15.72:6801/2903979134":"2024-04-27T23:34:58.266218+0000","172.21.15.72:6800/2903979134":"2024-04-27T23:34:58.266218+0000","172.21.15.72:0/1519142576":"2024-04-27T23:34:58.266218+0000","172.21.15.72:0/3669626460":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/945623297":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/3479150073":"2024-04-27T23:34:24.233319+0000","172.21.15.72:6800/4175896290":"2024-04-27T23:34:24.233319+0000","172.21.15.72:6801/4175896290":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/3783175012":"2024-04-27T23:34:58.266218+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:43:49.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.372+0000 7faa48e00700 1 -- 172.21.15.72:0/2416477974 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7faa280a3d50 msgr2=0x7faa280a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:49.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.372+0000 7faa48e00700 1 --2- 172.21.15.72:0/2416477974 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7faa280a3d50 0x7faa280a6210 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7faa441037c0 tx=0x7faa30019040 comp rx=0 tx=0).stop 2024-04-26T23:43:49.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.372+0000 7faa48e00700 1 -- 172.21.15.72:0/2416477974 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faa44105660 msgr2=0x7faa4419be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:43:49.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.372+0000 7faa48e00700 1 --2- 172.21.15.72:0/2416477974 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faa44105660 0x7faa4419be00 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7faa3400e9f0 tx=0x7faa3400ed00 comp rx=0 tx=0).stop 2024-04-26T23:43:49.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.372+0000 7faa48e00700 1 -- 172.21.15.72:0/2416477974 shutdown_connections 2024-04-26T23:43:49.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.372+0000 7faa48e00700 1 --2- 172.21.15.72:0/2416477974 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7faa280a3d50 0x7faa280a6210 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:49.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.372+0000 7faa48e00700 1 --2- 172.21.15.72:0/2416477974 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa4410b660 0x7faa441a0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:49.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.372+0000 7faa48e00700 1 --2- 172.21.15.72:0/2416477974 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faa44105660 0x7faa4419be00 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:49.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.372+0000 7faa48e00700 1 --2- 172.21.15.72:0/2416477974 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa44106020 0x7faa4419c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:43:49.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.372+0000 7faa48e00700 1 -- 172.21.15.72:0/2416477974 >> 172.21.15.72:0/2416477974 conn(0x7faa44101150 msgr2=0x7faa44101970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:43:49.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.372+0000 7faa48e00700 1 -- 172.21.15.72:0/2416477974 shutdown_connections 2024-04-26T23:43:49.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:49.372+0000 7faa48e00700 1 -- 172.21.15.72:0/2416477974 wait complete. 2024-04-26T23:43:49.780 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:49 smithi072 ceph-mon[28184]: pgmap v344: 129 pgs: 129 active+clean; 629 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-04-26T23:43:49.780 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:49 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/2416477974' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:43:49.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:49 smithi136 ceph-mon[33050]: pgmap v344: 129 pgs: 129 active+clean; 629 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-04-26T23:43:49.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:49 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/2416477974' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:43:49.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:49 smithi076 ceph-mon[32774]: pgmap v344: 129 pgs: 129 active+clean; 629 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-04-26T23:43:49.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:49 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/2416477974' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:43:50.250 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-26T23:43:50.251 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-26T23:43:50.251 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-26T23:43:50.252 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-26T23:43:50.252 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-26T23:43:50.253 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-26T23:43:50.253 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-26T23:43:50.253 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-26T23:43:50.254 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-26T23:43:50.254 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-26T23:43:50.254 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-26T23:43:50.255 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-26T23:43:51.379 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:43:51.384 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:43:51.405 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:43:51.458 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:43:51.721 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:51 smithi072 ceph-mon[28184]: pgmap v345: 129 pgs: 129 active+clean; 633 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-04-26T23:43:51.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:51 smithi136 ceph-mon[33050]: pgmap v345: 129 pgs: 129 active+clean; 633 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-04-26T23:43:51.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:51 smithi076 ceph-mon[32774]: pgmap v345: 129 pgs: 129 active+clean; 633 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-04-26T23:43:52.461 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:43:52.470 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:43:52.707 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:43:53.744 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:43:53.745 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:43:53.746 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:43:53.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:53 smithi072 ceph-mon[28184]: pgmap v346: 129 pgs: 129 active+clean; 633 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-26T23:43:53.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:53 smithi136 ceph-mon[33050]: pgmap v346: 129 pgs: 129 active+clean; 633 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-26T23:43:53.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:53 smithi076 ceph-mon[32774]: pgmap v346: 129 pgs: 129 active+clean; 633 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-26T23:43:55.806 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:55 smithi136 ceph-mon[33050]: pgmap v347: 129 pgs: 129 active+clean; 633 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 341 B/s wr, 7 op/s 2024-04-26T23:43:55.917 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:55 smithi072 ceph-mon[28184]: pgmap v347: 129 pgs: 129 active+clean; 633 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 341 B/s wr, 7 op/s 2024-04-26T23:43:55.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:55 smithi076 ceph-mon[32774]: pgmap v347: 129 pgs: 129 active+clean; 633 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 341 B/s wr, 7 op/s 2024-04-26T23:43:57.626 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:57 smithi072 ceph-mon[28184]: pgmap v348: 129 pgs: 129 active+clean; 633 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-26T23:43:57.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:57 smithi136 ceph-mon[33050]: pgmap v348: 129 pgs: 129 active+clean; 633 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-26T23:43:57.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:57 smithi076 ceph-mon[32774]: pgmap v348: 129 pgs: 129 active+clean; 633 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-26T23:43:58.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:58 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-04-26T23:43:58.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:58 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]: dispatch 2024-04-26T23:43:58.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:58 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]: dispatch 2024-04-26T23:43:58.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:58 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-04-26T23:43:58.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:58 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]: dispatch 2024-04-26T23:43:58.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:58 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 6]}]: dispatch 2024-04-26T23:43:58.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:58 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]: dispatch 2024-04-26T23:43:58.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:58 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.22", "id": [3, 9]}]: dispatch 2024-04-26T23:43:58.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:58 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 6]}]: dispatch 2024-04-26T23:43:58.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:58 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 7]}]: dispatch 2024-04-26T23:43:58.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:58 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-04-26T23:43:58.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:58 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]: dispatch 2024-04-26T23:43:58.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:58 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]: dispatch 2024-04-26T23:43:58.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:58 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-04-26T23:43:58.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:58 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]: dispatch 2024-04-26T23:43:58.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:58 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 6]}]: dispatch 2024-04-26T23:43:58.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:58 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]: dispatch 2024-04-26T23:43:58.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:58 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.22", "id": [3, 9]}]: dispatch 2024-04-26T23:43:58.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:58 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 6]}]: dispatch 2024-04-26T23:43:58.930 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:58 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 7]}]: dispatch 2024-04-26T23:43:59.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:58 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-04-26T23:43:59.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:58 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]: dispatch 2024-04-26T23:43:59.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:58 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]: dispatch 2024-04-26T23:43:59.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:58 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-04-26T23:43:59.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:58 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]: dispatch 2024-04-26T23:43:59.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:58 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 6]}]: dispatch 2024-04-26T23:43:59.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:58 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]: dispatch 2024-04-26T23:43:59.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:58 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.22", "id": [3, 9]}]: dispatch 2024-04-26T23:43:59.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:58 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 6]}]: dispatch 2024-04-26T23:43:59.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:58 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 7]}]: dispatch 2024-04-26T23:43:59.582 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:59 smithi072 ceph-mon[28184]: pgmap v349: 129 pgs: 129 active+clean; 633 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-26T23:43:59.582 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:59 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-04-26T23:43:59.582 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:59 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]': finished 2024-04-26T23:43:59.582 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:59 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]': finished 2024-04-26T23:43:59.582 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:59 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-04-26T23:43:59.582 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:59 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]': finished 2024-04-26T23:43:59.582 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:59 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 6]}]': finished 2024-04-26T23:43:59.582 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:59 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]': finished 2024-04-26T23:43:59.582 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:59 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.22", "id": [3, 9]}]': finished 2024-04-26T23:43:59.582 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:59 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 6]}]': finished 2024-04-26T23:43:59.583 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:59 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 7]}]': finished 2024-04-26T23:43:59.583 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:43:59 smithi072 ceph-mon[28184]: osdmap e90: 12 total, 12 up, 12 in 2024-04-26T23:43:59.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:59 smithi076 ceph-mon[32774]: pgmap v349: 129 pgs: 129 active+clean; 633 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-26T23:43:59.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:59 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-04-26T23:43:59.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:59 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]': finished 2024-04-26T23:43:59.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:59 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]': finished 2024-04-26T23:43:59.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:59 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-04-26T23:43:59.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:59 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]': finished 2024-04-26T23:43:59.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:59 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 6]}]': finished 2024-04-26T23:43:59.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:59 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]': finished 2024-04-26T23:43:59.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:59 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.22", "id": [3, 9]}]': finished 2024-04-26T23:43:59.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:59 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 6]}]': finished 2024-04-26T23:43:59.930 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:59 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 7]}]': finished 2024-04-26T23:43:59.930 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:43:59 smithi076 ceph-mon[32774]: osdmap e90: 12 total, 12 up, 12 in 2024-04-26T23:44:00.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:59 smithi136 ceph-mon[33050]: pgmap v349: 129 pgs: 129 active+clean; 633 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 341 B/s wr, 5 op/s 2024-04-26T23:44:00.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:59 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-04-26T23:44:00.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:59 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]': finished 2024-04-26T23:44:00.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:59 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]': finished 2024-04-26T23:44:00.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:59 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-04-26T23:44:00.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:59 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]': finished 2024-04-26T23:44:00.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:59 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 6]}]': finished 2024-04-26T23:44:00.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:59 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]': finished 2024-04-26T23:44:00.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:59 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.22", "id": [3, 9]}]': finished 2024-04-26T23:44:00.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:59 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 6]}]': finished 2024-04-26T23:44:00.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:59 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 7]}]': finished 2024-04-26T23:44:00.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:43:59 smithi136 ceph-mon[33050]: osdmap e90: 12 total, 12 up, 12 in 2024-04-26T23:44:00.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.478+0000 7f66be800700 1 -- 172.21.15.72:0/1676915152 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f66b810fa50 msgr2=0x7f66b810fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:00.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.478+0000 7f66be800700 1 --2- 172.21.15.72:0/1676915152 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f66b810fa50 0x7f66b810fe30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f66a8009990 tx=0x7f66a8009ca0 comp rx=0 tx=0).stop 2024-04-26T23:44:00.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.478+0000 7f66be800700 1 -- 172.21.15.72:0/1676915152 shutdown_connections 2024-04-26T23:44:00.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.478+0000 7f66be800700 1 --2- 172.21.15.72:0/1676915152 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f66b810a410 0x7f66b8116920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:00.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.478+0000 7f66be800700 1 --2- 172.21.15.72:0/1676915152 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f66b810fa50 0x7f66b810fe30 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:00.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.478+0000 7f66be800700 1 --2- 172.21.15.72:0/1676915152 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66b8109a50 0x7f66b8109ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:00.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.478+0000 7f66be800700 1 -- 172.21.15.72:0/1676915152 >> 172.21.15.72:0/1676915152 conn(0x7f66b8076bf0 msgr2=0x7f66b8108240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:44:00.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.478+0000 7f66be800700 1 -- 172.21.15.72:0/1676915152 shutdown_connections 2024-04-26T23:44:00.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.478+0000 7f66be800700 1 -- 172.21.15.72:0/1676915152 wait complete. 2024-04-26T23:44:00.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.479+0000 7f66be800700 1 Processor -- start 2024-04-26T23:44:00.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.480+0000 7f66be800700 1 -- start start 2024-04-26T23:44:00.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.480+0000 7f66be800700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f66b8109a50 0x7f66b81a00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:00.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.480+0000 7f66be800700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66b810a410 0x7f66b81a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:00.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.480+0000 7f66be800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f66b810fa50 0x7f66b81a49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:00.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.480+0000 7f66be800700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f66b8119ca0 con 0x7f66b810a410 2024-04-26T23:44:00.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.480+0000 7f66be800700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f66b8119b20 con 0x7f66b810fa50 2024-04-26T23:44:00.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.480+0000 7f66be800700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f66b8119e20 con 0x7f66b8109a50 2024-04-26T23:44:00.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.481+0000 7f66b7400700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f66b8109a50 0x7f66b81a00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:00.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.481+0000 7f66b7400700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f66b8109a50 0x7f66b81a00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.72:59458/0 (socket says 172.21.15.72:59458) 2024-04-26T23:44:00.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.481+0000 7f66b7400700 1 -- 172.21.15.72:0/974693765 learned_addr learned my addr 172.21.15.72:0/974693765 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:44:00.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.481+0000 7f66aea00700 1 --2- 172.21.15.72:0/974693765 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66b810a410 0x7f66b81a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:00.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.481+0000 7f66b7e00700 1 --2- 172.21.15.72:0/974693765 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f66b810fa50 0x7f66b81a49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:00.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.481+0000 7f66b7400700 1 -- 172.21.15.72:0/974693765 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f66b810fa50 msgr2=0x7f66b81a49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:00.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.481+0000 7f66b7400700 1 --2- 172.21.15.72:0/974693765 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f66b810fa50 0x7f66b81a49c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:00.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.481+0000 7f66b7400700 1 -- 172.21.15.72:0/974693765 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66b810a410 msgr2=0x7f66b81a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:00.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.481+0000 7f66b7400700 1 --2- 172.21.15.72:0/974693765 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66b810a410 0x7f66b81a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:00.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.481+0000 7f66b7400700 1 -- 172.21.15.72:0/974693765 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66b81a5100 con 0x7f66b8109a50 2024-04-26T23:44:00.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.482+0000 7f66b7400700 1 --2- 172.21.15.72:0/974693765 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f66b8109a50 0x7f66b81a00f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f66a8005910 tx=0x7f66a800b670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:00.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.482+0000 7f66b4c00700 1 -- 172.21.15.72:0/974693765 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a8027070 con 0x7f66b8109a50 2024-04-26T23:44:00.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.482+0000 7f66be800700 1 -- 172.21.15.72:0/974693765 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66b81a5390 con 0x7f66b8109a50 2024-04-26T23:44:00.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.482+0000 7f66b4c00700 1 -- 172.21.15.72:0/974693765 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66a8022470 con 0x7f66b8109a50 2024-04-26T23:44:00.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.482+0000 7f66b4c00700 1 -- 172.21.15.72:0/974693765 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a8017650 con 0x7f66b8109a50 2024-04-26T23:44:00.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.482+0000 7f66be800700 1 -- 172.21.15.72:0/974693765 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66b81acc70 con 0x7f66b8109a50 2024-04-26T23:44:00.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.483+0000 7f66be800700 1 -- 172.21.15.72:0/974693765 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f66b8063bb0 con 0x7f66b8109a50 2024-04-26T23:44:00.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.485+0000 7f66b4c00700 1 -- 172.21.15.72:0/974693765 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f66a802a070 con 0x7f66b8109a50 2024-04-26T23:44:00.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.487+0000 7f66b4c00700 1 --2- 172.21.15.72:0/974693765 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f669c0a3cd0 0x7f669c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:00.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.487+0000 7f66b4c00700 1 -- 172.21.15.72:0/974693765 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f66a80d1ab0 con 0x7f66b8109a50 2024-04-26T23:44:00.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.487+0000 7f66aea00700 1 --2- 172.21.15.72:0/974693765 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f669c0a3cd0 0x7f669c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:00.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.487+0000 7f66b4c00700 1 --2- 172.21.15.72:0/974693765 >> [v2:172.21.15.72:6818/389499207,v1:172.21.15.72:6819/389499207] conn(0x7f669c0aee30 0x7f669c0b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:00.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.487+0000 7f66b4c00700 1 -- 172.21.15.72:0/974693765 --> [v2:172.21.15.72:6818/389499207,v1:172.21.15.72:6819/389499207] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f66a8015ec0 con 0x7f669c0aee30 2024-04-26T23:44:00.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.487+0000 7f66b4c00700 1 -- 172.21.15.72:0/974693765 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f66a80d1e60 con 0x7f66b8109a50 2024-04-26T23:44:00.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.487+0000 7f66b7e00700 1 --2- 172.21.15.72:0/974693765 >> [v2:172.21.15.72:6818/389499207,v1:172.21.15.72:6819/389499207] conn(0x7f669c0aee30 0x7f669c0b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:00.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.488+0000 7f66aea00700 1 --2- 172.21.15.72:0/974693765 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f669c0a3cd0 0x7f669c0a6190 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f66b8078b80 tx=0x7f66a0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:00.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.488+0000 7f66b7e00700 1 --2- 172.21.15.72:0/974693765 >> [v2:172.21.15.72:6818/389499207,v1:172.21.15.72:6819/389499207] conn(0x7f669c0aee30 0x7f669c0b12d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:00.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.489+0000 7f66b4c00700 1 -- 172.21.15.72:0/974693765 <== osd.6 v2:172.21.15.72:6818/389499207 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f66a8015ec0 con 0x7f669c0aee30 2024-04-26T23:44:00.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.507+0000 7f66be800700 1 -- 172.21.15.72:0/974693765 --> [v2:172.21.15.72:6818/389499207,v1:172.21.15.72:6819/389499207] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f66b8063fb0 con 0x7f669c0aee30 2024-04-26T23:44:00.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.507+0000 7f66b4c00700 1 -- 172.21.15.72:0/974693765 <== osd.6 v2:172.21.15.72:6818/389499207 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f66b8063fb0 con 0x7f669c0aee30 2024-04-26T23:44:00.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.508+0000 7f66be800700 1 -- 172.21.15.72:0/974693765 >> [v2:172.21.15.72:6818/389499207,v1:172.21.15.72:6819/389499207] conn(0x7f669c0aee30 msgr2=0x7f669c0b12d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:00.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.508+0000 7f66be800700 1 --2- 172.21.15.72:0/974693765 >> [v2:172.21.15.72:6818/389499207,v1:172.21.15.72:6819/389499207] conn(0x7f669c0aee30 0x7f669c0b12d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:00.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.508+0000 7f66be800700 1 -- 172.21.15.72:0/974693765 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f669c0a3cd0 msgr2=0x7f669c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:00.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.508+0000 7f66be800700 1 --2- 172.21.15.72:0/974693765 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f669c0a3cd0 0x7f669c0a6190 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f66b8078b80 tx=0x7f66a0009260 comp rx=0 tx=0).stop 2024-04-26T23:44:00.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.508+0000 7f66be800700 1 -- 172.21.15.72:0/974693765 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f66b8109a50 msgr2=0x7f66b81a00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:00.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.508+0000 7f66be800700 1 --2- 172.21.15.72:0/974693765 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f66b8109a50 0x7f66b81a00f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f66a8005910 tx=0x7f66a800b670 comp rx=0 tx=0).stop 2024-04-26T23:44:00.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.508+0000 7f66b7e00700 1 -- 172.21.15.72:0/974693765 reap_dead start 2024-04-26T23:44:00.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.508+0000 7f66be800700 1 -- 172.21.15.72:0/974693765 shutdown_connections 2024-04-26T23:44:00.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.508+0000 7f66be800700 1 -- 172.21.15.72:0/974693765 >> 172.21.15.72:0/974693765 conn(0x7f66b8076bf0 msgr2=0x7f66b8077210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:44:00.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.509+0000 7f66be800700 1 -- 172.21.15.72:0/974693765 shutdown_connections 2024-04-26T23:44:00.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:00.509+0000 7f66be800700 1 -- 172.21.15.72:0/974693765 wait complete. 2024-04-26T23:44:00.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:00 smithi072 ceph-mon[28184]: osdmap e91: 12 total, 12 up, 12 in 2024-04-26T23:44:00.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:00 smithi076 ceph-mon[32774]: osdmap e91: 12 total, 12 up, 12 in 2024-04-26T23:44:01.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:00 smithi136 ceph-mon[33050]: osdmap e91: 12 total, 12 up, 12 in 2024-04-26T23:44:01.230 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:44:01.230 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:44:01.758 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:01 smithi072 ceph-mon[28184]: pgmap v352: 129 pgs: 1 activating, 2 peering, 126 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 383 B/s wr, 7 op/s 2024-04-26T23:44:01.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:01 smithi076 ceph-mon[32774]: pgmap v352: 129 pgs: 1 activating, 2 peering, 126 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 383 B/s wr, 7 op/s 2024-04-26T23:44:02.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.761+0000 7f076bc00700 1 -- 172.21.15.72:0/1146096792 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f076410b660 msgr2=0x7f076410ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:02.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.761+0000 7f076bc00700 1 --2- 172.21.15.72:0/1146096792 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f076410b660 0x7f076410ba40 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0758009990 tx=0x7f0758009ca0 comp rx=0 tx=0).stop 2024-04-26T23:44:02.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.761+0000 7f076bc00700 1 -- 172.21.15.72:0/1146096792 shutdown_connections 2024-04-26T23:44:02.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.761+0000 7f076bc00700 1 --2- 172.21.15.72:0/1146096792 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0764106020 0x7f0764112530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:02.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.761+0000 7f076bc00700 1 --2- 172.21.15.72:0/1146096792 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f076410b660 0x7f076410ba40 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:02.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.761+0000 7f076bc00700 1 --2- 172.21.15.72:0/1146096792 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0764105660 0x7f0764105ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:02.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.761+0000 7f076bc00700 1 -- 172.21.15.72:0/1146096792 >> 172.21.15.72:0/1146096792 conn(0x7f0764101150 msgr2=0x7f0764103570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:44:02.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.762+0000 7f076bc00700 1 -- 172.21.15.72:0/1146096792 shutdown_connections 2024-04-26T23:44:02.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.762+0000 7f076bc00700 1 -- 172.21.15.72:0/1146096792 wait complete. 2024-04-26T23:44:02.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.763+0000 7f076bc00700 1 Processor -- start 2024-04-26T23:44:02.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.763+0000 7f076bc00700 1 -- start start 2024-04-26T23:44:02.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.763+0000 7f076bc00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0764105660 0x7f076419bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:02.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.763+0000 7f076bc00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0764106020 0x7f076419c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:02.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.763+0000 7f076bc00700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f076410b660 0x7f07641a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:02.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.763+0000 7f076bc00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f0764115890 con 0x7f0764105660 2024-04-26T23:44:02.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.763+0000 7f076bc00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f0764115710 con 0x7f0764106020 2024-04-26T23:44:02.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.763+0000 7f076bc00700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f0764115a10 con 0x7f076410b660 2024-04-26T23:44:02.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.764+0000 7f0763e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0764106020 0x7f076419c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:02.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.764+0000 7f0769200700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0764105660 0x7f076419bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:02.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.764+0000 7f0769c00700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f076410b660 0x7f07641a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:02.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.764+0000 7f0763e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0764106020 0x7f076419c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:33644/0 (socket says 172.21.15.72:33644) 2024-04-26T23:44:02.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.764+0000 7f0763e00700 1 -- 172.21.15.72:0/177141244 learned_addr learned my addr 172.21.15.72:0/177141244 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:44:02.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.764+0000 7f0763e00700 1 -- 172.21.15.72:0/177141244 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f076410b660 msgr2=0x7f07641a0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:02.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.764+0000 7f0763e00700 1 --2- 172.21.15.72:0/177141244 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f076410b660 0x7f07641a0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:02.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.764+0000 7f0763e00700 1 -- 172.21.15.72:0/177141244 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0764105660 msgr2=0x7f076419bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:02.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.764+0000 7f0763e00700 1 --2- 172.21.15.72:0/177141244 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0764105660 0x7f076419bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:02.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.764+0000 7f0763e00700 1 -- 172.21.15.72:0/177141244 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07641a0d50 con 0x7f0764106020 2024-04-26T23:44:02.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.764+0000 7f0763e00700 1 --2- 172.21.15.72:0/177141244 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0764106020 0x7f076419c280 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f075000d980 tx=0x7f075000dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:02.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.765+0000 7f0761600700 1 -- 172.21.15.72:0/177141244 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0750016070 con 0x7f0764106020 2024-04-26T23:44:02.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.765+0000 7f0761600700 1 -- 172.21.15.72:0/177141244 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f075000fd40 con 0x7f0764106020 2024-04-26T23:44:02.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.765+0000 7f076bc00700 1 -- 172.21.15.72:0/177141244 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07641a1040 con 0x7f0764106020 2024-04-26T23:44:02.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.765+0000 7f0761600700 1 -- 172.21.15.72:0/177141244 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f075000e3b0 con 0x7f0764106020 2024-04-26T23:44:02.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.765+0000 7f076bc00700 1 -- 172.21.15.72:0/177141244 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0764077860 con 0x7f0764106020 2024-04-26T23:44:02.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.766+0000 7f076bc00700 1 -- 172.21.15.72:0/177141244 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0764063bb0 con 0x7f0764106020 2024-04-26T23:44:02.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.768+0000 7f0761600700 1 -- 172.21.15.72:0/177141244 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f0750019070 con 0x7f0764106020 2024-04-26T23:44:02.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.769+0000 7f0761600700 1 --2- 172.21.15.72:0/177141244 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f074c0a3cd0 0x7f074c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:02.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.769+0000 7f0761600700 1 -- 172.21.15.72:0/177141244 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f07500d0760 con 0x7f0764106020 2024-04-26T23:44:02.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.769+0000 7f0769200700 1 --2- 172.21.15.72:0/177141244 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f074c0a3cd0 0x7f074c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:02.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.770+0000 7f0761600700 1 --2- 172.21.15.72:0/177141244 >> [v2:172.21.15.136:6824/1135995193,v1:172.21.15.136:6825/1135995193] conn(0x7f074c0aedb0 0x7f074c0b1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:02.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.770+0000 7f0761600700 1 -- 172.21.15.72:0/177141244 --> [v2:172.21.15.136:6824/1135995193,v1:172.21.15.136:6825/1135995193] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f074c0b1920 con 0x7f074c0aedb0 2024-04-26T23:44:02.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.770+0000 7f0761600700 1 -- 172.21.15.72:0/177141244 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f07500d0b10 con 0x7f0764106020 2024-04-26T23:44:02.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.770+0000 7f0769c00700 1 --2- 172.21.15.72:0/177141244 >> [v2:172.21.15.136:6824/1135995193,v1:172.21.15.136:6825/1135995193] conn(0x7f074c0aedb0 0x7f074c0b1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:02.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.770+0000 7f0769200700 1 --2- 172.21.15.72:0/177141244 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f074c0a3cd0 0x7f074c0a6190 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0758006010 tx=0x7f0758022040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:02.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.771+0000 7f0769c00700 1 --2- 172.21.15.72:0/177141244 >> [v2:172.21.15.136:6824/1135995193,v1:172.21.15.136:6825/1135995193] conn(0x7f074c0aedb0 0x7f074c0b1250 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:02.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.772+0000 7f0761600700 1 -- 172.21.15.72:0/177141244 <== osd.11 v2:172.21.15.136:6824/1135995193 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f074c0b1920 con 0x7f074c0aedb0 2024-04-26T23:44:02.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.790+0000 7f076bc00700 1 -- 172.21.15.72:0/177141244 --> [v2:172.21.15.136:6824/1135995193,v1:172.21.15.136:6825/1135995193] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0764064030 con 0x7f074c0aedb0 2024-04-26T23:44:02.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.791+0000 7f0761600700 1 -- 172.21.15.72:0/177141244 <== osd.11 v2:172.21.15.136:6824/1135995193 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f0764064030 con 0x7f074c0aedb0 2024-04-26T23:44:02.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.792+0000 7f076bc00700 1 -- 172.21.15.72:0/177141244 >> [v2:172.21.15.136:6824/1135995193,v1:172.21.15.136:6825/1135995193] conn(0x7f074c0aedb0 msgr2=0x7f074c0b1250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:02.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.792+0000 7f076bc00700 1 --2- 172.21.15.72:0/177141244 >> [v2:172.21.15.136:6824/1135995193,v1:172.21.15.136:6825/1135995193] conn(0x7f074c0aedb0 0x7f074c0b1250 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:02.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.792+0000 7f076bc00700 1 -- 172.21.15.72:0/177141244 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f074c0a3cd0 msgr2=0x7f074c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:02.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.792+0000 7f076bc00700 1 --2- 172.21.15.72:0/177141244 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f074c0a3cd0 0x7f074c0a6190 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0758006010 tx=0x7f0758022040 comp rx=0 tx=0).stop 2024-04-26T23:44:02.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.792+0000 7f076bc00700 1 -- 172.21.15.72:0/177141244 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0764106020 msgr2=0x7f076419c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:02.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.792+0000 7f076bc00700 1 --2- 172.21.15.72:0/177141244 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0764106020 0x7f076419c280 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f075000d980 tx=0x7f075000dc90 comp rx=0 tx=0).stop 2024-04-26T23:44:02.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.792+0000 7f0769c00700 1 -- 172.21.15.72:0/177141244 reap_dead start 2024-04-26T23:44:02.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.792+0000 7f076bc00700 1 -- 172.21.15.72:0/177141244 shutdown_connections 2024-04-26T23:44:02.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.792+0000 7f076bc00700 1 -- 172.21.15.72:0/177141244 >> 172.21.15.72:0/177141244 conn(0x7f0764101150 msgr2=0x7f07641017f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:44:02.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.792+0000 7f076bc00700 1 -- 172.21.15.72:0/177141244 shutdown_connections 2024-04-26T23:44:02.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:01.792+0000 7f076bc00700 1 -- 172.21.15.72:0/177141244 wait complete. 2024-04-26T23:44:02.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:01 smithi136 ceph-mon[33050]: pgmap v352: 129 pgs: 1 activating, 2 peering, 126 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 383 B/s wr, 7 op/s 2024-04-26T23:44:03.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.578+0000 7f4a52a00700 1 -- 172.21.15.72:0/3465024615 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a4c102c20 msgr2=0x7f4a4c1070f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:03.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.578+0000 7f4a52a00700 1 --2- 172.21.15.72:0/3465024615 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a4c102c20 0x7f4a4c1070f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4a40009990 tx=0x7f4a40009ca0 comp rx=0 tx=0).stop 2024-04-26T23:44:03.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.578+0000 7f4a52a00700 1 -- 172.21.15.72:0/3465024615 shutdown_connections 2024-04-26T23:44:03.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.578+0000 7f4a52a00700 1 --2- 172.21.15.72:0/3465024615 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a4c107710 0x7f4a4c107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:03.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.578+0000 7f4a52a00700 1 --2- 172.21.15.72:0/3465024615 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4a4c102300 0x7f4a4c1026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:03.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.578+0000 7f4a52a00700 1 --2- 172.21.15.72:0/3465024615 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a4c102c20 0x7f4a4c1070f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:03.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.578+0000 7f4a52a00700 1 -- 172.21.15.72:0/3465024615 >> 172.21.15.72:0/3465024615 conn(0x7f4a4c0fdd40 msgr2=0x7f4a4c100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:44:03.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.578+0000 7f4a52a00700 1 -- 172.21.15.72:0/3465024615 shutdown_connections 2024-04-26T23:44:03.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.579+0000 7f4a52a00700 1 -- 172.21.15.72:0/3465024615 wait complete. 2024-04-26T23:44:03.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.580+0000 7f4a52a00700 1 Processor -- start 2024-04-26T23:44:03.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.580+0000 7f4a52a00700 1 -- start start 2024-04-26T23:44:03.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.580+0000 7f4a52a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4a4c102300 0x7f4a4c1a0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:03.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.580+0000 7f4a52a00700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a4c102c20 0x7f4a4c1a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:03.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.580+0000 7f4a52a00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a4c107710 0x7f4a4c1a4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:03.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.580+0000 7f4a52a00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f4a4c119c10 con 0x7f4a4c107710 2024-04-26T23:44:03.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.580+0000 7f4a52a00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f4a4c119a90 con 0x7f4a4c102300 2024-04-26T23:44:03.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.580+0000 7f4a52a00700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f4a4c119d90 con 0x7f4a4c102c20 2024-04-26T23:44:03.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.581+0000 7f4a52000700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a4c107710 0x7f4a4c1a4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:03.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.581+0000 7f4a52000700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a4c107710 0x7f4a4c1a4a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60840/0 (socket says 172.21.15.72:60840) 2024-04-26T23:44:03.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.581+0000 7f4a52000700 1 -- 172.21.15.72:0/2784676686 learned_addr learned my addr 172.21.15.72:0/2784676686 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:44:03.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.581+0000 7f4a51600700 1 --2- 172.21.15.72:0/2784676686 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4a4c102300 0x7f4a4c1a0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:03.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.581+0000 7f4a50c00700 1 --2- 172.21.15.72:0/2784676686 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a4c102c20 0x7f4a4c1a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:03.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.581+0000 7f4a52000700 1 -- 172.21.15.72:0/2784676686 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a4c102c20 msgr2=0x7f4a4c1a06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:03.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.581+0000 7f4a52000700 1 --2- 172.21.15.72:0/2784676686 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a4c102c20 0x7f4a4c1a06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:03.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.581+0000 7f4a52000700 1 -- 172.21.15.72:0/2784676686 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4a4c102300 msgr2=0x7f4a4c1a0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:03.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.581+0000 7f4a52000700 1 --2- 172.21.15.72:0/2784676686 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4a4c102300 0x7f4a4c1a0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:03.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.581+0000 7f4a52000700 1 -- 172.21.15.72:0/2784676686 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a4c1a5170 con 0x7f4a4c107710 2024-04-26T23:44:03.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.581+0000 7f4a51600700 1 --2- 172.21.15.72:0/2784676686 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4a4c102300 0x7f4a4c1a0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:44:03.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.582+0000 7f4a52000700 1 --2- 172.21.15.72:0/2784676686 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a4c107710 0x7f4a4c1a4a30 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4a44007ee0 tx=0x7f4a4400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:03.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.582+0000 7f4a4a000700 1 -- 172.21.15.72:0/2784676686 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a44004bb0 con 0x7f4a4c107710 2024-04-26T23:44:03.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.582+0000 7f4a52a00700 1 -- 172.21.15.72:0/2784676686 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a4c1a5460 con 0x7f4a4c107710 2024-04-26T23:44:03.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.582+0000 7f4a4a000700 1 -- 172.21.15.72:0/2784676686 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a44004d50 con 0x7f4a4c107710 2024-04-26T23:44:03.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.582+0000 7f4a4a000700 1 -- 172.21.15.72:0/2784676686 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a44005920 con 0x7f4a4c107710 2024-04-26T23:44:03.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.582+0000 7f4a52a00700 1 -- 172.21.15.72:0/2784676686 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a4c1acce0 con 0x7f4a4c107710 2024-04-26T23:44:03.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.583+0000 7f4a52a00700 1 -- 172.21.15.72:0/2784676686 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4a4c1118f0 con 0x7f4a4c107710 2024-04-26T23:44:03.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.585+0000 7f4a4a000700 1 -- 172.21.15.72:0/2784676686 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4a4401a020 con 0x7f4a4c107710 2024-04-26T23:44:03.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.586+0000 7f4a4a000700 1 --2- 172.21.15.72:0/2784676686 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f4a340a3cd0 0x7f4a340a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:03.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.587+0000 7f4a4a000700 1 -- 172.21.15.72:0/2784676686 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f4a440d06a0 con 0x7f4a4c107710 2024-04-26T23:44:03.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.587+0000 7f4a51600700 1 --2- 172.21.15.72:0/2784676686 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f4a340a3cd0 0x7f4a340a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:03.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.587+0000 7f4a4a000700 1 --2- 172.21.15.72:0/2784676686 >> [v2:172.21.15.76:6800/4185342193,v1:172.21.15.76:6801/4185342193] conn(0x7f4a340aee30 0x7f4a340b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:03.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.587+0000 7f4a4a000700 1 -- 172.21.15.72:0/2784676686 --> [v2:172.21.15.76:6800/4185342193,v1:172.21.15.76:6801/4185342193] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4a340b19a0 con 0x7f4a340aee30 2024-04-26T23:44:03.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.587+0000 7f4a4a000700 1 -- 172.21.15.72:0/2784676686 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f4a440d0a50 con 0x7f4a4c107710 2024-04-26T23:44:03.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.587+0000 7f4a50c00700 1 --2- 172.21.15.72:0/2784676686 >> [v2:172.21.15.76:6800/4185342193,v1:172.21.15.76:6801/4185342193] conn(0x7f4a340aee30 0x7f4a340b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:03.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.587+0000 7f4a51600700 1 --2- 172.21.15.72:0/2784676686 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f4a340a3cd0 0x7f4a340a6190 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4a38006fd0 tx=0x7f4a38009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:03.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.588+0000 7f4a50c00700 1 --2- 172.21.15.72:0/2784676686 >> [v2:172.21.15.76:6800/4185342193,v1:172.21.15.76:6801/4185342193] conn(0x7f4a340aee30 0x7f4a340b12d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:03.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.589+0000 7f4a4a000700 1 -- 172.21.15.72:0/2784676686 <== osd.1 v2:172.21.15.76:6800/4185342193 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4a340b19a0 con 0x7f4a340aee30 2024-04-26T23:44:03.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.607+0000 7f4a52a00700 1 -- 172.21.15.72:0/2784676686 --> [v2:172.21.15.76:6800/4185342193,v1:172.21.15.76:6801/4185342193] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4a4c06b9e0 con 0x7f4a340aee30 2024-04-26T23:44:03.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.607+0000 7f4a4a000700 1 -- 172.21.15.72:0/2784676686 <== osd.1 v2:172.21.15.76:6800/4185342193 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f4a4c06b9e0 con 0x7f4a340aee30 2024-04-26T23:44:03.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.608+0000 7f4a52a00700 1 -- 172.21.15.72:0/2784676686 >> [v2:172.21.15.76:6800/4185342193,v1:172.21.15.76:6801/4185342193] conn(0x7f4a340aee30 msgr2=0x7f4a340b12d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:03.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.608+0000 7f4a52a00700 1 --2- 172.21.15.72:0/2784676686 >> [v2:172.21.15.76:6800/4185342193,v1:172.21.15.76:6801/4185342193] conn(0x7f4a340aee30 0x7f4a340b12d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:03.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.608+0000 7f4a52a00700 1 -- 172.21.15.72:0/2784676686 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f4a340a3cd0 msgr2=0x7f4a340a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:03.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.608+0000 7f4a52a00700 1 --2- 172.21.15.72:0/2784676686 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f4a340a3cd0 0x7f4a340a6190 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4a38006fd0 tx=0x7f4a38009190 comp rx=0 tx=0).stop 2024-04-26T23:44:03.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.608+0000 7f4a52a00700 1 -- 172.21.15.72:0/2784676686 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a4c107710 msgr2=0x7f4a4c1a4a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:03.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.608+0000 7f4a52a00700 1 --2- 172.21.15.72:0/2784676686 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a4c107710 0x7f4a4c1a4a30 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4a44007ee0 tx=0x7f4a4400d3b0 comp rx=0 tx=0).stop 2024-04-26T23:44:03.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.608+0000 7f4a52000700 1 -- 172.21.15.72:0/2784676686 reap_dead start 2024-04-26T23:44:03.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.608+0000 7f4a52a00700 1 -- 172.21.15.72:0/2784676686 shutdown_connections 2024-04-26T23:44:03.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.608+0000 7f4a52a00700 1 -- 172.21.15.72:0/2784676686 >> 172.21.15.72:0/2784676686 conn(0x7f4a4c0fdd40 msgr2=0x7f4a4c110ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:44:03.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.608+0000 7f4a52a00700 1 -- 172.21.15.72:0/2784676686 shutdown_connections 2024-04-26T23:44:03.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:43:59.608+0000 7f4a52a00700 1 -- 172.21.15.72:0/2784676686 wait complete. 2024-04-26T23:44:03.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:03 smithi072 ceph-mon[28184]: pgmap v353: 129 pgs: 1 activating, 2 peering, 126 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 383 B/s wr, 4 op/s; 13 B/s, 1 objects/s recovering 2024-04-26T23:44:03.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:03 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:44:03.884 INFO:teuthology.orchestra.run.smithi072.stdout:197568495670 2024-04-26T23:44:03.885 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-26T23:44:03.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:03 smithi076 ceph-mon[32774]: pgmap v353: 129 pgs: 1 activating, 2 peering, 126 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 383 B/s wr, 4 op/s; 13 B/s, 1 objects/s recovering 2024-04-26T23:44:03.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:03 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:44:04.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:03 smithi136 ceph-mon[33050]: pgmap v353: 129 pgs: 1 activating, 2 peering, 126 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 383 B/s wr, 4 op/s; 13 B/s, 1 objects/s recovering 2024-04-26T23:44:04.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:03 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:44:05.561 INFO:teuthology.orchestra.run.smithi072.stdout:55834574930 2024-04-26T23:44:05.561 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-26T23:44:05.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:05 smithi072 ceph-mon[28184]: pgmap v354: 129 pgs: 1 activating, 2 peering, 126 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 383 B/s wr, 7 op/s; 13 B/s, 1 objects/s recovering 2024-04-26T23:44:05.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:05 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:44:05.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:05 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:44:05.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:05 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:44:05.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:05 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:44:05.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:05 smithi076 ceph-mon[32774]: pgmap v354: 129 pgs: 1 activating, 2 peering, 126 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 383 B/s wr, 7 op/s; 13 B/s, 1 objects/s recovering 2024-04-26T23:44:05.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:05 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:44:05.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:05 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:44:05.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:05 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:44:05.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:05 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:44:06.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:05 smithi136 ceph-mon[33050]: pgmap v354: 129 pgs: 1 activating, 2 peering, 126 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 383 B/s wr, 7 op/s; 13 B/s, 1 objects/s recovering 2024-04-26T23:44:06.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:05 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:44:06.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:05 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:44:06.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:05 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:44:06.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:05 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:44:07.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:07 smithi072 ceph-mon[28184]: pgmap v355: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 383 B/s wr, 7 op/s; 13 B/s, 1 objects/s recovering 2024-04-26T23:44:07.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:07 smithi076 ceph-mon[32774]: pgmap v355: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 383 B/s wr, 7 op/s; 13 B/s, 1 objects/s recovering 2024-04-26T23:44:08.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:07 smithi136 ceph-mon[33050]: pgmap v355: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 383 B/s wr, 7 op/s; 13 B/s, 1 objects/s recovering 2024-04-26T23:44:08.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.628+0000 7f42cac00700 1 -- 172.21.15.72:0/4028837063 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f42c4103470 msgr2=0x7f42c41038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:08.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.628+0000 7f42cac00700 1 --2- 172.21.15.72:0/4028837063 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f42c4103470 0x7f42c41038f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f42b0009990 tx=0x7f42b0009ca0 comp rx=0 tx=0).stop 2024-04-26T23:44:08.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.628+0000 7f42cac00700 1 -- 172.21.15.72:0/4028837063 shutdown_connections 2024-04-26T23:44:08.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.628+0000 7f42cac00700 1 --2- 172.21.15.72:0/4028837063 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42c4103e30 0x7f42c4110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:08.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.628+0000 7f42cac00700 1 --2- 172.21.15.72:0/4028837063 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42c41094c0 0x7f42c41098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:08.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.628+0000 7f42cac00700 1 --2- 172.21.15.72:0/4028837063 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f42c4103470 0x7f42c41038f0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:08.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.628+0000 7f42cac00700 1 -- 172.21.15.72:0/4028837063 >> 172.21.15.72:0/4028837063 conn(0x7f42c407a580 msgr2=0x7f42c407a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:44:08.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.629+0000 7f42cac00700 1 -- 172.21.15.72:0/4028837063 shutdown_connections 2024-04-26T23:44:08.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.629+0000 7f42cac00700 1 -- 172.21.15.72:0/4028837063 wait complete. 2024-04-26T23:44:08.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.630+0000 7f42cac00700 1 Processor -- start 2024-04-26T23:44:08.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.630+0000 7f42cac00700 1 -- start start 2024-04-26T23:44:08.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.630+0000 7f42cac00700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42c4103470 0x7f42c419bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:08.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.630+0000 7f42cac00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f42c4103e30 0x7f42c419c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:08.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.630+0000 7f42cac00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42c41094c0 0x7f42c41a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:08.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.630+0000 7f42cac00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f42c4113610 con 0x7f42c4103e30 2024-04-26T23:44:08.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.630+0000 7f42cac00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f42c4113490 con 0x7f42c41094c0 2024-04-26T23:44:08.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.630+0000 7f42cac00700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f42c4113790 con 0x7f42c4103470 2024-04-26T23:44:08.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.631+0000 7f42c3e00700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42c4103470 0x7f42c419bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:08.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.631+0000 7f42c3e00700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42c4103470 0x7f42c419bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.72:53878/0 (socket says 172.21.15.72:53878) 2024-04-26T23:44:08.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.631+0000 7f42c3e00700 1 -- 172.21.15.72:0/31275541 learned_addr learned my addr 172.21.15.72:0/31275541 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:44:08.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.631+0000 7f42c3400700 1 --2- 172.21.15.72:0/31275541 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f42c4103e30 0x7f42c419c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:08.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.631+0000 7f42c8c00700 1 --2- 172.21.15.72:0/31275541 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42c41094c0 0x7f42c41a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:08.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.631+0000 7f42c3e00700 1 -- 172.21.15.72:0/31275541 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42c41094c0 msgr2=0x7f42c41a05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:08.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.631+0000 7f42c3e00700 1 --2- 172.21.15.72:0/31275541 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42c41094c0 0x7f42c41a05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:08.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.631+0000 7f42c3e00700 1 -- 172.21.15.72:0/31275541 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f42c4103e30 msgr2=0x7f42c419c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:08.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.631+0000 7f42c3e00700 1 --2- 172.21.15.72:0/31275541 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f42c4103e30 0x7f42c419c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:08.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.631+0000 7f42c3e00700 1 -- 172.21.15.72:0/31275541 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42c41a0d30 con 0x7f42c4103470 2024-04-26T23:44:08.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.631+0000 7f42c8c00700 1 --2- 172.21.15.72:0/31275541 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f42c41094c0 0x7f42c41a05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:44:08.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.631+0000 7f42c3e00700 1 --2- 172.21.15.72:0/31275541 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42c4103470 0x7f42c419bd20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f42b400e9a0 tx=0x7f42b400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:08.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.632+0000 7f42c0c00700 1 -- 172.21.15.72:0/31275541 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b400cbd0 con 0x7f42c4103470 2024-04-26T23:44:08.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.632+0000 7f42c0c00700 1 -- 172.21.15.72:0/31275541 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42b400cd70 con 0x7f42c4103470 2024-04-26T23:44:08.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.632+0000 7f42c0c00700 1 -- 172.21.15.72:0/31275541 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42b4018680 con 0x7f42c4103470 2024-04-26T23:44:08.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.632+0000 7f42cac00700 1 -- 172.21.15.72:0/31275541 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42c41a1020 con 0x7f42c4103470 2024-04-26T23:44:08.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.632+0000 7f42cac00700 1 -- 172.21.15.72:0/31275541 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42c41a88a0 con 0x7f42c4103470 2024-04-26T23:44:08.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.633+0000 7f42cac00700 1 -- 172.21.15.72:0/31275541 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f42c4063bb0 con 0x7f42c4103470 2024-04-26T23:44:08.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.635+0000 7f42c0c00700 1 -- 172.21.15.72:0/31275541 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f42b4010430 con 0x7f42c4103470 2024-04-26T23:44:08.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.636+0000 7f42c0c00700 1 --2- 172.21.15.72:0/31275541 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f42a80a3d50 0x7f42a80a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:08.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.636+0000 7f42c0c00700 1 -- 172.21.15.72:0/31275541 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f42b40d1300 con 0x7f42c4103470 2024-04-26T23:44:08.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.637+0000 7f42c3400700 1 --2- 172.21.15.72:0/31275541 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f42a80a3d50 0x7f42a80a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:08.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.637+0000 7f42c0c00700 1 --2- 172.21.15.72:0/31275541 >> [v2:172.21.15.72:6826/3423789152,v1:172.21.15.72:6827/3423789152] conn(0x7f42a80aeeb0 0x7f42a80b1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:08.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.637+0000 7f42c0c00700 1 -- 172.21.15.72:0/31275541 --> [v2:172.21.15.72:6826/3423789152,v1:172.21.15.72:6827/3423789152] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f42b4027e50 con 0x7f42a80aeeb0 2024-04-26T23:44:08.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.637+0000 7f42c0c00700 1 -- 172.21.15.72:0/31275541 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f42b40d16b0 con 0x7f42c4103470 2024-04-26T23:44:08.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.637+0000 7f42c8c00700 1 --2- 172.21.15.72:0/31275541 >> [v2:172.21.15.72:6826/3423789152,v1:172.21.15.72:6827/3423789152] conn(0x7f42a80aeeb0 0x7f42a80b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:08.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.637+0000 7f42c3400700 1 --2- 172.21.15.72:0/31275541 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f42a80a3d50 0x7f42a80a6210 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f42c407ae20 tx=0x7f42b000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:08.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.638+0000 7f42c8c00700 1 --2- 172.21.15.72:0/31275541 >> [v2:172.21.15.72:6826/3423789152,v1:172.21.15.72:6827/3423789152] conn(0x7f42a80aeeb0 0x7f42a80b1350 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:08.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.639+0000 7f42c0c00700 1 -- 172.21.15.72:0/31275541 <== osd.9 v2:172.21.15.72:6826/3423789152 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f42b4027e50 con 0x7f42a80aeeb0 2024-04-26T23:44:08.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.657+0000 7f42cac00700 1 -- 172.21.15.72:0/31275541 --> [v2:172.21.15.72:6826/3423789152,v1:172.21.15.72:6827/3423789152] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f42c4063fb0 con 0x7f42a80aeeb0 2024-04-26T23:44:08.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.657+0000 7f42c0c00700 1 -- 172.21.15.72:0/31275541 <== osd.9 v2:172.21.15.72:6826/3423789152 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f42c4063fb0 con 0x7f42a80aeeb0 2024-04-26T23:44:08.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.658+0000 7f42cac00700 1 -- 172.21.15.72:0/31275541 >> [v2:172.21.15.72:6826/3423789152,v1:172.21.15.72:6827/3423789152] conn(0x7f42a80aeeb0 msgr2=0x7f42a80b1350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:08.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.658+0000 7f42cac00700 1 --2- 172.21.15.72:0/31275541 >> [v2:172.21.15.72:6826/3423789152,v1:172.21.15.72:6827/3423789152] conn(0x7f42a80aeeb0 0x7f42a80b1350 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:08.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.658+0000 7f42cac00700 1 -- 172.21.15.72:0/31275541 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f42a80a3d50 msgr2=0x7f42a80a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:08.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.658+0000 7f42cac00700 1 --2- 172.21.15.72:0/31275541 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f42a80a3d50 0x7f42a80a6210 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f42c407ae20 tx=0x7f42b000b260 comp rx=0 tx=0).stop 2024-04-26T23:44:08.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.658+0000 7f42cac00700 1 -- 172.21.15.72:0/31275541 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42c4103470 msgr2=0x7f42c419bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:08.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.658+0000 7f42cac00700 1 --2- 172.21.15.72:0/31275541 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42c4103470 0x7f42c419bd20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f42b400e9a0 tx=0x7f42b400ecb0 comp rx=0 tx=0).stop 2024-04-26T23:44:08.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.658+0000 7f42c8c00700 1 -- 172.21.15.72:0/31275541 reap_dead start 2024-04-26T23:44:08.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.658+0000 7f42cac00700 1 -- 172.21.15.72:0/31275541 shutdown_connections 2024-04-26T23:44:08.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.658+0000 7f42cac00700 1 -- 172.21.15.72:0/31275541 >> 172.21.15.72:0/31275541 conn(0x7f42c407a580 msgr2=0x7f42c410e950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:44:08.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.659+0000 7f42cac00700 1 -- 172.21.15.72:0/31275541 shutdown_connections 2024-04-26T23:44:08.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:08.659+0000 7f42cac00700 1 -- 172.21.15.72:0/31275541 wait complete. 2024-04-26T23:44:09.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.255+0000 7f0980000700 1 -- 172.21.15.72:0/1265082411 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0978106000 msgr2=0x7f0978112510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:09.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.255+0000 7f0980000700 1 --2- 172.21.15.72:0/1265082411 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0978106000 0x7f0978112510 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f097000b0a0 tx=0x7f097000b3b0 comp rx=0 tx=0).stop 2024-04-26T23:44:09.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.255+0000 7f0980000700 1 -- 172.21.15.72:0/1265082411 shutdown_connections 2024-04-26T23:44:09.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.255+0000 7f0980000700 1 --2- 172.21.15.72:0/1265082411 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0978106000 0x7f0978112510 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:09.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.255+0000 7f0980000700 1 --2- 172.21.15.72:0/1265082411 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f097810b640 0x7f097810ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:09.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.255+0000 7f0980000700 1 --2- 172.21.15.72:0/1265082411 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0978105640 0x7f0978105ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:09.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.255+0000 7f0980000700 1 -- 172.21.15.72:0/1265082411 >> 172.21.15.72:0/1265082411 conn(0x7f0978101130 msgr2=0x7f0978103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:44:09.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.255+0000 7f0980000700 1 -- 172.21.15.72:0/1265082411 shutdown_connections 2024-04-26T23:44:09.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.255+0000 7f0980000700 1 -- 172.21.15.72:0/1265082411 wait complete. 2024-04-26T23:44:09.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.256+0000 7f0980000700 1 Processor -- start 2024-04-26T23:44:09.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.256+0000 7f0980000700 1 -- start start 2024-04-26T23:44:09.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.257+0000 7f0980000700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0978105640 0x7f09781a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:09.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.257+0000 7f0980000700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0978106000 0x7f09781a1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:09.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.257+0000 7f0980000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f097810b640 0x7f097819baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:09.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.257+0000 7f0980000700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f0978115890 con 0x7f0978105640 2024-04-26T23:44:09.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.257+0000 7f0980000700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f0978115710 con 0x7f097810b640 2024-04-26T23:44:09.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.257+0000 7f0980000700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f0978115a10 con 0x7f0978106000 2024-04-26T23:44:09.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.257+0000 7f097d600700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0978105640 0x7f09781a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:09.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.257+0000 7f097d600700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0978105640 0x7f09781a1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47492/0 (socket says 172.21.15.72:47492) 2024-04-26T23:44:09.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.257+0000 7f097d600700 1 -- 172.21.15.72:0/744654625 learned_addr learned my addr 172.21.15.72:0/744654625 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:44:09.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.257+0000 7f097cc00700 1 --2- 172.21.15.72:0/744654625 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0978106000 0x7f09781a1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:09.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.257+0000 7f097e000700 1 --2- 172.21.15.72:0/744654625 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f097810b640 0x7f097819baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:09.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.258+0000 7f097d600700 1 -- 172.21.15.72:0/744654625 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0978106000 msgr2=0x7f09781a1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:09.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.258+0000 7f097d600700 1 --2- 172.21.15.72:0/744654625 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0978106000 0x7f09781a1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:09.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.258+0000 7f097d600700 1 -- 172.21.15.72:0/744654625 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f097810b640 msgr2=0x7f097819baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:09.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.258+0000 7f097d600700 1 --2- 172.21.15.72:0/744654625 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f097810b640 0x7f097819baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:09.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.258+0000 7f097d600700 1 -- 172.21.15.72:0/744654625 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f097819c250 con 0x7f0978105640 2024-04-26T23:44:09.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.258+0000 7f097d600700 1 --2- 172.21.15.72:0/744654625 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0978105640 0x7f09781a1930 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f096c00eb30 tx=0x7f096c00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:09.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.258+0000 7f0976000700 1 -- 172.21.15.72:0/744654625 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f096c00cc90 con 0x7f0978105640 2024-04-26T23:44:09.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.258+0000 7f0976000700 1 -- 172.21.15.72:0/744654625 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f096c0044d0 con 0x7f0978105640 2024-04-26T23:44:09.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.258+0000 7f0980000700 1 -- 172.21.15.72:0/744654625 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f097819c540 con 0x7f0978105640 2024-04-26T23:44:09.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.258+0000 7f0976000700 1 -- 172.21.15.72:0/744654625 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f096c010430 con 0x7f0978105640 2024-04-26T23:44:09.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.258+0000 7f0980000700 1 -- 172.21.15.72:0/744654625 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09781a8800 con 0x7f0978105640 2024-04-26T23:44:09.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.259+0000 7f0980000700 1 -- 172.21.15.72:0/744654625 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0978063bb0 con 0x7f0978105640 2024-04-26T23:44:09.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.262+0000 7f0976000700 1 -- 172.21.15.72:0/744654625 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f096c003ea0 con 0x7f0978105640 2024-04-26T23:44:09.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.263+0000 7f0976000700 1 --2- 172.21.15.72:0/744654625 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f09600a3d50 0x7f09600a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:09.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.263+0000 7f0976000700 1 -- 172.21.15.72:0/744654625 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f096c0d0d10 con 0x7f0978105640 2024-04-26T23:44:09.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.263+0000 7f097cc00700 1 --2- 172.21.15.72:0/744654625 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f09600a3d50 0x7f09600a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:09.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.264+0000 7f0976000700 1 --2- 172.21.15.72:0/744654625 >> [v2:172.21.15.136:6800/3033860799,v1:172.21.15.136:6801/3033860799] conn(0x7f09600aeeb0 0x7f09600b1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:09.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.264+0000 7f0976000700 1 -- 172.21.15.72:0/744654625 --> [v2:172.21.15.136:6800/3033860799,v1:172.21.15.136:6801/3033860799] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f096c01ee50 con 0x7f09600aeeb0 2024-04-26T23:44:09.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.264+0000 7f0976000700 1 -- 172.21.15.72:0/744654625 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f096c0d10c0 con 0x7f0978105640 2024-04-26T23:44:09.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.264+0000 7f097e000700 1 --2- 172.21.15.72:0/744654625 >> [v2:172.21.15.136:6800/3033860799,v1:172.21.15.136:6801/3033860799] conn(0x7f09600aeeb0 0x7f09600b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:09.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.264+0000 7f097cc00700 1 --2- 172.21.15.72:0/744654625 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f09600a3d50 0x7f09600a6210 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f097810abd0 tx=0x7f0964009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:09.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.265+0000 7f097e000700 1 --2- 172.21.15.72:0/744654625 >> [v2:172.21.15.136:6800/3033860799,v1:172.21.15.136:6801/3033860799] conn(0x7f09600aeeb0 0x7f09600b1350 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:09.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.266+0000 7f0976000700 1 -- 172.21.15.72:0/744654625 <== osd.2 v2:172.21.15.136:6800/3033860799 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f096c01ee50 con 0x7f09600aeeb0 2024-04-26T23:44:09.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.284+0000 7f0980000700 1 -- 172.21.15.72:0/744654625 --> [v2:172.21.15.136:6800/3033860799,v1:172.21.15.136:6801/3033860799] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0978063fb0 con 0x7f09600aeeb0 2024-04-26T23:44:09.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.285+0000 7f0976000700 1 -- 172.21.15.72:0/744654625 <== osd.2 v2:172.21.15.136:6800/3033860799 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f0978063fb0 con 0x7f09600aeeb0 2024-04-26T23:44:09.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.285+0000 7f0980000700 1 -- 172.21.15.72:0/744654625 >> [v2:172.21.15.136:6800/3033860799,v1:172.21.15.136:6801/3033860799] conn(0x7f09600aeeb0 msgr2=0x7f09600b1350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:09.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.285+0000 7f0980000700 1 --2- 172.21.15.72:0/744654625 >> [v2:172.21.15.136:6800/3033860799,v1:172.21.15.136:6801/3033860799] conn(0x7f09600aeeb0 0x7f09600b1350 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:09.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.285+0000 7f0980000700 1 -- 172.21.15.72:0/744654625 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f09600a3d50 msgr2=0x7f09600a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:09.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.285+0000 7f0980000700 1 --2- 172.21.15.72:0/744654625 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f09600a3d50 0x7f09600a6210 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f097810abd0 tx=0x7f0964009260 comp rx=0 tx=0).stop 2024-04-26T23:44:09.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.285+0000 7f0980000700 1 -- 172.21.15.72:0/744654625 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0978105640 msgr2=0x7f09781a1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:09.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.285+0000 7f0980000700 1 --2- 172.21.15.72:0/744654625 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0978105640 0x7f09781a1930 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f096c00eb30 tx=0x7f096c00ee40 comp rx=0 tx=0).stop 2024-04-26T23:44:09.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.285+0000 7f097e000700 1 -- 172.21.15.72:0/744654625 reap_dead start 2024-04-26T23:44:09.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.285+0000 7f0980000700 1 -- 172.21.15.72:0/744654625 shutdown_connections 2024-04-26T23:44:09.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.285+0000 7f0980000700 1 -- 172.21.15.72:0/744654625 >> 172.21.15.72:0/744654625 conn(0x7f0978101130 msgr2=0x7f09781016b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:44:09.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.286+0000 7f0980000700 1 -- 172.21.15.72:0/744654625 shutdown_connections 2024-04-26T23:44:09.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:09.286+0000 7f0980000700 1 -- 172.21.15.72:0/744654625 wait complete. 2024-04-26T23:44:09.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:09 smithi072 ceph-mon[28184]: pgmap v356: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s; 11 B/s, 0 objects/s recovering 2024-04-26T23:44:09.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:09 smithi076 ceph-mon[32774]: pgmap v356: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s; 11 B/s, 0 objects/s recovering 2024-04-26T23:44:10.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:09 smithi136 ceph-mon[33050]: pgmap v356: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 3 op/s; 11 B/s, 0 objects/s recovering 2024-04-26T23:44:11.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:11 smithi072 ceph-mon[28184]: pgmap v357: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s; 10 B/s, 0 objects/s recovering 2024-04-26T23:44:11.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:11 smithi076 ceph-mon[32774]: pgmap v357: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s; 10 B/s, 0 objects/s recovering 2024-04-26T23:44:12.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:11 smithi136 ceph-mon[33050]: pgmap v357: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s; 10 B/s, 0 objects/s recovering 2024-04-26T23:44:12.160 INFO:teuthology.orchestra.run.smithi072.stdout:343597383707 2024-04-26T23:44:12.161 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-26T23:44:13.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.839+0000 7f9594e00700 1 -- 172.21.15.72:0/3487206333 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9590108390 msgr2=0x7f9590108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:13.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.839+0000 7f9594e00700 1 --2- 172.21.15.72:0/3487206333 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9590108390 0x7f9590108770 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9580009990 tx=0x7f9580009ca0 comp rx=0 tx=0).stop 2024-04-26T23:44:13.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.839+0000 7f9594e00700 1 -- 172.21.15.72:0/3487206333 shutdown_connections 2024-04-26T23:44:13.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.839+0000 7f9594e00700 1 --2- 172.21.15.72:0/3487206333 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9590102ce0 0x7f959010f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:13.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.839+0000 7f9594e00700 1 --2- 172.21.15.72:0/3487206333 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9590108390 0x7f9590108770 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:13.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.839+0000 7f9594e00700 1 --2- 172.21.15.72:0/3487206333 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9590102320 0x7f95901027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:13.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.839+0000 7f9594e00700 1 -- 172.21.15.72:0/3487206333 >> 172.21.15.72:0/3487206333 conn(0x7f95900792d0 msgr2=0x7f95900796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:44:13.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.839+0000 7f9594e00700 1 -- 172.21.15.72:0/3487206333 shutdown_connections 2024-04-26T23:44:13.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.839+0000 7f9594e00700 1 -- 172.21.15.72:0/3487206333 wait complete. 2024-04-26T23:44:13.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.841+0000 7f9594e00700 1 Processor -- start 2024-04-26T23:44:13.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.841+0000 7f9594e00700 1 -- start start 2024-04-26T23:44:13.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.841+0000 7f9594e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9590102320 0x7f95901091e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:13.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.841+0000 7f9594e00700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9590102ce0 0x7f9590109720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:13.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.841+0000 7f9594e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9590108390 0x7f9590109c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:13.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.841+0000 7f9594e00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f9590112530 con 0x7f9590102320 2024-04-26T23:44:13.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.841+0000 7f9594e00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f95901123b0 con 0x7f9590108390 2024-04-26T23:44:13.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.841+0000 7f9594e00700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f95901126b0 con 0x7f9590102ce0 2024-04-26T23:44:13.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.841+0000 7f958de00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9590102320 0x7f95901091e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:13.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.841+0000 7f958e800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9590108390 0x7f9590109c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:13.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.841+0000 7f958de00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9590102320 0x7f95901091e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47522/0 (socket says 172.21.15.72:47522) 2024-04-26T23:44:13.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.841+0000 7f958de00700 1 -- 172.21.15.72:0/3259425086 learned_addr learned my addr 172.21.15.72:0/3259425086 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:44:13.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.842+0000 7f958d400700 1 --2- 172.21.15.72:0/3259425086 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9590102ce0 0x7f9590109720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:13.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.842+0000 7f958e800700 1 -- 172.21.15.72:0/3259425086 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9590102ce0 msgr2=0x7f9590109720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:13.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.842+0000 7f958e800700 1 --2- 172.21.15.72:0/3259425086 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9590102ce0 0x7f9590109720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:13.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.842+0000 7f958e800700 1 -- 172.21.15.72:0/3259425086 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9590102320 msgr2=0x7f95901091e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:13.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.842+0000 7f958e800700 1 --2- 172.21.15.72:0/3259425086 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9590102320 0x7f95901091e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:13.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.842+0000 7f958e800700 1 -- 172.21.15.72:0/3259425086 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95901ac440 con 0x7f9590108390 2024-04-26T23:44:13.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.842+0000 7f958de00700 1 --2- 172.21.15.72:0/3259425086 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9590102320 0x7f95901091e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:44:13.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.842+0000 7f958d400700 1 --2- 172.21.15.72:0/3259425086 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9590102ce0 0x7f9590109720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:44:13.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.842+0000 7f958e800700 1 --2- 172.21.15.72:0/3259425086 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9590108390 0x7f9590109c60 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9588009f60 tx=0x7f9588009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:13.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.843+0000 7f9586a00700 1 -- 172.21.15.72:0/3259425086 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95880092e0 con 0x7f9590108390 2024-04-26T23:44:13.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.843+0000 7f9586a00700 1 -- 172.21.15.72:0/3259425086 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9588004bb0 con 0x7f9590108390 2024-04-26T23:44:13.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.843+0000 7f9594e00700 1 -- 172.21.15.72:0/3259425086 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95901ac6d0 con 0x7f9590108390 2024-04-26T23:44:13.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.843+0000 7f9586a00700 1 -- 172.21.15.72:0/3259425086 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9588004020 con 0x7f9590108390 2024-04-26T23:44:13.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.843+0000 7f9594e00700 1 -- 172.21.15.72:0/3259425086 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95901acc10 con 0x7f9590108390 2024-04-26T23:44:13.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.844+0000 7f9594e00700 1 -- 172.21.15.72:0/3259425086 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f959006bb50 con 0x7f9590108390 2024-04-26T23:44:13.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.847+0000 7f9586a00700 1 -- 172.21.15.72:0/3259425086 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f958801b030 con 0x7f9590108390 2024-04-26T23:44:13.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.849+0000 7f9586a00700 1 --2- 172.21.15.72:0/3259425086 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f95740a3cd0 0x7f95740a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:13.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.849+0000 7f9586a00700 1 -- 172.21.15.72:0/3259425086 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f95880d0cb0 con 0x7f9590108390 2024-04-26T23:44:13.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.849+0000 7f958de00700 1 --2- 172.21.15.72:0/3259425086 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f95740a3cd0 0x7f95740a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:13.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.849+0000 7f9586a00700 1 --2- 172.21.15.72:0/3259425086 >> [v2:172.21.15.76:6808/147449238,v1:172.21.15.76:6809/147449238] conn(0x7f95740aee30 0x7f95740b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:13.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.849+0000 7f9586a00700 1 -- 172.21.15.72:0/3259425086 --> [v2:172.21.15.76:6808/147449238,v1:172.21.15.76:6809/147449238] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9588010ec0 con 0x7f95740aee30 2024-04-26T23:44:13.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.849+0000 7f9586a00700 1 -- 172.21.15.72:0/3259425086 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f95880d1060 con 0x7f9590108390 2024-04-26T23:44:13.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.849+0000 7f958d400700 1 --2- 172.21.15.72:0/3259425086 >> [v2:172.21.15.76:6808/147449238,v1:172.21.15.76:6809/147449238] conn(0x7f95740aee30 0x7f95740b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:13.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.850+0000 7f958de00700 1 --2- 172.21.15.72:0/3259425086 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f95740a3cd0 0x7f95740a6190 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9580006010 tx=0x7f958001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:13.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.850+0000 7f958d400700 1 --2- 172.21.15.72:0/3259425086 >> [v2:172.21.15.76:6808/147449238,v1:172.21.15.76:6809/147449238] conn(0x7f95740aee30 0x7f95740b12d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:13.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.851+0000 7f9586a00700 1 -- 172.21.15.72:0/3259425086 <== osd.4 v2:172.21.15.76:6808/147449238 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9588010ec0 con 0x7f95740aee30 2024-04-26T23:44:13.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.869+0000 7f9594e00700 1 -- 172.21.15.72:0/3259425086 --> [v2:172.21.15.76:6808/147449238,v1:172.21.15.76:6809/147449238] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9590103a40 con 0x7f95740aee30 2024-04-26T23:44:13.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.870+0000 7f9586a00700 1 -- 172.21.15.72:0/3259425086 <== osd.4 v2:172.21.15.76:6808/147449238 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f9590103a40 con 0x7f95740aee30 2024-04-26T23:44:13.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.870+0000 7f9594e00700 1 -- 172.21.15.72:0/3259425086 >> [v2:172.21.15.76:6808/147449238,v1:172.21.15.76:6809/147449238] conn(0x7f95740aee30 msgr2=0x7f95740b12d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:13.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.870+0000 7f9594e00700 1 --2- 172.21.15.72:0/3259425086 >> [v2:172.21.15.76:6808/147449238,v1:172.21.15.76:6809/147449238] conn(0x7f95740aee30 0x7f95740b12d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:13.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.871+0000 7f9594e00700 1 -- 172.21.15.72:0/3259425086 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f95740a3cd0 msgr2=0x7f95740a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:13.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.871+0000 7f9594e00700 1 --2- 172.21.15.72:0/3259425086 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f95740a3cd0 0x7f95740a6190 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9580006010 tx=0x7f958001a040 comp rx=0 tx=0).stop 2024-04-26T23:44:13.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.871+0000 7f9594e00700 1 -- 172.21.15.72:0/3259425086 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9590108390 msgr2=0x7f9590109c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:13.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.871+0000 7f9594e00700 1 --2- 172.21.15.72:0/3259425086 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9590108390 0x7f9590109c60 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9588009f60 tx=0x7f9588009f90 comp rx=0 tx=0).stop 2024-04-26T23:44:13.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.871+0000 7f958e800700 1 -- 172.21.15.72:0/3259425086 reap_dead start 2024-04-26T23:44:13.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.871+0000 7f9594e00700 1 -- 172.21.15.72:0/3259425086 shutdown_connections 2024-04-26T23:44:13.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.871+0000 7f9594e00700 1 -- 172.21.15.72:0/3259425086 >> 172.21.15.72:0/3259425086 conn(0x7f95900792d0 msgr2=0x7f959006b130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:44:13.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.871+0000 7f9594e00700 1 -- 172.21.15.72:0/3259425086 shutdown_connections 2024-04-26T23:44:13.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.871+0000 7f9594e00700 1 -- 172.21.15.72:0/3259425086 wait complete. 2024-04-26T23:44:13.724 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:13 smithi072 ceph-mon[28184]: pgmap v358: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 9 B/s, 0 objects/s recovering 2024-04-26T23:44:13.780 INFO:teuthology.orchestra.run.smithi072.stdout:77309411407 2024-04-26T23:44:13.781 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-26T23:44:13.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:13 smithi076 ceph-mon[32774]: pgmap v358: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 9 B/s, 0 objects/s recovering 2024-04-26T23:44:14.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:13 smithi136 ceph-mon[33050]: pgmap v358: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 9 B/s, 0 objects/s recovering 2024-04-26T23:44:14.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.086+0000 7f3634400700 1 -- 172.21.15.72:0/234572176 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f362c103470 msgr2=0x7f362c1038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:14.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.086+0000 7f3634400700 1 --2- 172.21.15.72:0/234572176 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f362c103470 0x7f362c1038f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3618009990 tx=0x7f3618009ca0 comp rx=0 tx=0).stop 2024-04-26T23:44:14.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.086+0000 7f3634400700 1 -- 172.21.15.72:0/234572176 shutdown_connections 2024-04-26T23:44:14.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.086+0000 7f3634400700 1 --2- 172.21.15.72:0/234572176 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f362c103e30 0x7f362c110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:14.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.086+0000 7f3634400700 1 --2- 172.21.15.72:0/234572176 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f362c1094c0 0x7f362c1098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:14.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.086+0000 7f3634400700 1 --2- 172.21.15.72:0/234572176 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f362c103470 0x7f362c1038f0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:14.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.086+0000 7f3634400700 1 -- 172.21.15.72:0/234572176 >> 172.21.15.72:0/234572176 conn(0x7f362c07a580 msgr2=0x7f362c07a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:44:14.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.086+0000 7f3634400700 1 -- 172.21.15.72:0/234572176 shutdown_connections 2024-04-26T23:44:14.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.086+0000 7f3634400700 1 -- 172.21.15.72:0/234572176 wait complete. 2024-04-26T23:44:14.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.088+0000 7f3634400700 1 Processor -- start 2024-04-26T23:44:14.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.088+0000 7f3634400700 1 -- start start 2024-04-26T23:44:14.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.088+0000 7f3634400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f362c103470 0x7f362c19bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:14.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.088+0000 7f3634400700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f362c103e30 0x7f362c19c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:14.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.088+0000 7f3634400700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f362c1094c0 0x7f362c1a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:14.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.088+0000 7f3634400700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f362c113640 con 0x7f362c1094c0 2024-04-26T23:44:14.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.088+0000 7f3634400700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f362c1134c0 con 0x7f362c103470 2024-04-26T23:44:14.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.088+0000 7f3634400700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f362c1137c0 con 0x7f362c103e30 2024-04-26T23:44:14.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.089+0000 7f3631a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f362c103470 0x7f362c19bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:14.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.089+0000 7f3631a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f362c103470 0x7f362c19bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:36234/0 (socket says 172.21.15.72:36234) 2024-04-26T23:44:14.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.089+0000 7f3631a00700 1 -- 172.21.15.72:0/2390706352 learned_addr learned my addr 172.21.15.72:0/2390706352 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:44:14.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.089+0000 7f3632400700 1 --2- 172.21.15.72:0/2390706352 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f362c1094c0 0x7f362c1a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:14.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.089+0000 7f3631a00700 1 -- 172.21.15.72:0/2390706352 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f362c103e30 msgr2=0x7f362c19c290 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T23:44:14.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.089+0000 7f3631a00700 1 --2- 172.21.15.72:0/2390706352 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f362c103e30 0x7f362c19c290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:14.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.089+0000 7f3631a00700 1 -- 172.21.15.72:0/2390706352 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f362c1094c0 msgr2=0x7f362c1a0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:14.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.089+0000 7f3631a00700 1 --2- 172.21.15.72:0/2390706352 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f362c1094c0 0x7f362c1a0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:14.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.089+0000 7f3631a00700 1 -- 172.21.15.72:0/2390706352 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f362c1a0d60 con 0x7f362c103470 2024-04-26T23:44:14.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.089+0000 7f3631a00700 1 --2- 172.21.15.72:0/2390706352 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f362c103470 0x7f362c19bd50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f362000d980 tx=0x7f362000dd40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:14.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.089+0000 7f362a000700 1 -- 172.21.15.72:0/2390706352 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3620016070 con 0x7f362c103470 2024-04-26T23:44:14.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.089+0000 7f362a000700 1 -- 172.21.15.72:0/2390706352 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f36200044d0 con 0x7f362c103470 2024-04-26T23:44:14.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.089+0000 7f362a000700 1 -- 172.21.15.72:0/2390706352 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3620004d10 con 0x7f362c103470 2024-04-26T23:44:14.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.089+0000 7f3634400700 1 -- 172.21.15.72:0/2390706352 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f362c1a1050 con 0x7f362c103470 2024-04-26T23:44:14.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.090+0000 7f3634400700 1 -- 172.21.15.72:0/2390706352 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f362c1a88d0 con 0x7f362c103470 2024-04-26T23:44:14.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.091+0000 7f3634400700 1 -- 172.21.15.72:0/2390706352 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f362c063b70 con 0x7f362c103470 2024-04-26T23:44:14.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.093+0000 7f362a000700 1 -- 172.21.15.72:0/2390706352 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f362000bbf0 con 0x7f362c103470 2024-04-26T23:44:14.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.094+0000 7f362a000700 1 --2- 172.21.15.72:0/2390706352 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f36140a3cd0 0x7f36140a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:14.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.094+0000 7f362a000700 1 -- 172.21.15.72:0/2390706352 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f36200d0e10 con 0x7f362c103470 2024-04-26T23:44:14.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.094+0000 7f3631000700 1 --2- 172.21.15.72:0/2390706352 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f36140a3cd0 0x7f36140a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:14.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.094+0000 7f362a000700 1 --2- 172.21.15.72:0/2390706352 >> [v2:172.21.15.76:6816/375888575,v1:172.21.15.76:6817/375888575] conn(0x7f36140aee30 0x7f36140b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:14.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.094+0000 7f362a000700 1 -- 172.21.15.72:0/2390706352 --> [v2:172.21.15.76:6816/375888575,v1:172.21.15.76:6817/375888575] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3620015ec0 con 0x7f36140aee30 2024-04-26T23:44:14.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.094+0000 7f362a000700 1 -- 172.21.15.72:0/2390706352 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f36200d11c0 con 0x7f362c103470 2024-04-26T23:44:14.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.095+0000 7f3632400700 1 --2- 172.21.15.72:0/2390706352 >> [v2:172.21.15.76:6816/375888575,v1:172.21.15.76:6817/375888575] conn(0x7f36140aee30 0x7f36140b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:14.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.095+0000 7f3631000700 1 --2- 172.21.15.72:0/2390706352 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f36140a3cd0 0x7f36140a6190 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f36180098e0 tx=0x7f361800b610 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:14.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.095+0000 7f3632400700 1 --2- 172.21.15.72:0/2390706352 >> [v2:172.21.15.76:6816/375888575,v1:172.21.15.76:6817/375888575] conn(0x7f36140aee30 0x7f36140b12d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:14.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.096+0000 7f362a000700 1 -- 172.21.15.72:0/2390706352 <== osd.7 v2:172.21.15.76:6816/375888575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3620015ec0 con 0x7f36140aee30 2024-04-26T23:44:14.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.115+0000 7f3634400700 1 -- 172.21.15.72:0/2390706352 --> [v2:172.21.15.76:6816/375888575,v1:172.21.15.76:6817/375888575] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f362c063ff0 con 0x7f36140aee30 2024-04-26T23:44:14.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.116+0000 7f362a000700 1 -- 172.21.15.72:0/2390706352 <== osd.7 v2:172.21.15.76:6816/375888575 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f362c063ff0 con 0x7f36140aee30 2024-04-26T23:44:14.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.116+0000 7f3634400700 1 -- 172.21.15.72:0/2390706352 >> [v2:172.21.15.76:6816/375888575,v1:172.21.15.76:6817/375888575] conn(0x7f36140aee30 msgr2=0x7f36140b12d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:14.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.116+0000 7f3634400700 1 --2- 172.21.15.72:0/2390706352 >> [v2:172.21.15.76:6816/375888575,v1:172.21.15.76:6817/375888575] conn(0x7f36140aee30 0x7f36140b12d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:14.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.116+0000 7f3634400700 1 -- 172.21.15.72:0/2390706352 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f36140a3cd0 msgr2=0x7f36140a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:14.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.116+0000 7f3634400700 1 --2- 172.21.15.72:0/2390706352 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f36140a3cd0 0x7f36140a6190 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f36180098e0 tx=0x7f361800b610 comp rx=0 tx=0).stop 2024-04-26T23:44:14.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.116+0000 7f3634400700 1 -- 172.21.15.72:0/2390706352 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f362c103470 msgr2=0x7f362c19bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:14.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.116+0000 7f3634400700 1 --2- 172.21.15.72:0/2390706352 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f362c103470 0x7f362c19bd50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f362000d980 tx=0x7f362000dd40 comp rx=0 tx=0).stop 2024-04-26T23:44:14.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.116+0000 7f3632400700 1 -- 172.21.15.72:0/2390706352 reap_dead start 2024-04-26T23:44:14.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.117+0000 7f3634400700 1 -- 172.21.15.72:0/2390706352 shutdown_connections 2024-04-26T23:44:14.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.117+0000 7f3634400700 1 -- 172.21.15.72:0/2390706352 >> 172.21.15.72:0/2390706352 conn(0x7f362c07a580 msgr2=0x7f362c10e950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:44:14.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.117+0000 7f3634400700 1 -- 172.21.15.72:0/2390706352 shutdown_connections 2024-04-26T23:44:14.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.117+0000 7f3634400700 1 -- 172.21.15.72:0/2390706352 wait complete. 2024-04-26T23:44:15.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.971+0000 7f5178200700 1 -- 172.21.15.72:0/2656178231 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5170103e30 msgr2=0x7f51701102c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:15.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.971+0000 7f5178200700 1 --2- 172.21.15.72:0/2656178231 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5170103e30 0x7f51701102c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f516800b0a0 tx=0x7f516800b3b0 comp rx=0 tx=0).stop 2024-04-26T23:44:15.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.971+0000 7f5178200700 1 -- 172.21.15.72:0/2656178231 shutdown_connections 2024-04-26T23:44:15.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.971+0000 7f5178200700 1 --2- 172.21.15.72:0/2656178231 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5170103e30 0x7f51701102c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:15.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.971+0000 7f5178200700 1 --2- 172.21.15.72:0/2656178231 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f51701094f0 0x7f51701098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:15.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.971+0000 7f5178200700 1 --2- 172.21.15.72:0/2656178231 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5170103470 0x7f51701038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:15.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.971+0000 7f5178200700 1 -- 172.21.15.72:0/2656178231 >> 172.21.15.72:0/2656178231 conn(0x7f517007a5d0 msgr2=0x7f517007a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:44:15.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.971+0000 7f5178200700 1 -- 172.21.15.72:0/2656178231 shutdown_connections 2024-04-26T23:44:15.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.971+0000 7f5178200700 1 -- 172.21.15.72:0/2656178231 wait complete. 2024-04-26T23:44:15.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.972+0000 7f5178200700 1 Processor -- start 2024-04-26T23:44:15.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.973+0000 7f5178200700 1 -- start start 2024-04-26T23:44:15.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.973+0000 7f5178200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5170103470 0x7f5170075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:15.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.973+0000 7f5178200700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5170103e30 0x7f5170075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:15.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.973+0000 7f5178200700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51701094f0 0x7f51700761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:15.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.973+0000 7f5178200700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f51701135f0 con 0x7f51701094f0 2024-04-26T23:44:15.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.973+0000 7f5178200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f5170113470 con 0x7f5170103470 2024-04-26T23:44:15.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.973+0000 7f5178200700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f5170113770 con 0x7f5170103e30 2024-04-26T23:44:15.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.973+0000 7f5176200700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51701094f0 0x7f51700761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:15.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.973+0000 7f5175800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5170103470 0x7f5170075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:15.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.973+0000 7f5174e00700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5170103e30 0x7f5170075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:15.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.973+0000 7f5176200700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51701094f0 0x7f51700761a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47554/0 (socket says 172.21.15.72:47554) 2024-04-26T23:44:15.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.974+0000 7f5175800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5170103470 0x7f5170075720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:36214/0 (socket says 172.21.15.72:36214) 2024-04-26T23:44:15.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.974+0000 7f5176200700 1 -- 172.21.15.72:0/3716284556 learned_addr learned my addr 172.21.15.72:0/3716284556 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:44:15.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.974+0000 7f5175800700 1 -- 172.21.15.72:0/3716284556 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5170103e30 msgr2=0x7f5170075c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:15.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.974+0000 7f5175800700 1 --2- 172.21.15.72:0/3716284556 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5170103e30 0x7f5170075c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:15.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.974+0000 7f5175800700 1 -- 172.21.15.72:0/3716284556 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51701094f0 msgr2=0x7f51700761a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:15.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.974+0000 7f5175800700 1 --2- 172.21.15.72:0/3716284556 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51701094f0 0x7f51700761a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:15.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.974+0000 7f5175800700 1 -- 172.21.15.72:0/3716284556 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51701a7fc0 con 0x7f5170103470 2024-04-26T23:44:15.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.974+0000 7f5176200700 1 --2- 172.21.15.72:0/3716284556 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51701094f0 0x7f51700761a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:44:15.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.974+0000 7f5175800700 1 --2- 172.21.15.72:0/3716284556 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5170103470 0x7f5170075720 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f516400e9f0 tx=0x7f516400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:15.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.975+0000 7f516e000700 1 -- 172.21.15.72:0/3716284556 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f516400c9c0 con 0x7f5170103470 2024-04-26T23:44:15.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.975+0000 7f516e000700 1 -- 172.21.15.72:0/3716284556 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f516400cb60 con 0x7f5170103470 2024-04-26T23:44:15.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.975+0000 7f516e000700 1 -- 172.21.15.72:0/3716284556 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5164018680 con 0x7f5170103470 2024-04-26T23:44:15.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.975+0000 7f5178200700 1 -- 172.21.15.72:0/3716284556 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51701a82b0 con 0x7f5170103470 2024-04-26T23:44:15.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.975+0000 7f5178200700 1 -- 172.21.15.72:0/3716284556 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51701a87a0 con 0x7f5170103470 2024-04-26T23:44:15.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.976+0000 7f5178200700 1 -- 172.21.15.72:0/3716284556 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5170063b30 con 0x7f5170103470 2024-04-26T23:44:15.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.978+0000 7f516e000700 1 -- 172.21.15.72:0/3716284556 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5164010430 con 0x7f5170103470 2024-04-26T23:44:15.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.979+0000 7f516e000700 1 --2- 172.21.15.72:0/3716284556 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f51580a3d50 0x7f51580a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:15.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.979+0000 7f516e000700 1 -- 172.21.15.72:0/3716284556 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f51640d0a70 con 0x7f5170103470 2024-04-26T23:44:15.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.979+0000 7f5174e00700 1 --2- 172.21.15.72:0/3716284556 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f51580a3d50 0x7f51580a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:15.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.980+0000 7f516e000700 1 --2- 172.21.15.72:0/3716284556 >> [v2:172.21.15.136:6816/3446774052,v1:172.21.15.136:6817/3446774052] conn(0x7f51580aeeb0 0x7f51580b1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:15.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.980+0000 7f516e000700 1 -- 172.21.15.72:0/3716284556 --> [v2:172.21.15.136:6816/3446774052,v1:172.21.15.136:6817/3446774052] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f516400be50 con 0x7f51580aeeb0 2024-04-26T23:44:15.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.980+0000 7f516e000700 1 -- 172.21.15.72:0/3716284556 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f51640d0e20 con 0x7f5170103470 2024-04-26T23:44:15.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.980+0000 7f5176200700 1 --2- 172.21.15.72:0/3716284556 >> [v2:172.21.15.136:6816/3446774052,v1:172.21.15.136:6817/3446774052] conn(0x7f51580aeeb0 0x7f51580b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:15.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.980+0000 7f5174e00700 1 --2- 172.21.15.72:0/3716284556 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f51580a3d50 0x7f51580a6210 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f517007ac40 tx=0x7f515c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:15.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.981+0000 7f5176200700 1 --2- 172.21.15.72:0/3716284556 >> [v2:172.21.15.136:6816/3446774052,v1:172.21.15.136:6817/3446774052] conn(0x7f51580aeeb0 0x7f51580b1350 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:15.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:12.982+0000 7f516e000700 1 -- 172.21.15.72:0/3716284556 <== osd.8 v2:172.21.15.136:6816/3446774052 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f516400be50 con 0x7f51580aeeb0 2024-04-26T23:44:15.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.017+0000 7f5178200700 1 -- 172.21.15.72:0/3716284556 --> [v2:172.21.15.136:6816/3446774052,v1:172.21.15.136:6817/3446774052] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5170064000 con 0x7f51580aeeb0 2024-04-26T23:44:15.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.018+0000 7f516e000700 1 -- 172.21.15.72:0/3716284556 <== osd.8 v2:172.21.15.136:6816/3446774052 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f5170064000 con 0x7f51580aeeb0 2024-04-26T23:44:15.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.019+0000 7f5178200700 1 -- 172.21.15.72:0/3716284556 >> [v2:172.21.15.136:6816/3446774052,v1:172.21.15.136:6817/3446774052] conn(0x7f51580aeeb0 msgr2=0x7f51580b1350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:15.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.019+0000 7f5178200700 1 --2- 172.21.15.72:0/3716284556 >> [v2:172.21.15.136:6816/3446774052,v1:172.21.15.136:6817/3446774052] conn(0x7f51580aeeb0 0x7f51580b1350 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:15.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.019+0000 7f5178200700 1 -- 172.21.15.72:0/3716284556 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f51580a3d50 msgr2=0x7f51580a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:15.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.019+0000 7f5178200700 1 --2- 172.21.15.72:0/3716284556 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f51580a3d50 0x7f51580a6210 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f517007ac40 tx=0x7f515c00b410 comp rx=0 tx=0).stop 2024-04-26T23:44:15.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.019+0000 7f5178200700 1 -- 172.21.15.72:0/3716284556 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5170103470 msgr2=0x7f5170075720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:15.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.019+0000 7f5178200700 1 --2- 172.21.15.72:0/3716284556 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5170103470 0x7f5170075720 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f516400e9f0 tx=0x7f516400ed00 comp rx=0 tx=0).stop 2024-04-26T23:44:15.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.019+0000 7f5176200700 1 -- 172.21.15.72:0/3716284556 reap_dead start 2024-04-26T23:44:15.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.019+0000 7f5178200700 1 -- 172.21.15.72:0/3716284556 shutdown_connections 2024-04-26T23:44:15.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.019+0000 7f5178200700 1 -- 172.21.15.72:0/3716284556 >> 172.21.15.72:0/3716284556 conn(0x7f517007a5d0 msgr2=0x7f51701020d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:44:15.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.020+0000 7f5178200700 1 -- 172.21.15.72:0/3716284556 shutdown_connections 2024-04-26T23:44:15.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:13.020+0000 7f5178200700 1 -- 172.21.15.72:0/3716284556 wait complete. 2024-04-26T23:44:15.863 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:15 smithi072 ceph-mon[28184]: pgmap v359: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-04-26T23:44:15.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:15 smithi076 ceph-mon[32774]: pgmap v359: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-04-26T23:44:16.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:15 smithi136 ceph-mon[33050]: pgmap v359: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-04-26T23:44:16.219 INFO:teuthology.orchestra.run.smithi072.stdout:287762808870 2024-04-26T23:44:16.219 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-26T23:44:17.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.171+0000 7fd4e8600700 1 -- 172.21.15.72:0/212033050 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4e010fe20 msgr2=0x7fd4e0112210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:17.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.171+0000 7fd4e8600700 1 --2- 172.21.15.72:0/212033050 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4e010fe20 0x7fd4e0112210 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd4d8009f90 tx=0x7fd4d800b330 comp rx=0 tx=0).stop 2024-04-26T23:44:17.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.171+0000 7fd4e8600700 1 -- 172.21.15.72:0/212033050 shutdown_connections 2024-04-26T23:44:17.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.171+0000 7fd4e8600700 1 --2- 172.21.15.72:0/212033050 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4e010fe20 0x7fd4e0112210 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:17.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.171+0000 7fd4e8600700 1 --2- 172.21.15.72:0/212033050 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd4e0075650 0x7fd4e0075a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:17.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.171+0000 7fd4e8600700 1 --2- 172.21.15.72:0/212033050 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4e0075f70 0x7fd4e010f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:17.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.171+0000 7fd4e8600700 1 -- 172.21.15.72:0/212033050 >> 172.21.15.72:0/212033050 conn(0x7fd4e007a580 msgr2=0x7fd4e007a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:44:17.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.171+0000 7fd4e8600700 1 -- 172.21.15.72:0/212033050 shutdown_connections 2024-04-26T23:44:17.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.171+0000 7fd4e8600700 1 -- 172.21.15.72:0/212033050 wait complete. 2024-04-26T23:44:17.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.172+0000 7fd4e8600700 1 Processor -- start 2024-04-26T23:44:17.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.173+0000 7fd4e8600700 1 -- start start 2024-04-26T23:44:17.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.173+0000 7fd4e8600700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4e0075650 0x7fd4e01a1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:17.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.173+0000 7fd4e8600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd4e0075f70 0x7fd4e01a1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:17.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.173+0000 7fd4e8600700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4e010fe20 0x7fd4e019ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:17.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.173+0000 7fd4e8600700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fd4e01157c0 con 0x7fd4e0075650 2024-04-26T23:44:17.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.173+0000 7fd4e8600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fd4e0115640 con 0x7fd4e0075f70 2024-04-26T23:44:17.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.173+0000 7fd4e8600700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd4e0115940 con 0x7fd4e010fe20 2024-04-26T23:44:17.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.173+0000 7fd4e5c00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4e0075650 0x7fd4e01a1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:17.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.173+0000 7fd4e5c00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4e0075650 0x7fd4e01a1880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60710/0 (socket says 172.21.15.72:60710) 2024-04-26T23:44:17.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.173+0000 7fd4e5200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd4e0075f70 0x7fd4e01a1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:17.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.173+0000 7fd4e5c00700 1 -- 172.21.15.72:0/1623087705 learned_addr learned my addr 172.21.15.72:0/1623087705 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:44:17.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.174+0000 7fd4e6600700 1 --2- 172.21.15.72:0/1623087705 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4e010fe20 0x7fd4e019ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:17.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.174+0000 7fd4e5200700 1 -- 172.21.15.72:0/1623087705 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4e010fe20 msgr2=0x7fd4e019ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:17.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.174+0000 7fd4e5200700 1 --2- 172.21.15.72:0/1623087705 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4e010fe20 0x7fd4e019ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:17.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.174+0000 7fd4e5200700 1 -- 172.21.15.72:0/1623087705 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4e0075650 msgr2=0x7fd4e01a1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:17.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.174+0000 7fd4e5200700 1 --2- 172.21.15.72:0/1623087705 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4e0075650 0x7fd4e01a1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:17.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.174+0000 7fd4e5200700 1 -- 172.21.15.72:0/1623087705 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4e019c1a0 con 0x7fd4e0075f70 2024-04-26T23:44:17.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.174+0000 7fd4e6600700 1 --2- 172.21.15.72:0/1623087705 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4e010fe20 0x7fd4e019ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:44:17.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.174+0000 7fd4e5c00700 1 --2- 172.21.15.72:0/1623087705 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4e0075650 0x7fd4e01a1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:44:17.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.174+0000 7fd4e5200700 1 --2- 172.21.15.72:0/1623087705 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd4e0075f70 0x7fd4e01a1dc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc005fd0 tx=0x7fd4cc00eca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:17.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.175+0000 7fd4de000700 1 -- 172.21.15.72:0/1623087705 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4cc004220 con 0x7fd4e0075f70 2024-04-26T23:44:17.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.175+0000 7fd4de000700 1 -- 172.21.15.72:0/1623087705 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd4cc0043c0 con 0x7fd4e0075f70 2024-04-26T23:44:17.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.175+0000 7fd4de000700 1 -- 172.21.15.72:0/1623087705 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4cc003c00 con 0x7fd4e0075f70 2024-04-26T23:44:17.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.175+0000 7fd4e8600700 1 -- 172.21.15.72:0/1623087705 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4e019c490 con 0x7fd4e0075f70 2024-04-26T23:44:17.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.175+0000 7fd4e8600700 1 -- 172.21.15.72:0/1623087705 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4e01a8750 con 0x7fd4e0075f70 2024-04-26T23:44:17.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.176+0000 7fd4e8600700 1 -- 172.21.15.72:0/1623087705 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd4e0101050 con 0x7fd4e0075f70 2024-04-26T23:44:17.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.178+0000 7fd4de000700 1 -- 172.21.15.72:0/1623087705 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fd4cc010700 con 0x7fd4e0075f70 2024-04-26T23:44:17.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.179+0000 7fd4de000700 1 --2- 172.21.15.72:0/1623087705 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fd4c80a3cd0 0x7fd4c80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:17.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.179+0000 7fd4de000700 1 -- 172.21.15.72:0/1623087705 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd4cc0d0e80 con 0x7fd4e0075f70 2024-04-26T23:44:17.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.179+0000 7fd4e5c00700 1 --2- 172.21.15.72:0/1623087705 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fd4c80a3cd0 0x7fd4c80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:17.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.180+0000 7fd4de000700 1 --2- 172.21.15.72:0/1623087705 >> [v2:172.21.15.136:6808/2320149212,v1:172.21.15.136:6809/2320149212] conn(0x7fd4c80aee30 0x7fd4c80b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:17.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.180+0000 7fd4de000700 1 -- 172.21.15.72:0/1623087705 --> [v2:172.21.15.136:6808/2320149212,v1:172.21.15.136:6809/2320149212] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd4cc01aec0 con 0x7fd4c80aee30 2024-04-26T23:44:17.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.180+0000 7fd4de000700 1 -- 172.21.15.72:0/1623087705 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fd4cc0d1230 con 0x7fd4e0075f70 2024-04-26T23:44:17.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.180+0000 7fd4e5c00700 1 --2- 172.21.15.72:0/1623087705 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fd4c80a3cd0 0x7fd4c80a6190 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd4e019cb80 tx=0x7fd4d4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:17.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.180+0000 7fd4e6600700 1 --2- 172.21.15.72:0/1623087705 >> [v2:172.21.15.136:6808/2320149212,v1:172.21.15.136:6809/2320149212] conn(0x7fd4c80aee30 0x7fd4c80b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:17.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.181+0000 7fd4e6600700 1 --2- 172.21.15.72:0/1623087705 >> [v2:172.21.15.136:6808/2320149212,v1:172.21.15.136:6809/2320149212] conn(0x7fd4c80aee30 0x7fd4c80b12d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:17.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.182+0000 7fd4de000700 1 -- 172.21.15.72:0/1623087705 <== osd.5 v2:172.21.15.136:6808/2320149212 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd4cc01aec0 con 0x7fd4c80aee30 2024-04-26T23:44:17.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.200+0000 7fd4e8600700 1 -- 172.21.15.72:0/1623087705 --> [v2:172.21.15.136:6808/2320149212,v1:172.21.15.136:6809/2320149212] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd4e0076d40 con 0x7fd4c80aee30 2024-04-26T23:44:17.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.200+0000 7fd4de000700 1 -- 172.21.15.72:0/1623087705 <== osd.5 v2:172.21.15.136:6808/2320149212 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fd4e0076d40 con 0x7fd4c80aee30 2024-04-26T23:44:17.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.201+0000 7fd4e8600700 1 -- 172.21.15.72:0/1623087705 >> [v2:172.21.15.136:6808/2320149212,v1:172.21.15.136:6809/2320149212] conn(0x7fd4c80aee30 msgr2=0x7fd4c80b12d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:17.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.201+0000 7fd4e8600700 1 --2- 172.21.15.72:0/1623087705 >> [v2:172.21.15.136:6808/2320149212,v1:172.21.15.136:6809/2320149212] conn(0x7fd4c80aee30 0x7fd4c80b12d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:17.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.201+0000 7fd4e8600700 1 -- 172.21.15.72:0/1623087705 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fd4c80a3cd0 msgr2=0x7fd4c80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:17.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.201+0000 7fd4e8600700 1 --2- 172.21.15.72:0/1623087705 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fd4c80a3cd0 0x7fd4c80a6190 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd4e019cb80 tx=0x7fd4d4009260 comp rx=0 tx=0).stop 2024-04-26T23:44:17.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.201+0000 7fd4e8600700 1 -- 172.21.15.72:0/1623087705 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd4e0075f70 msgr2=0x7fd4e01a1dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:17.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.201+0000 7fd4e8600700 1 --2- 172.21.15.72:0/1623087705 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd4e0075f70 0x7fd4e01a1dc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc005fd0 tx=0x7fd4cc00eca0 comp rx=0 tx=0).stop 2024-04-26T23:44:17.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.201+0000 7fd4e6600700 1 -- 172.21.15.72:0/1623087705 reap_dead start 2024-04-26T23:44:17.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.201+0000 7fd4e8600700 1 -- 172.21.15.72:0/1623087705 shutdown_connections 2024-04-26T23:44:17.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.201+0000 7fd4e8600700 1 -- 172.21.15.72:0/1623087705 >> 172.21.15.72:0/1623087705 conn(0x7fd4e007a580 msgr2=0x7fd4e01005f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:44:17.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.201+0000 7fd4e8600700 1 -- 172.21.15.72:0/1623087705 shutdown_connections 2024-04-26T23:44:17.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:17.201+0000 7fd4e8600700 1 -- 172.21.15.72:0/1623087705 wait complete. 2024-04-26T23:44:17.790 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:17 smithi072 ceph-mon[28184]: pgmap v360: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:44:17.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:17 smithi076 ceph-mon[32774]: pgmap v360: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:44:17.939 INFO:teuthology.orchestra.run.smithi072.stdout:219043332147 2024-04-26T23:44:17.939 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-26T23:44:18.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:17 smithi136 ceph-mon[33050]: pgmap v360: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:44:18.307 INFO:teuthology.orchestra.run.smithi072.stdout:141733920836 2024-04-26T23:44:18.307 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-26T23:44:18.711 INFO:teuthology.orchestra.run.smithi072.stdout:253403070509 2024-04-26T23:44:18.711 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-26T23:44:18.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:18 smithi076 ceph-mon[32774]: pgmap v361: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-26T23:44:18.990 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:18 smithi072 ceph-mon[28184]: pgmap v361: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-26T23:44:19.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:18 smithi136 ceph-mon[33050]: pgmap v361: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-26T23:44:19.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.822+0000 7fd057200700 1 -- 172.21.15.72:0/4163953010 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0581020c0 msgr2=0x7fd0581024a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:19.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.822+0000 7fd057200700 1 --2- 172.21.15.72:0/4163953010 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0581020c0 0x7fd0581024a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd048009990 tx=0x7fd048009ca0 comp rx=0 tx=0).stop 2024-04-26T23:44:19.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.822+0000 7fd057200700 1 -- 172.21.15.72:0/4163953010 shutdown_connections 2024-04-26T23:44:19.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.822+0000 7fd057200700 1 --2- 172.21.15.72:0/4163953010 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd05810fc70 0x7fd058112060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:19.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.822+0000 7fd057200700 1 --2- 172.21.15.72:0/4163953010 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0581020c0 0x7fd0581024a0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:19.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.822+0000 7fd057200700 1 --2- 172.21.15.72:0/4163953010 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd058102a70 0x7fd05810f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:19.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.822+0000 7fd057200700 1 -- 172.21.15.72:0/4163953010 >> 172.21.15.72:0/4163953010 conn(0x7fd0580fdd40 msgr2=0x7fd058100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:44:19.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.822+0000 7fd057200700 1 -- 172.21.15.72:0/4163953010 shutdown_connections 2024-04-26T23:44:19.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.822+0000 7fd057200700 1 -- 172.21.15.72:0/4163953010 wait complete. 2024-04-26T23:44:19.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.823+0000 7fd057200700 1 Processor -- start 2024-04-26T23:44:19.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.823+0000 7fd057200700 1 -- start start 2024-04-26T23:44:19.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.824+0000 7fd057200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd0581020c0 0x7fd0581a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:19.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.824+0000 7fd057200700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd058102a70 0x7fd0581a1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:19.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.824+0000 7fd057200700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd05810fc70 0x7fd05819ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:19.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.824+0000 7fd057200700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fd058115790 con 0x7fd05810fc70 2024-04-26T23:44:19.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.824+0000 7fd057200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fd058115610 con 0x7fd0581020c0 2024-04-26T23:44:19.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.824+0000 7fd057200700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fd058115910 con 0x7fd058102a70 2024-04-26T23:44:19.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.824+0000 7fd056800700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd05810fc70 0x7fd05819ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:19.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.824+0000 7fd055e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd0581020c0 0x7fd0581a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:19.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.824+0000 7fd056800700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd05810fc70 0x7fd05819ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60696/0 (socket says 172.21.15.72:60696) 2024-04-26T23:44:19.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.824+0000 7fd056800700 1 -- 172.21.15.72:0/3037336173 learned_addr learned my addr 172.21.15.72:0/3037336173 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:44:19.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.824+0000 7fd055400700 1 --2- 172.21.15.72:0/3037336173 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd058102a70 0x7fd0581a1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:19.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.825+0000 7fd056800700 1 -- 172.21.15.72:0/3037336173 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd058102a70 msgr2=0x7fd0581a1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:19.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.825+0000 7fd056800700 1 --2- 172.21.15.72:0/3037336173 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd058102a70 0x7fd0581a1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:19.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.825+0000 7fd056800700 1 -- 172.21.15.72:0/3037336173 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd0581020c0 msgr2=0x7fd0581a18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:19.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.825+0000 7fd056800700 1 --2- 172.21.15.72:0/3037336173 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd0581020c0 0x7fd0581a18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:19.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.825+0000 7fd056800700 1 -- 172.21.15.72:0/3037336173 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd05819c1d0 con 0x7fd05810fc70 2024-04-26T23:44:19.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.825+0000 7fd056800700 1 --2- 172.21.15.72:0/3037336173 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd05810fc70 0x7fd05819ba70 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd050009fd0 tx=0x7fd05000ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:19.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.826+0000 7fd04ea00700 1 -- 172.21.15.72:0/3037336173 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0500092e0 con 0x7fd05810fc70 2024-04-26T23:44:19.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.826+0000 7fd04ea00700 1 -- 172.21.15.72:0/3037336173 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd050004bb0 con 0x7fd05810fc70 2024-04-26T23:44:19.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.826+0000 7fd04ea00700 1 -- 172.21.15.72:0/3037336173 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd050004020 con 0x7fd05810fc70 2024-04-26T23:44:19.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.826+0000 7fd057200700 1 -- 172.21.15.72:0/3037336173 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd05819c460 con 0x7fd05810fc70 2024-04-26T23:44:19.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.826+0000 7fd057200700 1 -- 172.21.15.72:0/3037336173 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0581a8780 con 0x7fd05810fc70 2024-04-26T23:44:19.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.827+0000 7fd043e00700 1 -- 172.21.15.72:0/3037336173 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd030001030 con 0x7fd05810fc70 2024-04-26T23:44:19.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.829+0000 7fd04ea00700 1 -- 172.21.15.72:0/3037336173 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fd05001b030 con 0x7fd05810fc70 2024-04-26T23:44:19.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.832+0000 7fd04ea00700 1 --2- 172.21.15.72:0/3037336173 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fd03c0a3d50 0x7fd03c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:19.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.832+0000 7fd04ea00700 1 -- 172.21.15.72:0/3037336173 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd0500d02d0 con 0x7fd05810fc70 2024-04-26T23:44:19.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.832+0000 7fd055e00700 1 --2- 172.21.15.72:0/3037336173 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fd03c0a3d50 0x7fd03c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:19.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.832+0000 7fd04ea00700 1 --2- 172.21.15.72:0/3037336173 >> [v2:172.21.15.76:6824/1120991574,v1:172.21.15.76:6825/1120991574] conn(0x7fd03c0aeeb0 0x7fd03c0b1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:19.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.832+0000 7fd04ea00700 1 -- 172.21.15.72:0/3037336173 --> [v2:172.21.15.76:6824/1120991574,v1:172.21.15.76:6825/1120991574] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd03c0b1a20 con 0x7fd03c0aeeb0 2024-04-26T23:44:19.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.832+0000 7fd04ea00700 1 -- 172.21.15.72:0/3037336173 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fd0500d7050 con 0x7fd05810fc70 2024-04-26T23:44:19.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.833+0000 7fd055400700 1 --2- 172.21.15.72:0/3037336173 >> [v2:172.21.15.76:6824/1120991574,v1:172.21.15.76:6825/1120991574] conn(0x7fd03c0aeeb0 0x7fd03c0b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:19.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.833+0000 7fd055e00700 1 --2- 172.21.15.72:0/3037336173 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fd03c0a3d50 0x7fd03c0a6210 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd05819cbb0 tx=0x7fd048005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:19.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.833+0000 7fd055400700 1 --2- 172.21.15.72:0/3037336173 >> [v2:172.21.15.76:6824/1120991574,v1:172.21.15.76:6825/1120991574] conn(0x7fd03c0aeeb0 0x7fd03c0b1350 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:19.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.834+0000 7fd04ea00700 1 -- 172.21.15.72:0/3037336173 <== osd.10 v2:172.21.15.76:6824/1120991574 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd03c0b1a20 con 0x7fd03c0aeeb0 2024-04-26T23:44:19.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.854+0000 7fd043e00700 1 -- 172.21.15.72:0/3037336173 --> [v2:172.21.15.76:6824/1120991574,v1:172.21.15.76:6825/1120991574] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd030002e60 con 0x7fd03c0aeeb0 2024-04-26T23:44:19.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.856+0000 7fd04ea00700 1 -- 172.21.15.72:0/3037336173 <== osd.10 v2:172.21.15.76:6824/1120991574 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fd030002e60 con 0x7fd03c0aeeb0 2024-04-26T23:44:19.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.856+0000 7fd043e00700 1 -- 172.21.15.72:0/3037336173 >> [v2:172.21.15.76:6824/1120991574,v1:172.21.15.76:6825/1120991574] conn(0x7fd03c0aeeb0 msgr2=0x7fd03c0b1350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:19.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.856+0000 7fd043e00700 1 --2- 172.21.15.72:0/3037336173 >> [v2:172.21.15.76:6824/1120991574,v1:172.21.15.76:6825/1120991574] conn(0x7fd03c0aeeb0 0x7fd03c0b1350 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:19.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.856+0000 7fd043e00700 1 -- 172.21.15.72:0/3037336173 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fd03c0a3d50 msgr2=0x7fd03c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:19.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.856+0000 7fd043e00700 1 --2- 172.21.15.72:0/3037336173 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fd03c0a3d50 0x7fd03c0a6210 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd05819cbb0 tx=0x7fd048005960 comp rx=0 tx=0).stop 2024-04-26T23:44:19.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.856+0000 7fd043e00700 1 -- 172.21.15.72:0/3037336173 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd05810fc70 msgr2=0x7fd05819ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:19.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.856+0000 7fd043e00700 1 --2- 172.21.15.72:0/3037336173 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd05810fc70 0x7fd05819ba70 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd050009fd0 tx=0x7fd05000ef90 comp rx=0 tx=0).stop 2024-04-26T23:44:19.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.856+0000 7fd056800700 1 -- 172.21.15.72:0/3037336173 reap_dead start 2024-04-26T23:44:19.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.856+0000 7fd043e00700 1 -- 172.21.15.72:0/3037336173 shutdown_connections 2024-04-26T23:44:19.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.856+0000 7fd043e00700 1 -- 172.21.15.72:0/3037336173 >> 172.21.15.72:0/3037336173 conn(0x7fd0580fdd40 msgr2=0x7fd058100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:44:19.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.857+0000 7fd043e00700 1 -- 172.21.15.72:0/3037336173 shutdown_connections 2024-04-26T23:44:19.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:16.857+0000 7fd043e00700 1 -- 172.21.15.72:0/3037336173 wait complete. 2024-04-26T23:44:19.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.692+0000 7f0c2bc00700 1 -- 172.21.15.72:0/2217389398 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c241094a0 msgr2=0x7f0c24109880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:19.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.692+0000 7f0c2bc00700 1 --2- 172.21.15.72:0/2217389398 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c241094a0 0x7f0c24109880 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0c180098f0 tx=0x7f0c18009c00 comp rx=0 tx=0).stop 2024-04-26T23:44:19.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.692+0000 7f0c2bc00700 1 -- 172.21.15.72:0/2217389398 shutdown_connections 2024-04-26T23:44:19.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.692+0000 7f0c2bc00700 1 --2- 172.21.15.72:0/2217389398 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0c2406bad0 0x7f0c241102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:19.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.692+0000 7f0c2bc00700 1 --2- 172.21.15.72:0/2217389398 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c241094a0 0x7f0c24109880 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:19.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.692+0000 7f0c2bc00700 1 --2- 172.21.15.72:0/2217389398 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0c2406b130 0x7f0c2406b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:19.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.692+0000 7f0c2bc00700 1 -- 172.21.15.72:0/2217389398 >> 172.21.15.72:0/2217389398 conn(0x7f0c240792d0 msgr2=0x7f0c240796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:44:19.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.693+0000 7f0c2bc00700 1 -- 172.21.15.72:0/2217389398 shutdown_connections 2024-04-26T23:44:19.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.693+0000 7f0c2bc00700 1 -- 172.21.15.72:0/2217389398 wait complete. 2024-04-26T23:44:19.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.694+0000 7f0c2bc00700 1 Processor -- start 2024-04-26T23:44:19.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.694+0000 7f0c2bc00700 1 -- start start 2024-04-26T23:44:19.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.694+0000 7f0c2bc00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0c2406b130 0x7f0c2419f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:19.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.694+0000 7f0c2bc00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0c2406bad0 0x7f0c2419fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:19.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.694+0000 7f0c2bc00700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c241094a0 0x7f0c24199980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:19.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.694+0000 7f0c2bc00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f0c24113600 con 0x7f0c2406bad0 2024-04-26T23:44:19.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.694+0000 7f0c2bc00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f0c24113480 con 0x7f0c2406b130 2024-04-26T23:44:19.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.694+0000 7f0c2bc00700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f0c24113780 con 0x7f0c241094a0 2024-04-26T23:44:19.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.694+0000 7f0c29200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0c2406b130 0x7f0c2419f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:19.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.695+0000 7f0c29200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0c2406b130 0x7f0c2419f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:45818/0 (socket says 172.21.15.72:45818) 2024-04-26T23:44:19.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.695+0000 7f0c29200700 1 -- 172.21.15.72:0/2043518697 learned_addr learned my addr 172.21.15.72:0/2043518697 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:44:19.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.695+0000 7f0c23e00700 1 --2- 172.21.15.72:0/2043518697 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0c2406bad0 0x7f0c2419fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:19.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.695+0000 7f0c29c00700 1 --2- 172.21.15.72:0/2043518697 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c241094a0 0x7f0c24199980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:19.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.695+0000 7f0c29200700 1 -- 172.21.15.72:0/2043518697 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c241094a0 msgr2=0x7f0c24199980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:19.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.695+0000 7f0c29200700 1 --2- 172.21.15.72:0/2043518697 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c241094a0 0x7f0c24199980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:19.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.695+0000 7f0c29200700 1 -- 172.21.15.72:0/2043518697 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0c2406bad0 msgr2=0x7f0c2419fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:19.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.695+0000 7f0c29200700 1 --2- 172.21.15.72:0/2043518697 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0c2406bad0 0x7f0c2419fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:19.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.695+0000 7f0c29200700 1 -- 172.21.15.72:0/2043518697 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c2419a0e0 con 0x7f0c2406b130 2024-04-26T23:44:19.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.695+0000 7f0c29200700 1 --2- 172.21.15.72:0/2043518697 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0c2406b130 0x7f0c2419f7c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0c18009230 tx=0x7f0c1800fa70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:19.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.696+0000 7f0c21600700 1 -- 172.21.15.72:0/2043518697 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c18005140 con 0x7f0c2406b130 2024-04-26T23:44:19.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.696+0000 7f0c2bc00700 1 -- 172.21.15.72:0/2043518697 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c2419a370 con 0x7f0c2406b130 2024-04-26T23:44:19.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.696+0000 7f0c21600700 1 -- 172.21.15.72:0/2043518697 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c1800b560 con 0x7f0c2406b130 2024-04-26T23:44:19.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.696+0000 7f0c21600700 1 -- 172.21.15.72:0/2043518697 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c18003970 con 0x7f0c2406b130 2024-04-26T23:44:19.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.696+0000 7f0c2bc00700 1 -- 172.21.15.72:0/2043518697 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c241a6690 con 0x7f0c2406b130 2024-04-26T23:44:19.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.697+0000 7f0c2bc00700 1 -- 172.21.15.72:0/2043518697 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0c24063bb0 con 0x7f0c2406b130 2024-04-26T23:44:19.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.699+0000 7f0c21600700 1 -- 172.21.15.72:0/2043518697 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f0c1800ba50 con 0x7f0c2406b130 2024-04-26T23:44:19.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.700+0000 7f0c21600700 1 --2- 172.21.15.72:0/2043518697 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f0c0c0a3cd0 0x7f0c0c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:19.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.701+0000 7f0c21600700 1 -- 172.21.15.72:0/2043518697 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f0c180d1ca0 con 0x7f0c2406b130 2024-04-26T23:44:19.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.701+0000 7f0c23e00700 1 --2- 172.21.15.72:0/2043518697 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f0c0c0a3cd0 0x7f0c0c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:19.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.701+0000 7f0c21600700 1 --2- 172.21.15.72:0/2043518697 >> [v2:172.21.15.72:6810/2071107815,v1:172.21.15.72:6811/2071107815] conn(0x7f0c0c0aed00 0x7f0c0c0b11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:19.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.701+0000 7f0c21600700 1 -- 172.21.15.72:0/2043518697 --> [v2:172.21.15.72:6810/2071107815,v1:172.21.15.72:6811/2071107815] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0c1800bdb0 con 0x7f0c0c0aed00 2024-04-26T23:44:19.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.701+0000 7f0c21600700 1 -- 172.21.15.72:0/2043518697 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f0c180d2050 con 0x7f0c2406b130 2024-04-26T23:44:19.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.701+0000 7f0c29c00700 1 --2- 172.21.15.72:0/2043518697 >> [v2:172.21.15.72:6810/2071107815,v1:172.21.15.72:6811/2071107815] conn(0x7f0c0c0aed00 0x7f0c0c0b11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:19.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.701+0000 7f0c23e00700 1 --2- 172.21.15.72:0/2043518697 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f0c0c0a3cd0 0x7f0c0c0a6190 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0c240799d0 tx=0x7f0c10009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:19.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.702+0000 7f0c29c00700 1 --2- 172.21.15.72:0/2043518697 >> [v2:172.21.15.72:6810/2071107815,v1:172.21.15.72:6811/2071107815] conn(0x7f0c0c0aed00 0x7f0c0c0b11a0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:19.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.703+0000 7f0c21600700 1 -- 172.21.15.72:0/2043518697 <== osd.3 v2:172.21.15.72:6810/2071107815 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0c1800bdb0 con 0x7f0c0c0aed00 2024-04-26T23:44:19.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.720+0000 7f0c2bc00700 1 -- 172.21.15.72:0/2043518697 --> [v2:172.21.15.72:6810/2071107815,v1:172.21.15.72:6811/2071107815] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0c24063fb0 con 0x7f0c0c0aed00 2024-04-26T23:44:19.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.721+0000 7f0c21600700 1 -- 172.21.15.72:0/2043518697 <== osd.3 v2:172.21.15.72:6810/2071107815 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f0c24063fb0 con 0x7f0c0c0aed00 2024-04-26T23:44:19.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.721+0000 7f0c2bc00700 1 -- 172.21.15.72:0/2043518697 >> [v2:172.21.15.72:6810/2071107815,v1:172.21.15.72:6811/2071107815] conn(0x7f0c0c0aed00 msgr2=0x7f0c0c0b11a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:19.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.721+0000 7f0c2bc00700 1 --2- 172.21.15.72:0/2043518697 >> [v2:172.21.15.72:6810/2071107815,v1:172.21.15.72:6811/2071107815] conn(0x7f0c0c0aed00 0x7f0c0c0b11a0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:19.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.722+0000 7f0c2bc00700 1 -- 172.21.15.72:0/2043518697 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f0c0c0a3cd0 msgr2=0x7f0c0c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:19.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.722+0000 7f0c2bc00700 1 --2- 172.21.15.72:0/2043518697 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f0c0c0a3cd0 0x7f0c0c0a6190 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0c240799d0 tx=0x7f0c10009310 comp rx=0 tx=0).stop 2024-04-26T23:44:19.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.722+0000 7f0c2bc00700 1 -- 172.21.15.72:0/2043518697 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0c2406b130 msgr2=0x7f0c2419f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:19.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.722+0000 7f0c2bc00700 1 --2- 172.21.15.72:0/2043518697 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0c2406b130 0x7f0c2419f7c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0c18009230 tx=0x7f0c1800fa70 comp rx=0 tx=0).stop 2024-04-26T23:44:19.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.722+0000 7f0c29c00700 1 -- 172.21.15.72:0/2043518697 reap_dead start 2024-04-26T23:44:19.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.722+0000 7f0c2bc00700 1 -- 172.21.15.72:0/2043518697 shutdown_connections 2024-04-26T23:44:19.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.722+0000 7f0c2bc00700 1 -- 172.21.15.72:0/2043518697 >> 172.21.15.72:0/2043518697 conn(0x7f0c240792d0 msgr2=0x7f0c241004d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:44:19.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.722+0000 7f0c2bc00700 1 -- 172.21.15.72:0/2043518697 shutdown_connections 2024-04-26T23:44:19.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:19.722+0000 7f0c2bc00700 1 -- 172.21.15.72:0/2043518697 wait complete. 2024-04-26T23:44:20.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.422+0000 7fc1b3e00700 1 -- 172.21.15.72:0/2135808709 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc1b410d9e0 msgr2=0x7fc1b410fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:20.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.422+0000 7fc1b3e00700 1 --2- 172.21.15.72:0/2135808709 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc1b410d9e0 0x7fc1b410fe50 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac00b0a0 tx=0x7fc1ac00b3b0 comp rx=0 tx=0).stop 2024-04-26T23:44:20.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.422+0000 7fc1b3e00700 1 -- 172.21.15.72:0/2135808709 shutdown_connections 2024-04-26T23:44:20.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.422+0000 7fc1b3e00700 1 --2- 172.21.15.72:0/2135808709 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc1b410d9e0 0x7fc1b410fe50 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:20.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.422+0000 7fc1b3e00700 1 --2- 172.21.15.72:0/2135808709 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc1b4104600 0x7fc1b41049e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:20.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.422+0000 7fc1b3e00700 1 --2- 172.21.15.72:0/2135808709 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc1b4104f20 0x7fc1b410d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:20.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.422+0000 7fc1b3e00700 1 -- 172.21.15.72:0/2135808709 >> 172.21.15.72:0/2135808709 conn(0x7fc1b40fddb0 msgr2=0x7fc1b41001d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:44:20.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.422+0000 7fc1b3e00700 1 -- 172.21.15.72:0/2135808709 shutdown_connections 2024-04-26T23:44:20.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.422+0000 7fc1b3e00700 1 -- 172.21.15.72:0/2135808709 wait complete. 2024-04-26T23:44:20.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.423+0000 7fc1b3e00700 1 Processor -- start 2024-04-26T23:44:20.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.423+0000 7fc1b3e00700 1 -- start start 2024-04-26T23:44:20.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.424+0000 7fc1b3e00700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc1b4104600 0x7fc1b4102260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:20.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.424+0000 7fc1b3e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc1b4104f20 0x7fc1b41027a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:20.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.424+0000 7fc1b3e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc1b410d9e0 0x7fc1b4102ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:20.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.424+0000 7fc1b3e00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b41135b0 con 0x7fc1b4104f20 2024-04-26T23:44:20.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.424+0000 7fc1b3e00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b4113430 con 0x7fc1b410d9e0 2024-04-26T23:44:20.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.424+0000 7fc1b3e00700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b4113730 con 0x7fc1b4104600 2024-04-26T23:44:20.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.424+0000 7fc1b2000700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc1b4104f20 0x7fc1b41027a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:20.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.424+0000 7fc1b2000700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc1b4104f20 0x7fc1b41027a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60752/0 (socket says 172.21.15.72:60752) 2024-04-26T23:44:20.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.424+0000 7fc1b2000700 1 -- 172.21.15.72:0/2431871594 learned_addr learned my addr 172.21.15.72:0/2431871594 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:44:20.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.424+0000 7fc1b2a00700 1 --2- 172.21.15.72:0/2431871594 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc1b4104600 0x7fc1b4102260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:20.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.425+0000 7fc1b2000700 1 -- 172.21.15.72:0/2431871594 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc1b4104600 msgr2=0x7fc1b4102260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:20.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.425+0000 7fc1b2000700 1 --2- 172.21.15.72:0/2431871594 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc1b4104600 0x7fc1b4102260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:20.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.425+0000 7fc1b2000700 1 -- 172.21.15.72:0/2431871594 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc1b410d9e0 msgr2=0x7fc1b4102ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:20.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.425+0000 7fc1b2000700 1 --2- 172.21.15.72:0/2431871594 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc1b410d9e0 0x7fc1b4102ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:20.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.425+0000 7fc1b2000700 1 -- 172.21.15.72:0/2431871594 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1b41a7f70 con 0x7fc1b4104f20 2024-04-26T23:44:20.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.425+0000 7fc1b2a00700 1 --2- 172.21.15.72:0/2431871594 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc1b4104600 0x7fc1b4102260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:44:20.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.425+0000 7fc1b2000700 1 --2- 172.21.15.72:0/2431871594 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc1b4104f20 0x7fc1b41027a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc1a000e9f0 tx=0x7fc1a000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:20.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.425+0000 7fc1ab400700 1 -- 172.21.15.72:0/2431871594 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1a000cbd0 con 0x7fc1b4104f20 2024-04-26T23:44:20.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.425+0000 7fc1ab400700 1 -- 172.21.15.72:0/2431871594 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1a000cd70 con 0x7fc1b4104f20 2024-04-26T23:44:20.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.425+0000 7fc1b3e00700 1 -- 172.21.15.72:0/2431871594 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1b41a8260 con 0x7fc1b4104f20 2024-04-26T23:44:20.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.425+0000 7fc1ab400700 1 -- 172.21.15.72:0/2431871594 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1a0005650 con 0x7fc1b4104f20 2024-04-26T23:44:20.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.426+0000 7fc1b3e00700 1 -- 172.21.15.72:0/2431871594 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1b41a87a0 con 0x7fc1b4104f20 2024-04-26T23:44:20.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.426+0000 7fc1b3e00700 1 -- 172.21.15.72:0/2431871594 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc1b4063b30 con 0x7fc1b4104f20 2024-04-26T23:44:20.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.430+0000 7fc1ab400700 1 -- 172.21.15.72:0/2431871594 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc1a0004750 con 0x7fc1b4104f20 2024-04-26T23:44:20.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.431+0000 7fc1ab400700 1 --2- 172.21.15.72:0/2431871594 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fc1980a3d50 0x7fc1980a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:20.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.431+0000 7fc1ab400700 1 -- 172.21.15.72:0/2431871594 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fc1a00d0fb0 con 0x7fc1b4104f20 2024-04-26T23:44:20.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.431+0000 7fc1b2a00700 1 --2- 172.21.15.72:0/2431871594 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fc1980a3d50 0x7fc1980a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:20.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.432+0000 7fc1ab400700 1 --2- 172.21.15.72:0/2431871594 >> [v2:172.21.15.72:6802/1639298875,v1:172.21.15.72:6803/1639298875] conn(0x7fc1980aeeb0 0x7fc1980b1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:20.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.432+0000 7fc1ab400700 1 -- 172.21.15.72:0/2431871594 --> [v2:172.21.15.72:6802/1639298875,v1:172.21.15.72:6803/1639298875] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc1a0017e50 con 0x7fc1980aeeb0 2024-04-26T23:44:20.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.432+0000 7fc1ab400700 1 -- 172.21.15.72:0/2431871594 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fc1a00d1360 con 0x7fc1b4104f20 2024-04-26T23:44:20.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.432+0000 7fc1b3400700 1 --2- 172.21.15.72:0/2431871594 >> [v2:172.21.15.72:6802/1639298875,v1:172.21.15.72:6803/1639298875] conn(0x7fc1980aeeb0 0x7fc1980b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:20.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.432+0000 7fc1b2a00700 1 --2- 172.21.15.72:0/2431871594 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fc1980a3d50 0x7fc1980a6210 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fc1a4011490 tx=0x7fc1a4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:20.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.432+0000 7fc1b3400700 1 --2- 172.21.15.72:0/2431871594 >> [v2:172.21.15.72:6802/1639298875,v1:172.21.15.72:6803/1639298875] conn(0x7fc1980aeeb0 0x7fc1980b1350 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:20.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.433+0000 7fc1ab400700 1 -- 172.21.15.72:0/2431871594 <== osd.0 v2:172.21.15.72:6802/1639298875 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc1a0017e50 con 0x7fc1980aeeb0 2024-04-26T23:44:20.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.451+0000 7fc1b3e00700 1 -- 172.21.15.72:0/2431871594 --> [v2:172.21.15.72:6802/1639298875,v1:172.21.15.72:6803/1639298875] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc1b4063f30 con 0x7fc1980aeeb0 2024-04-26T23:44:20.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.452+0000 7fc1ab400700 1 -- 172.21.15.72:0/2431871594 <== osd.0 v2:172.21.15.72:6802/1639298875 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fc1b4063f30 con 0x7fc1980aeeb0 2024-04-26T23:44:20.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.452+0000 7fc1b3e00700 1 -- 172.21.15.72:0/2431871594 >> [v2:172.21.15.72:6802/1639298875,v1:172.21.15.72:6803/1639298875] conn(0x7fc1980aeeb0 msgr2=0x7fc1980b1350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:20.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.452+0000 7fc1b3e00700 1 --2- 172.21.15.72:0/2431871594 >> [v2:172.21.15.72:6802/1639298875,v1:172.21.15.72:6803/1639298875] conn(0x7fc1980aeeb0 0x7fc1980b1350 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:20.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.452+0000 7fc1b3e00700 1 -- 172.21.15.72:0/2431871594 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fc1980a3d50 msgr2=0x7fc1980a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:20.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.452+0000 7fc1b3e00700 1 --2- 172.21.15.72:0/2431871594 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fc1980a3d50 0x7fc1980a6210 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fc1a4011490 tx=0x7fc1a4009260 comp rx=0 tx=0).stop 2024-04-26T23:44:20.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.452+0000 7fc1b3e00700 1 -- 172.21.15.72:0/2431871594 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc1b4104f20 msgr2=0x7fc1b41027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:20.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.452+0000 7fc1b3e00700 1 --2- 172.21.15.72:0/2431871594 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc1b4104f20 0x7fc1b41027a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc1a000e9f0 tx=0x7fc1a000ed00 comp rx=0 tx=0).stop 2024-04-26T23:44:20.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.452+0000 7fc1b3400700 1 -- 172.21.15.72:0/2431871594 reap_dead start 2024-04-26T23:44:20.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.453+0000 7fc1b3e00700 1 -- 172.21.15.72:0/2431871594 shutdown_connections 2024-04-26T23:44:20.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.453+0000 7fc1b3e00700 1 -- 172.21.15.72:0/2431871594 >> 172.21.15.72:0/2431871594 conn(0x7fc1b40fddb0 msgr2=0x7fc1b40fea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:44:20.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.453+0000 7fc1b3e00700 1 -- 172.21.15.72:0/2431871594 shutdown_connections 2024-04-26T23:44:20.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:20.453+0000 7fc1b3e00700 1 -- 172.21.15.72:0/2431871594 wait complete. 2024-04-26T23:44:21.137 INFO:teuthology.orchestra.run.smithi072.stdout:163208757312 2024-04-26T23:44:21.138 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-26T23:44:21.485 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:44:21.491 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:44:21.532 INFO:teuthology.orchestra.run.smithi072.stdout:107374182474 2024-04-26T23:44:21.532 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-26T23:44:21.589 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:21 smithi072 ceph-mon[28184]: pgmap v362: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:44:21.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:21 smithi136 ceph-mon[33050]: pgmap v362: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:44:21.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:21 smithi076 ceph-mon[32774]: pgmap v362: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:44:21.936 INFO:teuthology.orchestra.run.smithi072.stdout:309237645346 2024-04-26T23:44:21.937 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-26T23:44:22.611 INFO:teuthology.orchestra.run.smithi072.stdout:34359738459 2024-04-26T23:44:22.611 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-26T23:44:23.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:23 smithi072 ceph-mon[28184]: pgmap v363: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:44:23.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:23 smithi136 ceph-mon[33050]: pgmap v363: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:44:23.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:23 smithi076 ceph-mon[32774]: pgmap v363: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:44:24.271 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:44:24.394 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:44:24.395 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:44:24.443 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:44:25.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:25 smithi072 ceph-mon[28184]: pgmap v364: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:44:25.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:25 smithi136 ceph-mon[33050]: pgmap v364: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:44:25.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:25 smithi076 ceph-mon[32774]: pgmap v364: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:44:27.741 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:27 smithi072 ceph-mon[28184]: pgmap v365: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:44:27.741 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:27 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/581472342' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-26T23:44:27.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:27 smithi136 ceph-mon[33050]: pgmap v365: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:44:27.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:27 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/581472342' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-26T23:44:27.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:27 smithi076 ceph-mon[32774]: pgmap v365: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:44:27.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:27 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/581472342' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-26T23:44:28.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.224+0000 7f31e0200700 1 -- 172.21.15.72:0/2560878806 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31d810b640 msgr2=0x7f31d810ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:28.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.224+0000 7f31e0200700 1 --2- 172.21.15.72:0/2560878806 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31d810b640 0x7f31d810ba20 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f31cc009990 tx=0x7f31cc009ca0 comp rx=0 tx=0).stop 2024-04-26T23:44:28.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.225+0000 7f31e0200700 1 -- 172.21.15.72:0/2560878806 shutdown_connections 2024-04-26T23:44:28.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.225+0000 7f31e0200700 1 --2- 172.21.15.72:0/2560878806 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f31d8106000 0x7f31d8112510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:28.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.225+0000 7f31e0200700 1 --2- 172.21.15.72:0/2560878806 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31d810b640 0x7f31d810ba20 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:28.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.225+0000 7f31e0200700 1 --2- 172.21.15.72:0/2560878806 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f31d8105640 0x7f31d8105ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:28.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.225+0000 7f31e0200700 1 -- 172.21.15.72:0/2560878806 >> 172.21.15.72:0/2560878806 conn(0x7f31d8101130 msgr2=0x7f31d8103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:44:28.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.225+0000 7f31e0200700 1 -- 172.21.15.72:0/2560878806 shutdown_connections 2024-04-26T23:44:28.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.225+0000 7f31e0200700 1 -- 172.21.15.72:0/2560878806 wait complete. 2024-04-26T23:44:28.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.226+0000 7f31e0200700 1 Processor -- start 2024-04-26T23:44:28.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.226+0000 7f31e0200700 1 -- start start 2024-04-26T23:44:28.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.226+0000 7f31e0200700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f31d8105640 0x7f31d81a1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:28.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.226+0000 7f31e0200700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31d8106000 0x7f31d81a1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:28.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.226+0000 7f31e0200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f31d810b640 0x7f31d819bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:28.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.226+0000 7f31e0200700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f31d8115840 con 0x7f31d8106000 2024-04-26T23:44:28.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.226+0000 7f31e0200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f31d81156c0 con 0x7f31d810b640 2024-04-26T23:44:28.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.226+0000 7f31e0200700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f31d81159c0 con 0x7f31d8105640 2024-04-26T23:44:28.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.227+0000 7f31dce00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31d8106000 0x7f31d81a1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:28.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.227+0000 7f31dd800700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f31d8105640 0x7f31d81a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:28.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.227+0000 7f31dce00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31d8106000 0x7f31d81a1e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44290/0 (socket says 172.21.15.72:44290) 2024-04-26T23:44:28.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.227+0000 7f31dce00700 1 -- 172.21.15.72:0/581472342 learned_addr learned my addr 172.21.15.72:0/581472342 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:44:28.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.227+0000 7f31de200700 1 --2- 172.21.15.72:0/581472342 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f31d810b640 0x7f31d819bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:28.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.227+0000 7f31dce00700 1 -- 172.21.15.72:0/581472342 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f31d8105640 msgr2=0x7f31d81a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:28.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.227+0000 7f31dce00700 1 --2- 172.21.15.72:0/581472342 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f31d8105640 0x7f31d81a1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:28.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.227+0000 7f31dce00700 1 -- 172.21.15.72:0/581472342 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f31d810b640 msgr2=0x7f31d819bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:28.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.227+0000 7f31dce00700 1 --2- 172.21.15.72:0/581472342 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f31d810b640 0x7f31d819bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:28.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.227+0000 7f31dce00700 1 -- 172.21.15.72:0/581472342 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31d819c240 con 0x7f31d8106000 2024-04-26T23:44:28.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.227+0000 7f31dce00700 1 --2- 172.21.15.72:0/581472342 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31d8106000 0x7f31d81a1e60 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f31c400e9c0 tx=0x7f31c400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:28.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.228+0000 7f31d6000700 1 -- 172.21.15.72:0/581472342 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31c400cbd0 con 0x7f31d8106000 2024-04-26T23:44:28.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.228+0000 7f31d6000700 1 -- 172.21.15.72:0/581472342 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31c400ee60 con 0x7f31d8106000 2024-04-26T23:44:28.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.228+0000 7f31e0200700 1 -- 172.21.15.72:0/581472342 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31d819c530 con 0x7f31d8106000 2024-04-26T23:44:28.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.228+0000 7f31d6000700 1 -- 172.21.15.72:0/581472342 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31c40055c0 con 0x7f31d8106000 2024-04-26T23:44:28.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.228+0000 7f31e0200700 1 -- 172.21.15.72:0/581472342 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31d81a87f0 con 0x7f31d8106000 2024-04-26T23:44:28.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.229+0000 7f31e0200700 1 -- 172.21.15.72:0/581472342 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31d810dbd0 con 0x7f31d8106000 2024-04-26T23:44:28.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.231+0000 7f31d6000700 1 -- 172.21.15.72:0/581472342 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f31c4004750 con 0x7f31d8106000 2024-04-26T23:44:28.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.232+0000 7f31d6000700 1 --2- 172.21.15.72:0/581472342 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f31c00a3d50 0x7f31c00a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:28.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.232+0000 7f31d6000700 1 -- 172.21.15.72:0/581472342 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f31c40d0b30 con 0x7f31d8106000 2024-04-26T23:44:28.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.233+0000 7f31dd800700 1 --2- 172.21.15.72:0/581472342 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f31c00a3d50 0x7f31c00a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:28.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.233+0000 7f31dd800700 1 --2- 172.21.15.72:0/581472342 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f31c00a3d50 0x7f31c00a6210 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f31cc0098e0 tx=0x7f31cc005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:28.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.237+0000 7f31d6000700 1 -- 172.21.15.72:0/581472342 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f31c409cff0 con 0x7f31d8106000 2024-04-26T23:44:28.516 INFO:teuthology.orchestra.run.smithi072.stdout:55834574936 2024-04-26T23:44:28.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.468+0000 7f31e0200700 1 -- 172.21.15.72:0/581472342 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f31d819cfd0 con 0x7f31d8106000 2024-04-26T23:44:28.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.469+0000 7f31d6000700 1 -- 172.21.15.72:0/581472342 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f31c409c8f0 con 0x7f31d8106000 2024-04-26T23:44:28.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.472+0000 7f31e0200700 1 -- 172.21.15.72:0/581472342 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f31c00a3d50 msgr2=0x7f31c00a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:28.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.472+0000 7f31e0200700 1 --2- 172.21.15.72:0/581472342 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f31c00a3d50 0x7f31c00a6210 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f31cc0098e0 tx=0x7f31cc005960 comp rx=0 tx=0).stop 2024-04-26T23:44:28.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.472+0000 7f31e0200700 1 -- 172.21.15.72:0/581472342 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31d8106000 msgr2=0x7f31d81a1e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:28.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.472+0000 7f31e0200700 1 --2- 172.21.15.72:0/581472342 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31d8106000 0x7f31d81a1e60 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f31c400e9c0 tx=0x7f31c400ecd0 comp rx=0 tx=0).stop 2024-04-26T23:44:28.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.472+0000 7f31e0200700 1 -- 172.21.15.72:0/581472342 shutdown_connections 2024-04-26T23:44:28.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.472+0000 7f31e0200700 1 --2- 172.21.15.72:0/581472342 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f31c00a3d50 0x7f31c00a6210 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:28.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.472+0000 7f31e0200700 1 --2- 172.21.15.72:0/581472342 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f31d810b640 0x7f31d819bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:28.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.472+0000 7f31e0200700 1 --2- 172.21.15.72:0/581472342 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f31d8105640 0x7f31d81a1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:28.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.472+0000 7f31e0200700 1 --2- 172.21.15.72:0/581472342 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31d8106000 0x7f31d81a1e60 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:28.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.472+0000 7f31e0200700 1 -- 172.21.15.72:0/581472342 >> 172.21.15.72:0/581472342 conn(0x7f31d8101130 msgr2=0x7f31d81018d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:44:28.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.473+0000 7f31e0200700 1 -- 172.21.15.72:0/581472342 shutdown_connections 2024-04-26T23:44:28.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:27.473+0000 7f31e0200700 1 -- 172.21.15.72:0/581472342 wait complete. 2024-04-26T23:44:29.575 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:44:29.578 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:44:29.578 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:44:29.662 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:29 smithi072 ceph-mon[28184]: pgmap v366: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:44:29.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:29 smithi136 ceph-mon[33050]: pgmap v366: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:44:29.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:29 smithi076 ceph-mon[32774]: pgmap v366: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:44:31.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.609+0000 7f311ee00700 1 -- 172.21.15.72:0/695561978 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3118105fd0 msgr2=0x7f31181124e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:31.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.609+0000 7f311ee00700 1 --2- 172.21.15.72:0/695561978 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3118105fd0 0x7f31181124e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f311000b0a0 tx=0x7f311000b3b0 comp rx=0 tx=0).stop 2024-04-26T23:44:31.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.609+0000 7f311ee00700 1 -- 172.21.15.72:0/695561978 shutdown_connections 2024-04-26T23:44:31.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.609+0000 7f311ee00700 1 --2- 172.21.15.72:0/695561978 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3118105fd0 0x7f31181124e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:31.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.609+0000 7f311ee00700 1 --2- 172.21.15.72:0/695561978 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f311810b6a0 0x7f311810ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:31.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.609+0000 7f311ee00700 1 --2- 172.21.15.72:0/695561978 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3118105610 0x7f3118105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:31.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.609+0000 7f311ee00700 1 -- 172.21.15.72:0/695561978 >> 172.21.15.72:0/695561978 conn(0x7f3118101100 msgr2=0x7f3118103520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:44:31.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.610+0000 7f311ee00700 1 -- 172.21.15.72:0/695561978 shutdown_connections 2024-04-26T23:44:31.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.610+0000 7f311ee00700 1 -- 172.21.15.72:0/695561978 wait complete. 2024-04-26T23:44:31.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.611+0000 7f311ee00700 1 Processor -- start 2024-04-26T23:44:31.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.611+0000 7f311ee00700 1 -- start start 2024-04-26T23:44:31.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.611+0000 7f311ee00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3118105610 0x7f31181a1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:31.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.611+0000 7f311ee00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3118105fd0 0x7f31181a1d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:31.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.611+0000 7f311ee00700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f311810b6a0 0x7f311819ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:31.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.611+0000 7f311ee00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f3118115810 con 0x7f3118105fd0 2024-04-26T23:44:31.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.611+0000 7f311ee00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3118115690 con 0x7f3118105610 2024-04-26T23:44:31.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.611+0000 7f311ee00700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f3118115990 con 0x7f311810b6a0 2024-04-26T23:44:31.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.612+0000 7f3117e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3118105610 0x7f31181a1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:31.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.612+0000 7f3117400700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3118105fd0 0x7f31181a1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:31.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.612+0000 7f3117e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3118105610 0x7f31181a1840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:56406/0 (socket says 172.21.15.72:56406) 2024-04-26T23:44:31.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.612+0000 7f3117e00700 1 -- 172.21.15.72:0/1439749121 learned_addr learned my addr 172.21.15.72:0/1439749121 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:44:31.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.612+0000 7f311ce00700 1 --2- 172.21.15.72:0/1439749121 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f311810b6a0 0x7f311819ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:31.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.612+0000 7f3117e00700 1 -- 172.21.15.72:0/1439749121 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f311810b6a0 msgr2=0x7f311819ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:31.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.612+0000 7f3117e00700 1 --2- 172.21.15.72:0/1439749121 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f311810b6a0 0x7f311819ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:31.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.612+0000 7f3117e00700 1 -- 172.21.15.72:0/1439749121 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3118105fd0 msgr2=0x7f31181a1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:31.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.612+0000 7f3117e00700 1 --2- 172.21.15.72:0/1439749121 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3118105fd0 0x7f31181a1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:31.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.612+0000 7f3117e00700 1 -- 172.21.15.72:0/1439749121 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f311819c1b0 con 0x7f3118105610 2024-04-26T23:44:31.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.612+0000 7f3117e00700 1 --2- 172.21.15.72:0/1439749121 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3118105610 0x7f31181a1840 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f310800d980 tx=0x7f310800dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:31.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.613+0000 7f3114c00700 1 -- 172.21.15.72:0/1439749121 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3108017070 con 0x7f3118105610 2024-04-26T23:44:31.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.613+0000 7f3114c00700 1 -- 172.21.15.72:0/1439749121 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f310800fd40 con 0x7f3118105610 2024-04-26T23:44:31.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.613+0000 7f3114c00700 1 -- 172.21.15.72:0/1439749121 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f310800e3b0 con 0x7f3118105610 2024-04-26T23:44:31.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.613+0000 7f311ee00700 1 -- 172.21.15.72:0/1439749121 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f311819c4a0 con 0x7f3118105610 2024-04-26T23:44:31.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.613+0000 7f311ee00700 1 -- 172.21.15.72:0/1439749121 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31181a8710 con 0x7f3118105610 2024-04-26T23:44:31.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.614+0000 7f311ee00700 1 -- 172.21.15.72:0/1439749121 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f311806a110 con 0x7f3118105610 2024-04-26T23:44:31.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.616+0000 7f3114c00700 1 -- 172.21.15.72:0/1439749121 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f310800f890 con 0x7f3118105610 2024-04-26T23:44:31.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.617+0000 7f3114c00700 1 --2- 172.21.15.72:0/1439749121 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f30fc0a3d50 0x7f30fc0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:31.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.617+0000 7f3114c00700 1 -- 172.21.15.72:0/1439749121 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f31080d0360 con 0x7f3118105610 2024-04-26T23:44:31.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.617+0000 7f3117400700 1 --2- 172.21.15.72:0/1439749121 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f30fc0a3d50 0x7f30fc0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:31.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.618+0000 7f3117400700 1 --2- 172.21.15.72:0/1439749121 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f30fc0a3d50 0x7f30fc0a6210 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f311810ad20 tx=0x7f3104009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:31.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.622+0000 7f3114c00700 1 -- 172.21.15.72:0/1439749121 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f310809c820 con 0x7f3118105610 2024-04-26T23:44:31.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:31 smithi072 ceph-mon[28184]: pgmap v367: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-26T23:44:31.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:31 smithi136 ceph-mon[33050]: pgmap v367: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-26T23:44:31.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.861+0000 7f311ee00700 1 -- 172.21.15.72:0/1439749121 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f311802e490 con 0x7f3118105610 2024-04-26T23:44:31.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.861+0000 7f3114c00700 1 -- 172.21.15.72:0/1439749121 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f310809c120 con 0x7f3118105610 2024-04-26T23:44:31.866 INFO:teuthology.orchestra.run.smithi072.stdout:141733920840 2024-04-26T23:44:31.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.864+0000 7f311ee00700 1 -- 172.21.15.72:0/1439749121 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f30fc0a3d50 msgr2=0x7f30fc0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:31.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.864+0000 7f311ee00700 1 --2- 172.21.15.72:0/1439749121 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f30fc0a3d50 0x7f30fc0a6210 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f311810ad20 tx=0x7f3104009260 comp rx=0 tx=0).stop 2024-04-26T23:44:31.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.864+0000 7f311ee00700 1 -- 172.21.15.72:0/1439749121 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3118105610 msgr2=0x7f31181a1840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:31.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.864+0000 7f311ee00700 1 --2- 172.21.15.72:0/1439749121 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3118105610 0x7f31181a1840 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f310800d980 tx=0x7f310800dc90 comp rx=0 tx=0).stop 2024-04-26T23:44:31.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.865+0000 7f311ee00700 1 -- 172.21.15.72:0/1439749121 shutdown_connections 2024-04-26T23:44:31.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.865+0000 7f311ee00700 1 --2- 172.21.15.72:0/1439749121 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f30fc0a3d50 0x7f30fc0a6210 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:31.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.865+0000 7f311ee00700 1 --2- 172.21.15.72:0/1439749121 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f311810b6a0 0x7f311819ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:31.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.865+0000 7f311ee00700 1 --2- 172.21.15.72:0/1439749121 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3118105610 0x7f31181a1840 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:31.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.865+0000 7f311ee00700 1 --2- 172.21.15.72:0/1439749121 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3118105fd0 0x7f31181a1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:31.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.865+0000 7f311ee00700 1 -- 172.21.15.72:0/1439749121 >> 172.21.15.72:0/1439749121 conn(0x7f3118101100 msgr2=0x7f3118102760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:44:31.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.865+0000 7f311ee00700 1 -- 172.21.15.72:0/1439749121 shutdown_connections 2024-04-26T23:44:31.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:31.865+0000 7f311ee00700 1 -- 172.21.15.72:0/1439749121 wait complete. 2024-04-26T23:44:31.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:31 smithi076 ceph-mon[32774]: pgmap v367: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-26T23:44:31.934 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:44:32.368 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574930 got 55834574936 for osd.1 2024-04-26T23:44:32.368 DEBUG:teuthology.parallel:result is None 2024-04-26T23:44:32.637 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:32 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/1439749121' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-26T23:44:32.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:32 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/1439749121' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-26T23:44:32.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:32 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/1439749121' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-26T23:44:33.748 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:33 smithi072 ceph-mon[28184]: pgmap v368: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:44:33.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:33 smithi136 ceph-mon[33050]: pgmap v368: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:44:33.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:33 smithi076 ceph-mon[32774]: pgmap v368: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:44:34.272 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920836 got 141733920840 for osd.4 2024-04-26T23:44:34.272 DEBUG:teuthology.parallel:result is None 2024-04-26T23:44:35.071 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:44:35.594 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:35 smithi072 ceph-mon[28184]: pgmap v369: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:44:35.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:35 smithi136 ceph-mon[33050]: pgmap v369: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:44:35.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.759+0000 7f0d71200700 1 -- 172.21.15.72:0/1686742889 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d6c108350 msgr2=0x7f0d6c06b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:35.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.759+0000 7f0d71200700 1 --2- 172.21.15.72:0/1686742889 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d6c108350 0x7f0d6c06b130 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c009990 tx=0x7f0d5c009ca0 comp rx=0 tx=0).stop 2024-04-26T23:44:35.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.759+0000 7f0d71200700 1 -- 172.21.15.72:0/1686742889 shutdown_connections 2024-04-26T23:44:35.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.759+0000 7f0d71200700 1 --2- 172.21.15.72:0/1686742889 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d6c105e60 0x7f0d6c1062f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:35.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.759+0000 7f0d71200700 1 --2- 172.21.15.72:0/1686742889 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d6c108350 0x7f0d6c06b130 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:35.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.759+0000 7f0d71200700 1 --2- 172.21.15.72:0/1686742889 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d6c06b670 0x7f0d6c06bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:35.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.759+0000 7f0d71200700 1 -- 172.21.15.72:0/1686742889 >> 172.21.15.72:0/1686742889 conn(0x7f0d6c0792d0 msgr2=0x7f0d6c0796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:44:35.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.760+0000 7f0d71200700 1 -- 172.21.15.72:0/1686742889 shutdown_connections 2024-04-26T23:44:35.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.760+0000 7f0d71200700 1 -- 172.21.15.72:0/1686742889 wait complete. 2024-04-26T23:44:35.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.761+0000 7f0d71200700 1 Processor -- start 2024-04-26T23:44:35.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.761+0000 7f0d71200700 1 -- start start 2024-04-26T23:44:35.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.761+0000 7f0d71200700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d6c06b670 0x7f0d6c1a0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:35.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.761+0000 7f0d71200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d6c105e60 0x7f0d6c1a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:35.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.761+0000 7f0d71200700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d6c108350 0x7f0d6c1a4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:35.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.761+0000 7f0d71200700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f0d6c119b90 con 0x7f0d6c06b670 2024-04-26T23:44:35.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.761+0000 7f0d71200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f0d6c119a10 con 0x7f0d6c105e60 2024-04-26T23:44:35.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.761+0000 7f0d71200700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f0d6c119d10 con 0x7f0d6c108350 2024-04-26T23:44:35.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.761+0000 7f0d69e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d6c06b670 0x7f0d6c1a0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:35.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.762+0000 7f0d69400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d6c105e60 0x7f0d6c1a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:35.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.762+0000 7f0d6a800700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d6c108350 0x7f0d6c1a4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:35.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.762+0000 7f0d69e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d6c06b670 0x7f0d6c1a0140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44338/0 (socket says 172.21.15.72:44338) 2024-04-26T23:44:35.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.762+0000 7f0d69400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d6c105e60 0x7f0d6c1a0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:56426/0 (socket says 172.21.15.72:56426) 2024-04-26T23:44:35.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.762+0000 7f0d69e00700 1 -- 172.21.15.72:0/1174615457 learned_addr learned my addr 172.21.15.72:0/1174615457 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:44:35.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.762+0000 7f0d69e00700 1 -- 172.21.15.72:0/1174615457 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d6c108350 msgr2=0x7f0d6c1a4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:35.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.762+0000 7f0d69e00700 1 --2- 172.21.15.72:0/1174615457 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d6c108350 0x7f0d6c1a4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:35.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.762+0000 7f0d69e00700 1 -- 172.21.15.72:0/1174615457 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d6c105e60 msgr2=0x7f0d6c1a0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:35.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.762+0000 7f0d69e00700 1 --2- 172.21.15.72:0/1174615457 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d6c105e60 0x7f0d6c1a0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:35.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.762+0000 7f0d69e00700 1 -- 172.21.15.72:0/1174615457 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d6c1a5150 con 0x7f0d6c06b670 2024-04-26T23:44:35.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.762+0000 7f0d69400700 1 --2- 172.21.15.72:0/1174615457 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d6c105e60 0x7f0d6c1a0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:44:35.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.762+0000 7f0d69e00700 1 --2- 172.21.15.72:0/1174615457 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d6c06b670 0x7f0d6c1a0140 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c0057f0 tx=0x7f0d5c00bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:35.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.763+0000 7f0d62a00700 1 -- 172.21.15.72:0/1174615457 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d5c026070 con 0x7f0d6c06b670 2024-04-26T23:44:35.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.763+0000 7f0d62a00700 1 -- 172.21.15.72:0/1174615457 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0d5c021410 con 0x7f0d6c06b670 2024-04-26T23:44:35.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.763+0000 7f0d71200700 1 -- 172.21.15.72:0/1174615457 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d6c1a53e0 con 0x7f0d6c06b670 2024-04-26T23:44:35.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.763+0000 7f0d62a00700 1 -- 172.21.15.72:0/1174615457 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d5c017650 con 0x7f0d6c06b670 2024-04-26T23:44:35.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.763+0000 7f0d71200700 1 -- 172.21.15.72:0/1174615457 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d6c1accc0 con 0x7f0d6c06b670 2024-04-26T23:44:35.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.764+0000 7f0d71200700 1 -- 172.21.15.72:0/1174615457 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d6c06a110 con 0x7f0d6c06b670 2024-04-26T23:44:35.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.768+0000 7f0d62a00700 1 -- 172.21.15.72:0/1174615457 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f0d5c029070 con 0x7f0d6c06b670 2024-04-26T23:44:35.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.769+0000 7f0d62a00700 1 --2- 172.21.15.72:0/1174615457 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f0d500a3cd0 0x7f0d500a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:35.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.770+0000 7f0d62a00700 1 -- 172.21.15.72:0/1174615457 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f0d5c0d1730 con 0x7f0d6c06b670 2024-04-26T23:44:35.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.770+0000 7f0d69400700 1 --2- 172.21.15.72:0/1174615457 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f0d500a3cd0 0x7f0d500a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:35.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.771+0000 7f0d69400700 1 --2- 172.21.15.72:0/1174615457 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f0d500a3cd0 0x7f0d500a6190 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c079af0 tx=0x7f0d54009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:35.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.771+0000 7f0d62a00700 1 -- 172.21.15.72:0/1174615457 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f0d5c013070 con 0x7f0d6c06b670 2024-04-26T23:44:35.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.870+0000 7ff788e00700 1 -- 172.21.15.72:0/165225783 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff784104300 msgr2=0x7ff78406b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:35.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.870+0000 7ff788e00700 1 --2- 172.21.15.72:0/165225783 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff784104300 0x7ff78406b990 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff7740098f0 tx=0x7ff774009c00 comp rx=0 tx=0).stop 2024-04-26T23:44:35.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.870+0000 7ff788e00700 1 -- 172.21.15.72:0/165225783 shutdown_connections 2024-04-26T23:44:35.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.870+0000 7ff788e00700 1 --2- 172.21.15.72:0/165225783 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff78410fbe0 0x7ff784112010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:35.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.870+0000 7ff788e00700 1 --2- 172.21.15.72:0/165225783 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff784104300 0x7ff78406b990 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:35.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.870+0000 7ff788e00700 1 --2- 172.21.15.72:0/165225783 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff78406bed0 0x7ff78406c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:35.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.870+0000 7ff788e00700 1 -- 172.21.15.72:0/165225783 >> 172.21.15.72:0/165225783 conn(0x7ff784079160 msgr2=0x7ff784079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:44:35.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.870+0000 7ff788e00700 1 -- 172.21.15.72:0/165225783 shutdown_connections 2024-04-26T23:44:35.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.870+0000 7ff788e00700 1 -- 172.21.15.72:0/165225783 wait complete. 2024-04-26T23:44:35.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.872+0000 7ff788e00700 1 Processor -- start 2024-04-26T23:44:35.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.872+0000 7ff788e00700 1 -- start start 2024-04-26T23:44:35.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.872+0000 7ff788e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff78406bed0 0x7ff7841a1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:35.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.872+0000 7ff788e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff78410fbe0 0x7ff7841a1d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:35.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.872+0000 7ff788e00700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff78419ba10 0x7ff78419be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:35.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.872+0000 7ff788e00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7ff784115730 con 0x7ff78410fbe0 2024-04-26T23:44:35.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.872+0000 7ff788e00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff7841155b0 con 0x7ff78406bed0 2024-04-26T23:44:35.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.872+0000 7ff788e00700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff7841158b0 con 0x7ff78419ba10 2024-04-26T23:44:35.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.873+0000 7ff783400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff78406bed0 0x7ff7841a1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:35.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.873+0000 7ff783400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff78406bed0 0x7ff7841a1850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:56456/0 (socket says 172.21.15.72:56456) 2024-04-26T23:44:35.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.873+0000 7ff783400700 1 -- 172.21.15.72:0/1046750111 learned_addr learned my addr 172.21.15.72:0/1046750111 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:44:35.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.873+0000 7ff782a00700 1 --2- 172.21.15.72:0/1046750111 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff78410fbe0 0x7ff7841a1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:35.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.873+0000 7ff783400700 1 -- 172.21.15.72:0/1046750111 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff78419ba10 msgr2=0x7ff78419be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:35.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.873+0000 7ff783400700 1 --2- 172.21.15.72:0/1046750111 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff78419ba10 0x7ff78419be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:35.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.873+0000 7ff783400700 1 -- 172.21.15.72:0/1046750111 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff78410fbe0 msgr2=0x7ff7841a1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:35.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.873+0000 7ff783400700 1 --2- 172.21.15.72:0/1046750111 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff78410fbe0 0x7ff7841a1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:35.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.873+0000 7ff783400700 1 -- 172.21.15.72:0/1046750111 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff78419c590 con 0x7ff78406bed0 2024-04-26T23:44:35.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.873+0000 7ff783400700 1 --2- 172.21.15.72:0/1046750111 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff78406bed0 0x7ff7841a1850 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff77400f990 tx=0x7ff77400f9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:35.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.874+0000 7ff77be00700 1 -- 172.21.15.72:0/1046750111 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff774027070 con 0x7ff78406bed0 2024-04-26T23:44:35.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.874+0000 7ff77be00700 1 -- 172.21.15.72:0/1046750111 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff774005570 con 0x7ff78406bed0 2024-04-26T23:44:35.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.874+0000 7ff77be00700 1 -- 172.21.15.72:0/1046750111 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff774004930 con 0x7ff78406bed0 2024-04-26T23:44:35.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.874+0000 7ff788e00700 1 -- 172.21.15.72:0/1046750111 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff78419c820 con 0x7ff78406bed0 2024-04-26T23:44:35.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.874+0000 7ff788e00700 1 -- 172.21.15.72:0/1046750111 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7841a8950 con 0x7ff78406bed0 2024-04-26T23:44:35.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.875+0000 7ff788e00700 1 -- 172.21.15.72:0/1046750111 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff78406a110 con 0x7ff78406bed0 2024-04-26T23:44:35.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.877+0000 7ff77be00700 1 -- 172.21.15.72:0/1046750111 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff77402a070 con 0x7ff78406bed0 2024-04-26T23:44:35.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.878+0000 7ff77be00700 1 --2- 172.21.15.72:0/1046750111 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff7680a3cd0 0x7ff7680a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:35.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.879+0000 7ff77be00700 1 -- 172.21.15.72:0/1046750111 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff7740d1310 con 0x7ff78406bed0 2024-04-26T23:44:35.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.879+0000 7ff782a00700 1 --2- 172.21.15.72:0/1046750111 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff7680a3cd0 0x7ff7680a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:35.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.879+0000 7ff782a00700 1 --2- 172.21.15.72:0/1046750111 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff7680a3cd0 0x7ff7680a6190 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff770009750 tx=0x7ff770009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:35.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:35.883+0000 7ff77be00700 1 -- 172.21.15.72:0/1046750111 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff77405f060 con 0x7ff78406bed0 2024-04-26T23:44:35.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:35 smithi076 ceph-mon[32774]: pgmap v369: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:44:36.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:36.007+0000 7f0d71200700 1 -- 172.21.15.72:0/1174615457 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f0d6c1ace90 con 0x7f0d6c06b670 2024-04-26T23:44:36.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:36.008+0000 7f0d62a00700 1 -- 172.21.15.72:0/1174615457 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0d5c09dd10 con 0x7f0d6c06b670 2024-04-26T23:44:36.012 INFO:teuthology.orchestra.run.smithi072.stdout:77309411412 2024-04-26T23:44:36.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:36.011+0000 7f0d71200700 1 -- 172.21.15.72:0/1174615457 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f0d500a3cd0 msgr2=0x7f0d500a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:36.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:36.011+0000 7f0d71200700 1 --2- 172.21.15.72:0/1174615457 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f0d500a3cd0 0x7f0d500a6190 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c079af0 tx=0x7f0d54009260 comp rx=0 tx=0).stop 2024-04-26T23:44:36.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:36.011+0000 7f0d71200700 1 -- 172.21.15.72:0/1174615457 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d6c06b670 msgr2=0x7f0d6c1a0140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:36.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:36.011+0000 7f0d71200700 1 --2- 172.21.15.72:0/1174615457 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d6c06b670 0x7f0d6c1a0140 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c0057f0 tx=0x7f0d5c00bfd0 comp rx=0 tx=0).stop 2024-04-26T23:44:36.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:36.011+0000 7f0d71200700 1 -- 172.21.15.72:0/1174615457 shutdown_connections 2024-04-26T23:44:36.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:36.011+0000 7f0d71200700 1 --2- 172.21.15.72:0/1174615457 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f0d500a3cd0 0x7f0d500a6190 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:36.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:36.011+0000 7f0d71200700 1 --2- 172.21.15.72:0/1174615457 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d6c108350 0x7f0d6c1a4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:36.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:36.011+0000 7f0d71200700 1 --2- 172.21.15.72:0/1174615457 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d6c06b670 0x7f0d6c1a0140 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:36.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:36.011+0000 7f0d71200700 1 --2- 172.21.15.72:0/1174615457 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d6c105e60 0x7f0d6c1a0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:36.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:36.011+0000 7f0d71200700 1 -- 172.21.15.72:0/1174615457 >> 172.21.15.72:0/1174615457 conn(0x7f0d6c0792d0 msgr2=0x7f0d6c101310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:44:36.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:36.012+0000 7f0d71200700 1 -- 172.21.15.72:0/1174615457 shutdown_connections 2024-04-26T23:44:36.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:36.012+0000 7f0d71200700 1 -- 172.21.15.72:0/1174615457 wait complete. 2024-04-26T23:44:36.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:36.121+0000 7ff788e00700 1 -- 172.21.15.72:0/1046750111 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7ff784104330 con 0x7ff78406bed0 2024-04-26T23:44:36.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:36.122+0000 7ff77be00700 1 -- 172.21.15.72:0/1046750111 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7ff774013030 con 0x7ff78406bed0 2024-04-26T23:44:36.126 INFO:teuthology.orchestra.run.smithi072.stdout:197568495677 2024-04-26T23:44:36.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:36.125+0000 7ff788e00700 1 -- 172.21.15.72:0/1046750111 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff7680a3cd0 msgr2=0x7ff7680a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:36.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:36.125+0000 7ff788e00700 1 --2- 172.21.15.72:0/1046750111 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff7680a3cd0 0x7ff7680a6190 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff770009750 tx=0x7ff770009310 comp rx=0 tx=0).stop 2024-04-26T23:44:36.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:36.125+0000 7ff788e00700 1 -- 172.21.15.72:0/1046750111 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff78406bed0 msgr2=0x7ff7841a1850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:36.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:36.125+0000 7ff788e00700 1 --2- 172.21.15.72:0/1046750111 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff78406bed0 0x7ff7841a1850 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff77400f990 tx=0x7ff77400f9c0 comp rx=0 tx=0).stop 2024-04-26T23:44:36.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:36.125+0000 7ff788e00700 1 -- 172.21.15.72:0/1046750111 shutdown_connections 2024-04-26T23:44:36.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:36.125+0000 7ff788e00700 1 --2- 172.21.15.72:0/1046750111 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff7680a3cd0 0x7ff7680a6190 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:36.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:36.125+0000 7ff788e00700 1 --2- 172.21.15.72:0/1046750111 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff78419ba10 0x7ff78419be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:36.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:36.125+0000 7ff788e00700 1 --2- 172.21.15.72:0/1046750111 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff78406bed0 0x7ff7841a1850 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:36.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:36.125+0000 7ff788e00700 1 --2- 172.21.15.72:0/1046750111 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff78410fbe0 0x7ff7841a1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:36.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:36.125+0000 7ff788e00700 1 -- 172.21.15.72:0/1046750111 >> 172.21.15.72:0/1046750111 conn(0x7ff784079160 msgr2=0x7ff7841014d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:44:36.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:36.125+0000 7ff788e00700 1 -- 172.21.15.72:0/1046750111 shutdown_connections 2024-04-26T23:44:36.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:36.125+0000 7ff788e00700 1 -- 172.21.15.72:0/1046750111 wait complete. 2024-04-26T23:44:36.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:36 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/1174615457' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-26T23:44:36.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:36 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/1046750111' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-26T23:44:36.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:36 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/1174615457' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-26T23:44:36.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:36 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/1046750111' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-26T23:44:37.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:36 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/1174615457' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-26T23:44:37.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:36 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/1046750111' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-26T23:44:37.727 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:37 smithi072 ceph-mon[28184]: pgmap v370: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:44:37.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:37 smithi076 ceph-mon[32774]: pgmap v370: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:44:38.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:37 smithi136 ceph-mon[33050]: pgmap v370: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:44:38.496 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495670 got 197568495677 for osd.6 2024-04-26T23:44:38.496 DEBUG:teuthology.parallel:result is None 2024-04-26T23:44:38.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:38 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/3963796262' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-26T23:44:38.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:38 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/3963796262' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-26T23:44:38.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:37.745+0000 7fdf85a00700 1 -- 172.21.15.72:0/2636911002 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf800760e0 msgr2=0x7fdf80113e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:39.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:37.745+0000 7fdf85a00700 1 --2- 172.21.15.72:0/2636911002 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf800760e0 0x7fdf80113e20 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c0098f0 tx=0x7fdf6c009c00 comp rx=0 tx=0).stop 2024-04-26T23:44:39.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:37.745+0000 7fdf85a00700 1 -- 172.21.15.72:0/2636911002 shutdown_connections 2024-04-26T23:44:39.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:37.745+0000 7fdf85a00700 1 --2- 172.21.15.72:0/2636911002 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdf80114360 0x7fdf80116750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:39.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:37.745+0000 7fdf85a00700 1 --2- 172.21.15.72:0/2636911002 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf800757c0 0x7fdf80075ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:39.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:37.745+0000 7fdf85a00700 1 --2- 172.21.15.72:0/2636911002 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf800760e0 0x7fdf80113e20 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:39.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:37.745+0000 7fdf85a00700 1 -- 172.21.15.72:0/2636911002 >> 172.21.15.72:0/2636911002 conn(0x7fdf800feea0 msgr2=0x7fdf801012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:44:39.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:37.745+0000 7fdf85a00700 1 -- 172.21.15.72:0/2636911002 shutdown_connections 2024-04-26T23:44:39.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:37.745+0000 7fdf85a00700 1 -- 172.21.15.72:0/2636911002 wait complete. 2024-04-26T23:44:39.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:37.747+0000 7fdf85a00700 1 Processor -- start 2024-04-26T23:44:39.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:37.747+0000 7fdf85a00700 1 -- start start 2024-04-26T23:44:39.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:37.747+0000 7fdf85a00700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdf800757c0 0x7fdf801a00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:39.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:37.747+0000 7fdf85a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf800760e0 0x7fdf801a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:39.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:37.747+0000 7fdf85a00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf80114360 0x7fdf801a4980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:39.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:37.747+0000 7fdf85a00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fdf80119ba0 con 0x7fdf80114360 2024-04-26T23:44:39.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:37.747+0000 7fdf85a00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fdf80119a20 con 0x7fdf800760e0 2024-04-26T23:44:39.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:37.747+0000 7fdf85a00700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fdf80119d20 con 0x7fdf800757c0 2024-04-26T23:44:39.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:37.748+0000 7fdf85000700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf80114360 0x7fdf801a4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:39.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:37.748+0000 7fdf85000700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf80114360 0x7fdf801a4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37534/0 (socket says 172.21.15.72:37534) 2024-04-26T23:44:39.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:37.748+0000 7fdf85000700 1 -- 172.21.15.72:0/3963796262 learned_addr learned my addr 172.21.15.72:0/3963796262 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:44:39.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:37.748+0000 7fdf7fe00700 1 --2- 172.21.15.72:0/3963796262 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdf800757c0 0x7fdf801a00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:39.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:37.748+0000 7fdf7f400700 1 --2- 172.21.15.72:0/3963796262 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf800760e0 0x7fdf801a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:39.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:37.748+0000 7fdf85000700 1 -- 172.21.15.72:0/3963796262 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdf800757c0 msgr2=0x7fdf801a00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:39.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:37.748+0000 7fdf85000700 1 --2- 172.21.15.72:0/3963796262 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdf800757c0 0x7fdf801a00b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:39.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:37.748+0000 7fdf85000700 1 -- 172.21.15.72:0/3963796262 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf800760e0 msgr2=0x7fdf801a05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:39.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:37.748+0000 7fdf85000700 1 --2- 172.21.15.72:0/3963796262 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf800760e0 0x7fdf801a05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:39.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:37.748+0000 7fdf85000700 1 -- 172.21.15.72:0/3963796262 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf801a50c0 con 0x7fdf80114360 2024-04-26T23:44:39.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:37.749+0000 7fdf85000700 1 --2- 172.21.15.72:0/3963796262 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf80114360 0x7fdf801a4980 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fdf7800ef90 tx=0x7fdf7800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:39.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:37.749+0000 7fdf7cc00700 1 -- 172.21.15.72:0/3963796262 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf780092e0 con 0x7fdf80114360 2024-04-26T23:44:39.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:37.749+0000 7fdf85a00700 1 -- 172.21.15.72:0/3963796262 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf801a5350 con 0x7fdf80114360 2024-04-26T23:44:39.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:37.749+0000 7fdf7cc00700 1 -- 172.21.15.72:0/3963796262 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf78004bb0 con 0x7fdf80114360 2024-04-26T23:44:39.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:37.749+0000 7fdf7cc00700 1 -- 172.21.15.72:0/3963796262 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf78004280 con 0x7fdf80114360 2024-04-26T23:44:39.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:37.749+0000 7fdf85a00700 1 -- 172.21.15.72:0/3963796262 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf801acc30 con 0x7fdf80114360 2024-04-26T23:44:39.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:37.750+0000 7fdf85a00700 1 -- 172.21.15.72:0/3963796262 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf8006a110 con 0x7fdf80114360 2024-04-26T23:44:39.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:37.752+0000 7fdf7cc00700 1 -- 172.21.15.72:0/3963796262 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fdf7801b030 con 0x7fdf80114360 2024-04-26T23:44:39.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:37.755+0000 7fdf7cc00700 1 --2- 172.21.15.72:0/3963796262 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fdf640a3cd0 0x7fdf640a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:39.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:37.755+0000 7fdf7cc00700 1 -- 172.21.15.72:0/3963796262 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fdf780d03b0 con 0x7fdf80114360 2024-04-26T23:44:39.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:37.755+0000 7fdf7fe00700 1 --2- 172.21.15.72:0/3963796262 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fdf640a3cd0 0x7fdf640a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:39.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:37.756+0000 7fdf7fe00700 1 --2- 172.21.15.72:0/3963796262 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fdf640a3cd0 0x7fdf640a6190 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fdf70005fd0 tx=0x7fdf70009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:39.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:37.760+0000 7fdf7cc00700 1 -- 172.21.15.72:0/3963796262 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fdf78022020 con 0x7fdf80114360 2024-04-26T23:44:39.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:38.001+0000 7fdf85a00700 1 -- 172.21.15.72:0/3963796262 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fdf8002e490 con 0x7fdf80114360 2024-04-26T23:44:39.003 INFO:teuthology.orchestra.run.smithi072.stdout:219043332152 2024-04-26T23:44:39.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:38.002+0000 7fdf7cc00700 1 -- 172.21.15.72:0/3963796262 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fdf78010040 con 0x7fdf80114360 2024-04-26T23:44:39.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:38.004+0000 7fdf85a00700 1 -- 172.21.15.72:0/3963796262 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fdf640a3cd0 msgr2=0x7fdf640a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:39.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:38.004+0000 7fdf85a00700 1 --2- 172.21.15.72:0/3963796262 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fdf640a3cd0 0x7fdf640a6190 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fdf70005fd0 tx=0x7fdf70009310 comp rx=0 tx=0).stop 2024-04-26T23:44:39.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:38.004+0000 7fdf85a00700 1 -- 172.21.15.72:0/3963796262 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf80114360 msgr2=0x7fdf801a4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:39.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:38.004+0000 7fdf85a00700 1 --2- 172.21.15.72:0/3963796262 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf80114360 0x7fdf801a4980 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fdf7800ef90 tx=0x7fdf7800c3a0 comp rx=0 tx=0).stop 2024-04-26T23:44:39.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:38.005+0000 7fdf85a00700 1 -- 172.21.15.72:0/3963796262 shutdown_connections 2024-04-26T23:44:39.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:38.005+0000 7fdf85a00700 1 --2- 172.21.15.72:0/3963796262 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fdf640a3cd0 0x7fdf640a6190 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:39.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:38.005+0000 7fdf85a00700 1 --2- 172.21.15.72:0/3963796262 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf80114360 0x7fdf801a4980 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:39.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:38.005+0000 7fdf85a00700 1 --2- 172.21.15.72:0/3963796262 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdf800757c0 0x7fdf801a00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:39.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:38.005+0000 7fdf85a00700 1 --2- 172.21.15.72:0/3963796262 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf800760e0 0x7fdf801a05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:39.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:38.005+0000 7fdf85a00700 1 -- 172.21.15.72:0/3963796262 >> 172.21.15.72:0/3963796262 conn(0x7fdf800feea0 msgr2=0x7fdf801006a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:44:39.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:38.005+0000 7fdf85a00700 1 -- 172.21.15.72:0/3963796262 shutdown_connections 2024-04-26T23:44:39.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:38.005+0000 7fdf85a00700 1 -- 172.21.15.72:0/3963796262 wait complete. 2024-04-26T23:44:39.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:38 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/3963796262' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-26T23:44:39.661 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:39 smithi072 ceph-mon[28184]: pgmap v371: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:44:39.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:39 smithi076 ceph-mon[32774]: pgmap v371: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:44:40.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:39 smithi136 ceph-mon[33050]: pgmap v371: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:44:41.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:41 smithi072 ceph-mon[28184]: pgmap v372: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:44:41.913 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332147 got 219043332152 for osd.7 2024-04-26T23:44:41.913 DEBUG:teuthology.parallel:result is None 2024-04-26T23:44:41.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:41 smithi076 ceph-mon[32774]: pgmap v372: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:44:42.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:41 smithi136 ceph-mon[33050]: pgmap v372: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:44:43.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.414+0000 7fcc8a600700 1 -- 172.21.15.72:0/3229469751 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc84109a50 msgr2=0x7fcc84109ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:43.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.414+0000 7fcc8a600700 1 --2- 172.21.15.72:0/3229469751 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc84109a50 0x7fcc84109ed0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fcc700098f0 tx=0x7fcc70009c00 comp rx=0 tx=0).stop 2024-04-26T23:44:43.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.414+0000 7fcc8a600700 1 -- 172.21.15.72:0/3229469751 shutdown_connections 2024-04-26T23:44:43.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.414+0000 7fcc8a600700 1 --2- 172.21.15.72:0/3229469751 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc8410a410 0x7fcc84116920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:43.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.414+0000 7fcc8a600700 1 --2- 172.21.15.72:0/3229469751 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc8410fa50 0x7fcc8410fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:43.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.414+0000 7fcc8a600700 1 --2- 172.21.15.72:0/3229469751 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc84109a50 0x7fcc84109ed0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:43.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.414+0000 7fcc8a600700 1 -- 172.21.15.72:0/3229469751 >> 172.21.15.72:0/3229469751 conn(0x7fcc84076bf0 msgr2=0x7fcc84108240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:44:43.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.414+0000 7fcc8a600700 1 -- 172.21.15.72:0/3229469751 shutdown_connections 2024-04-26T23:44:43.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.414+0000 7fcc8a600700 1 -- 172.21.15.72:0/3229469751 wait complete. 2024-04-26T23:44:43.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.415+0000 7fcc8a600700 1 Processor -- start 2024-04-26T23:44:43.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.416+0000 7fcc8a600700 1 -- start start 2024-04-26T23:44:43.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.416+0000 7fcc8a600700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc84109a50 0x7fcc841a0220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:43.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.416+0000 7fcc8a600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc8410a410 0x7fcc841a0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:43.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.416+0000 7fcc8a600700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc8410fa50 0x7fcc841a4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:43.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.416+0000 7fcc8a600700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fcc84119ce0 con 0x7fcc84109a50 2024-04-26T23:44:43.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.416+0000 7fcc8a600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fcc84119b60 con 0x7fcc8410a410 2024-04-26T23:44:43.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.416+0000 7fcc8a600700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fcc84119e60 con 0x7fcc8410fa50 2024-04-26T23:44:43.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.416+0000 7fcc83400700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc84109a50 0x7fcc841a0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:43.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.416+0000 7fcc83400700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc84109a50 0x7fcc841a0220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37550/0 (socket says 172.21.15.72:37550) 2024-04-26T23:44:43.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.416+0000 7fcc83400700 1 -- 172.21.15.72:0/2555155078 learned_addr learned my addr 172.21.15.72:0/2555155078 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:44:43.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.416+0000 7fcc82a00700 1 --2- 172.21.15.72:0/2555155078 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc8410a410 0x7fcc841a0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:43.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.417+0000 7fcc83e00700 1 --2- 172.21.15.72:0/2555155078 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc8410fa50 0x7fcc841a4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:43.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.417+0000 7fcc82a00700 1 -- 172.21.15.72:0/2555155078 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc8410fa50 msgr2=0x7fcc841a4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:43.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.417+0000 7fcc82a00700 1 --2- 172.21.15.72:0/2555155078 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc8410fa50 0x7fcc841a4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:43.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.417+0000 7fcc82a00700 1 -- 172.21.15.72:0/2555155078 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc84109a50 msgr2=0x7fcc841a0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:43.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.417+0000 7fcc82a00700 1 --2- 172.21.15.72:0/2555155078 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc84109a50 0x7fcc841a0220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:43.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.417+0000 7fcc82a00700 1 -- 172.21.15.72:0/2555155078 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc841a5230 con 0x7fcc8410a410 2024-04-26T23:44:43.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.417+0000 7fcc83400700 1 --2- 172.21.15.72:0/2555155078 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc84109a50 0x7fcc841a0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:44:43.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.417+0000 7fcc82a00700 1 --2- 172.21.15.72:0/2555155078 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc8410a410 0x7fcc841a0760 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fcc7000c010 tx=0x7fcc7000fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:43.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.418+0000 7fcc7be00700 1 -- 172.21.15.72:0/2555155078 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc700050c0 con 0x7fcc8410a410 2024-04-26T23:44:43.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.418+0000 7fcc7be00700 1 -- 172.21.15.72:0/2555155078 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc7000b560 con 0x7fcc8410a410 2024-04-26T23:44:43.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.418+0000 7fcc7be00700 1 -- 172.21.15.72:0/2555155078 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc70003a00 con 0x7fcc8410a410 2024-04-26T23:44:43.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.418+0000 7fcc8a600700 1 -- 172.21.15.72:0/2555155078 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc841a54c0 con 0x7fcc8410a410 2024-04-26T23:44:43.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.418+0000 7fcc8a600700 1 -- 172.21.15.72:0/2555155078 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc841acda0 con 0x7fcc8410a410 2024-04-26T23:44:43.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.419+0000 7fcc8a600700 1 -- 172.21.15.72:0/2555155078 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc8410b190 con 0x7fcc8410a410 2024-04-26T23:44:43.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.421+0000 7fcc7be00700 1 -- 172.21.15.72:0/2555155078 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fcc7000b710 con 0x7fcc8410a410 2024-04-26T23:44:43.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.422+0000 7fcc7be00700 1 --2- 172.21.15.72:0/2555155078 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fcc680a3cd0 0x7fcc680a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:43.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.423+0000 7fcc7be00700 1 -- 172.21.15.72:0/2555155078 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fcc700d19b0 con 0x7fcc8410a410 2024-04-26T23:44:43.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.423+0000 7fcc83400700 1 --2- 172.21.15.72:0/2555155078 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fcc680a3cd0 0x7fcc680a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:43.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.423+0000 7fcc83400700 1 --2- 172.21.15.72:0/2555155078 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fcc680a3cd0 0x7fcc680a6190 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fcc74011550 tx=0x7fcc74009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:43.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.425+0000 7fcc7be00700 1 -- 172.21.15.72:0/2555155078 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fcc70013030 con 0x7fcc8410a410 2024-04-26T23:44:43.654 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:43 smithi072 ceph-mon[28184]: pgmap v373: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:44:43.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.669+0000 7fcc8a600700 1 -- 172.21.15.72:0/2555155078 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7fcc8402e490 con 0x7fcc8410a410 2024-04-26T23:44:43.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.670+0000 7fcc7be00700 1 -- 172.21.15.72:0/2555155078 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fcc7009df00 con 0x7fcc8410a410 2024-04-26T23:44:43.674 INFO:teuthology.orchestra.run.smithi072.stdout:287762808877 2024-04-26T23:44:43.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.674+0000 7fcc8a600700 1 -- 172.21.15.72:0/2555155078 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fcc680a3cd0 msgr2=0x7fcc680a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:43.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.674+0000 7fcc8a600700 1 --2- 172.21.15.72:0/2555155078 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fcc680a3cd0 0x7fcc680a6190 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fcc74011550 tx=0x7fcc74009260 comp rx=0 tx=0).stop 2024-04-26T23:44:43.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.674+0000 7fcc8a600700 1 -- 172.21.15.72:0/2555155078 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc8410a410 msgr2=0x7fcc841a0760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:43.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.674+0000 7fcc8a600700 1 --2- 172.21.15.72:0/2555155078 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc8410a410 0x7fcc841a0760 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fcc7000c010 tx=0x7fcc7000fa00 comp rx=0 tx=0).stop 2024-04-26T23:44:43.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.674+0000 7fcc8a600700 1 -- 172.21.15.72:0/2555155078 shutdown_connections 2024-04-26T23:44:43.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.674+0000 7fcc8a600700 1 --2- 172.21.15.72:0/2555155078 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fcc680a3cd0 0x7fcc680a6190 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:43.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.674+0000 7fcc8a600700 1 --2- 172.21.15.72:0/2555155078 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc8410fa50 0x7fcc841a4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:43.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.674+0000 7fcc8a600700 1 --2- 172.21.15.72:0/2555155078 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc84109a50 0x7fcc841a0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:43.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.674+0000 7fcc8a600700 1 --2- 172.21.15.72:0/2555155078 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc8410a410 0x7fcc841a0760 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:43.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.674+0000 7fcc8a600700 1 -- 172.21.15.72:0/2555155078 >> 172.21.15.72:0/2555155078 conn(0x7fcc84076bf0 msgr2=0x7fcc840772b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:44:43.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.674+0000 7fcc8a600700 1 -- 172.21.15.72:0/2555155078 shutdown_connections 2024-04-26T23:44:43.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:43.674+0000 7fcc8a600700 1 -- 172.21.15.72:0/2555155078 wait complete. 2024-04-26T23:44:43.852 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:43 smithi136 ceph-mon[33050]: pgmap v373: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:44:43.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:43 smithi076 ceph-mon[32774]: pgmap v373: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:44:44.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.404+0000 7f5a62800700 1 -- 172.21.15.72:0/818625083 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5a5c10fe20 msgr2=0x7f5a5c112210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:44.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.404+0000 7f5a62800700 1 --2- 172.21.15.72:0/818625083 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5a5c10fe20 0x7f5a5c112210 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5a5400b0a0 tx=0x7f5a5400b3b0 comp rx=0 tx=0).stop 2024-04-26T23:44:44.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.404+0000 7f5a62800700 1 -- 172.21.15.72:0/818625083 shutdown_connections 2024-04-26T23:44:44.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.404+0000 7f5a62800700 1 --2- 172.21.15.72:0/818625083 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5a5c10fe20 0x7f5a5c112210 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:44.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.404+0000 7f5a62800700 1 --2- 172.21.15.72:0/818625083 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5a5c075650 0x7f5a5c075a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:44.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.404+0000 7f5a62800700 1 --2- 172.21.15.72:0/818625083 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a5c075f70 0x7f5a5c10f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:44.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.404+0000 7f5a62800700 1 -- 172.21.15.72:0/818625083 >> 172.21.15.72:0/818625083 conn(0x7f5a5c07a580 msgr2=0x7f5a5c07a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:44:44.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.404+0000 7f5a62800700 1 -- 172.21.15.72:0/818625083 shutdown_connections 2024-04-26T23:44:44.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.405+0000 7f5a62800700 1 -- 172.21.15.72:0/818625083 wait complete. 2024-04-26T23:44:44.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.406+0000 7f5a62800700 1 Processor -- start 2024-04-26T23:44:44.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.406+0000 7f5a62800700 1 -- start start 2024-04-26T23:44:44.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.406+0000 7f5a62800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5a5c075650 0x7f5a5c1a1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:44.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.406+0000 7f5a62800700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a5c075f70 0x7f5a5c1a1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:44.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.406+0000 7f5a62800700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5a5c10fe20 0x7f5a5c19ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:44.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.406+0000 7f5a62800700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f5a5c1157d0 con 0x7f5a5c10fe20 2024-04-26T23:44:44.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.406+0000 7f5a62800700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f5a5c115650 con 0x7f5a5c075650 2024-04-26T23:44:44.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.406+0000 7f5a62800700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f5a5c115950 con 0x7f5a5c075f70 2024-04-26T23:44:44.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.406+0000 7f5a5b400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5a5c075650 0x7f5a5c1a1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:44.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.407+0000 7f5a5b400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5a5c075650 0x7f5a5c1a1860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:47508/0 (socket says 172.21.15.72:47508) 2024-04-26T23:44:44.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.407+0000 7f5a53e00700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a5c075f70 0x7f5a5c1a1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:44.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.407+0000 7f5a5b400700 1 -- 172.21.15.72:0/3906539501 learned_addr learned my addr 172.21.15.72:0/3906539501 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:44:44.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.407+0000 7f5a5be00700 1 --2- 172.21.15.72:0/3906539501 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5a5c10fe20 0x7f5a5c19ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:44.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.407+0000 7f5a53e00700 1 -- 172.21.15.72:0/3906539501 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5a5c075650 msgr2=0x7f5a5c1a1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:44.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.407+0000 7f5a53e00700 1 --2- 172.21.15.72:0/3906539501 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5a5c075650 0x7f5a5c1a1860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:44.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.407+0000 7f5a53e00700 1 -- 172.21.15.72:0/3906539501 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5a5c10fe20 msgr2=0x7f5a5c19ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:44.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.407+0000 7f5a53e00700 1 --2- 172.21.15.72:0/3906539501 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5a5c10fe20 0x7f5a5c19ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:44.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.407+0000 7f5a53e00700 1 -- 172.21.15.72:0/3906539501 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a5c19c1d0 con 0x7f5a5c075f70 2024-04-26T23:44:44.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.407+0000 7f5a5b400700 1 --2- 172.21.15.72:0/3906539501 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5a5c075650 0x7f5a5c1a1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:44:44.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.407+0000 7f5a53e00700 1 --2- 172.21.15.72:0/3906539501 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a5c075f70 0x7f5a5c1a1da0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5a4400e9c0 tx=0x7f5a4400ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:44.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.408+0000 7f5a58c00700 1 -- 172.21.15.72:0/3906539501 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a44004230 con 0x7f5a5c075f70 2024-04-26T23:44:44.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.408+0000 7f5a62800700 1 -- 172.21.15.72:0/3906539501 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a5c19c4c0 con 0x7f5a5c075f70 2024-04-26T23:44:44.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.408+0000 7f5a62800700 1 -- 172.21.15.72:0/3906539501 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a5c1a8730 con 0x7f5a5c075f70 2024-04-26T23:44:44.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.408+0000 7f5a58c00700 1 -- 172.21.15.72:0/3906539501 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a440043d0 con 0x7f5a5c075f70 2024-04-26T23:44:44.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.408+0000 7f5a58c00700 1 -- 172.21.15.72:0/3906539501 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a44003da0 con 0x7f5a5c075f70 2024-04-26T23:44:44.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.409+0000 7f5a62800700 1 -- 172.21.15.72:0/3906539501 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a5c06a110 con 0x7f5a5c075f70 2024-04-26T23:44:44.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.411+0000 7f5a58c00700 1 -- 172.21.15.72:0/3906539501 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5a4400c7c0 con 0x7f5a5c075f70 2024-04-26T23:44:44.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.412+0000 7f5a58c00700 1 --2- 172.21.15.72:0/3906539501 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f5a400a3d50 0x7f5a400a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:44.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.413+0000 7f5a58c00700 1 -- 172.21.15.72:0/3906539501 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f5a440d0bc0 con 0x7f5a5c075f70 2024-04-26T23:44:44.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.413+0000 7f5a5b400700 1 --2- 172.21.15.72:0/3906539501 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f5a400a3d50 0x7f5a400a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:44.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.413+0000 7f5a5b400700 1 --2- 172.21.15.72:0/3906539501 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f5a400a3d50 0x7f5a400a6210 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c103410 tx=0x7f5a4c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:44.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.415+0000 7f5a58c00700 1 -- 172.21.15.72:0/3906539501 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5a4409d080 con 0x7f5a5c075f70 2024-04-26T23:44:44.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.657+0000 7f5a62800700 1 -- 172.21.15.72:0/3906539501 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f5a5c19cf50 con 0x7f5a5c075f70 2024-04-26T23:44:44.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.657+0000 7f5a58c00700 1 -- 172.21.15.72:0/3906539501 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f5a4400cd70 con 0x7f5a5c075f70 2024-04-26T23:44:44.661 INFO:teuthology.orchestra.run.smithi072.stdout:309237645352 2024-04-26T23:44:44.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.660+0000 7f5a62800700 1 -- 172.21.15.72:0/3906539501 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f5a400a3d50 msgr2=0x7f5a400a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:44.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.660+0000 7f5a62800700 1 --2- 172.21.15.72:0/3906539501 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f5a400a3d50 0x7f5a400a6210 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c103410 tx=0x7f5a4c009260 comp rx=0 tx=0).stop 2024-04-26T23:44:44.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.660+0000 7f5a62800700 1 -- 172.21.15.72:0/3906539501 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a5c075f70 msgr2=0x7f5a5c1a1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:44.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.660+0000 7f5a62800700 1 --2- 172.21.15.72:0/3906539501 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a5c075f70 0x7f5a5c1a1da0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5a4400e9c0 tx=0x7f5a4400ecd0 comp rx=0 tx=0).stop 2024-04-26T23:44:44.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.660+0000 7f5a62800700 1 -- 172.21.15.72:0/3906539501 shutdown_connections 2024-04-26T23:44:44.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.660+0000 7f5a62800700 1 --2- 172.21.15.72:0/3906539501 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f5a400a3d50 0x7f5a400a6210 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:44.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.660+0000 7f5a62800700 1 --2- 172.21.15.72:0/3906539501 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5a5c10fe20 0x7f5a5c19ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:44.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.660+0000 7f5a62800700 1 --2- 172.21.15.72:0/3906539501 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5a5c075650 0x7f5a5c1a1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:44.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.660+0000 7f5a62800700 1 --2- 172.21.15.72:0/3906539501 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a5c075f70 0x7f5a5c1a1da0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:44.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.660+0000 7f5a62800700 1 -- 172.21.15.72:0/3906539501 >> 172.21.15.72:0/3906539501 conn(0x7f5a5c07a580 msgr2=0x7f5a5c1005d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:44:44.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.660+0000 7f5a62800700 1 -- 172.21.15.72:0/3906539501 shutdown_connections 2024-04-26T23:44:44.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:44.661+0000 7f5a62800700 1 -- 172.21.15.72:0/3906539501 wait complete. 2024-04-26T23:44:44.832 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:44 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/2555155078' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-26T23:44:44.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:44 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/2555155078' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-26T23:44:45.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:44 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/2555155078' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-26T23:44:45.831 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:45 smithi072 ceph-mon[28184]: pgmap v374: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:44:45.831 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:45 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/3906539501' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-26T23:44:45.891 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645346 got 309237645352 for osd.10 2024-04-26T23:44:45.892 DEBUG:teuthology.parallel:result is None 2024-04-26T23:44:45.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:45 smithi076 ceph-mon[32774]: pgmap v374: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:44:45.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:45 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/3906539501' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-26T23:44:46.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:45 smithi136 ceph-mon[33050]: pgmap v374: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:44:46.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:45 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/3906539501' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-26T23:44:46.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.141+0000 7f1e21e00700 1 -- 172.21.15.72:0/3044071608 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1e1c107590 msgr2=0x7f1e1c10f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:46.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.141+0000 7f1e21e00700 1 --2- 172.21.15.72:0/3044071608 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1e1c107590 0x7f1e1c10f160 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1e14009f90 tx=0x7f1e1400b330 comp rx=0 tx=0).stop 2024-04-26T23:44:46.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.141+0000 7f1e21e00700 1 -- 172.21.15.72:0/3044071608 shutdown_connections 2024-04-26T23:44:46.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.141+0000 7f1e21e00700 1 --2- 172.21.15.72:0/3044071608 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1e1c107590 0x7f1e1c10f160 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:46.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.141+0000 7f1e21e00700 1 --2- 172.21.15.72:0/3044071608 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1e1c1024b0 0x7f1e1c102890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:46.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.141+0000 7f1e21e00700 1 --2- 172.21.15.72:0/3044071608 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1e1c102e60 0x7f1e1c106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:46.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.141+0000 7f1e21e00700 1 -- 172.21.15.72:0/3044071608 >> 172.21.15.72:0/3044071608 conn(0x7f1e1c0fdd40 msgr2=0x7f1e1c100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:44:46.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.141+0000 7f1e21e00700 1 -- 172.21.15.72:0/3044071608 shutdown_connections 2024-04-26T23:44:46.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.141+0000 7f1e21e00700 1 -- 172.21.15.72:0/3044071608 wait complete. 2024-04-26T23:44:46.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.143+0000 7f1e21e00700 1 Processor -- start 2024-04-26T23:44:46.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.143+0000 7f1e21e00700 1 -- start start 2024-04-26T23:44:46.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.143+0000 7f1e21e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1e1c1024b0 0x7f1e1c1a18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:46.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.143+0000 7f1e21e00700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1e1c102e60 0x7f1e1c1a1de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:46.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.143+0000 7f1e21e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1e1c107590 0x7f1e1c19bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:46.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.143+0000 7f1e21e00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f1e1c112480 con 0x7f1e1c107590 2024-04-26T23:44:46.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.143+0000 7f1e21e00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f1e1c112300 con 0x7f1e1c1024b0 2024-04-26T23:44:46.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.143+0000 7f1e21e00700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f1e1c112600 con 0x7f1e1c102e60 2024-04-26T23:44:46.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.144+0000 7f1e20a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1e1c1024b0 0x7f1e1c1a18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:46.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.144+0000 7f1e20a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1e1c1024b0 0x7f1e1c1a18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:47530/0 (socket says 172.21.15.72:47530) 2024-04-26T23:44:46.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.144+0000 7f1e20a00700 1 -- 172.21.15.72:0/3894489640 learned_addr learned my addr 172.21.15.72:0/3894489640 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:44:46.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.144+0000 7f1e21400700 1 --2- 172.21.15.72:0/3894489640 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1e1c107590 0x7f1e1c19bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:46.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.144+0000 7f1e1be00700 1 --2- 172.21.15.72:0/3894489640 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1e1c102e60 0x7f1e1c1a1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:46.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.144+0000 7f1e20a00700 1 -- 172.21.15.72:0/3894489640 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1e1c102e60 msgr2=0x7f1e1c1a1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:46.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.144+0000 7f1e20a00700 1 --2- 172.21.15.72:0/3894489640 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1e1c102e60 0x7f1e1c1a1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:46.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.144+0000 7f1e20a00700 1 -- 172.21.15.72:0/3894489640 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1e1c107590 msgr2=0x7f1e1c19bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:46.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.144+0000 7f1e20a00700 1 --2- 172.21.15.72:0/3894489640 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1e1c107590 0x7f1e1c19bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:46.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.144+0000 7f1e20a00700 1 -- 172.21.15.72:0/3894489640 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e1c19c210 con 0x7f1e1c1024b0 2024-04-26T23:44:46.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.144+0000 7f1e20a00700 1 --2- 172.21.15.72:0/3894489640 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1e1c1024b0 0x7f1e1c1a18a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1e1000ec10 tx=0x7f1e1000ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:46.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.145+0000 7f1e19600700 1 -- 172.21.15.72:0/3894489640 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e1000cc90 con 0x7f1e1c1024b0 2024-04-26T23:44:46.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.145+0000 7f1e19600700 1 -- 172.21.15.72:0/3894489640 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e1000ce30 con 0x7f1e1c1024b0 2024-04-26T23:44:46.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.145+0000 7f1e19600700 1 -- 172.21.15.72:0/3894489640 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e10010430 con 0x7f1e1c1024b0 2024-04-26T23:44:46.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.145+0000 7f1e21e00700 1 -- 172.21.15.72:0/3894489640 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e1c19c500 con 0x7f1e1c1024b0 2024-04-26T23:44:46.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.145+0000 7f1e21e00700 1 -- 172.21.15.72:0/3894489640 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e1c1a8770 con 0x7f1e1c1024b0 2024-04-26T23:44:46.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.146+0000 7f1e21e00700 1 -- 172.21.15.72:0/3894489640 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e1c06a110 con 0x7f1e1c1024b0 2024-04-26T23:44:46.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.148+0000 7f1e19600700 1 -- 172.21.15.72:0/3894489640 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1e10004000 con 0x7f1e1c1024b0 2024-04-26T23:44:46.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.149+0000 7f1e19600700 1 --2- 172.21.15.72:0/3894489640 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f1e040a3d50 0x7f1e040a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:46.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.150+0000 7f1e19600700 1 -- 172.21.15.72:0/3894489640 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f1e100d0b50 con 0x7f1e1c1024b0 2024-04-26T23:44:46.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.150+0000 7f1e1be00700 1 --2- 172.21.15.72:0/3894489640 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f1e040a3d50 0x7f1e040a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:46.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.150+0000 7f1e1be00700 1 --2- 172.21.15.72:0/3894489640 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f1e040a3d50 0x7f1e040a6210 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1e08009680 tx=0x7f1e08009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:46.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.153+0000 7f1e19600700 1 -- 172.21.15.72:0/3894489640 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1e1009d010 con 0x7f1e1c1024b0 2024-04-26T23:44:46.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.389+0000 7f1e21e00700 1 -- 172.21.15.72:0/3894489640 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f1e1c19cf90 con 0x7f1e1c1024b0 2024-04-26T23:44:46.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.390+0000 7f1e19600700 1 -- 172.21.15.72:0/3894489640 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f1e1009c910 con 0x7f1e1c1024b0 2024-04-26T23:44:46.393 INFO:teuthology.orchestra.run.smithi072.stdout:343597383715 2024-04-26T23:44:46.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.392+0000 7f1e21e00700 1 -- 172.21.15.72:0/3894489640 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f1e040a3d50 msgr2=0x7f1e040a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:46.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.392+0000 7f1e21e00700 1 --2- 172.21.15.72:0/3894489640 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f1e040a3d50 0x7f1e040a6210 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1e08009680 tx=0x7f1e08009260 comp rx=0 tx=0).stop 2024-04-26T23:44:46.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.392+0000 7f1e21e00700 1 -- 172.21.15.72:0/3894489640 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1e1c1024b0 msgr2=0x7f1e1c1a18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:46.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.392+0000 7f1e21e00700 1 --2- 172.21.15.72:0/3894489640 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1e1c1024b0 0x7f1e1c1a18a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1e1000ec10 tx=0x7f1e1000ef20 comp rx=0 tx=0).stop 2024-04-26T23:44:46.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.392+0000 7f1e21e00700 1 -- 172.21.15.72:0/3894489640 shutdown_connections 2024-04-26T23:44:46.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.392+0000 7f1e21e00700 1 --2- 172.21.15.72:0/3894489640 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f1e040a3d50 0x7f1e040a6210 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:46.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.392+0000 7f1e21e00700 1 --2- 172.21.15.72:0/3894489640 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1e1c107590 0x7f1e1c19bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:46.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.392+0000 7f1e21e00700 1 --2- 172.21.15.72:0/3894489640 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1e1c1024b0 0x7f1e1c1a18a0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:46.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.392+0000 7f1e21e00700 1 --2- 172.21.15.72:0/3894489640 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1e1c102e60 0x7f1e1c1a1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:46.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.393+0000 7f1e21e00700 1 -- 172.21.15.72:0/3894489640 >> 172.21.15.72:0/3894489640 conn(0x7f1e1c0fdd40 msgr2=0x7f1e1c06b240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:44:46.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.393+0000 7f1e21e00700 1 -- 172.21.15.72:0/3894489640 shutdown_connections 2024-04-26T23:44:46.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:46.393+0000 7f1e21e00700 1 -- 172.21.15.72:0/3894489640 wait complete. 2024-04-26T23:44:47.163 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808870 got 287762808877 for osd.9 2024-04-26T23:44:47.163 DEBUG:teuthology.parallel:result is None 2024-04-26T23:44:47.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:47 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:44:47.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:47 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:44:47.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:47 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/3894489640' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-26T23:44:47.372 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:47 smithi072 ceph-mon[28184]: pgmap v375: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:44:47.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:47 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:44:47.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:47 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:44:47.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:47 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/3894489640' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-26T23:44:47.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:47 smithi076 ceph-mon[32774]: pgmap v375: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:44:47.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.533+0000 7f4604200700 1 -- 172.21.15.72:0/1420395936 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f45fc10fa50 msgr2=0x7f45fc10fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:47.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.533+0000 7f4604200700 1 --2- 172.21.15.72:0/1420395936 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f45fc10fa50 0x7f45fc10fe30 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f45f0009990 tx=0x7f45f0009ca0 comp rx=0 tx=0).stop 2024-04-26T23:44:47.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.533+0000 7f4604200700 1 -- 172.21.15.72:0/1420395936 shutdown_connections 2024-04-26T23:44:47.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.533+0000 7f4604200700 1 --2- 172.21.15.72:0/1420395936 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45fc10a410 0x7f45fc116920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:47.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.533+0000 7f4604200700 1 --2- 172.21.15.72:0/1420395936 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f45fc10fa50 0x7f45fc10fe30 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:47.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.533+0000 7f4604200700 1 --2- 172.21.15.72:0/1420395936 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f45fc109a50 0x7f45fc109ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:47.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.533+0000 7f4604200700 1 -- 172.21.15.72:0/1420395936 >> 172.21.15.72:0/1420395936 conn(0x7f45fc076bf0 msgr2=0x7f45fc108240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:44:47.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.533+0000 7f4604200700 1 -- 172.21.15.72:0/1420395936 shutdown_connections 2024-04-26T23:44:47.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.533+0000 7f4604200700 1 -- 172.21.15.72:0/1420395936 wait complete. 2024-04-26T23:44:47.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.534+0000 7f4604200700 1 Processor -- start 2024-04-26T23:44:47.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.534+0000 7f4604200700 1 -- start start 2024-04-26T23:44:47.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.534+0000 7f4604200700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f45fc109a50 0x7f45fc1a0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:47.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.535+0000 7f4604200700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45fc10a410 0x7f45fc1a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:47.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.535+0000 7f4604200700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f45fc10fa50 0x7f45fc1a4a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:47.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.535+0000 7f4604200700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f45fc119c90 con 0x7f45fc10fa50 2024-04-26T23:44:47.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.535+0000 7f4604200700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f45fc119b10 con 0x7f45fc109a50 2024-04-26T23:44:47.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.535+0000 7f4604200700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f45fc119e10 con 0x7f45fc10a410 2024-04-26T23:44:47.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.535+0000 7f4602200700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f45fc10fa50 0x7f45fc1a4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:47.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.535+0000 7f4600e00700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45fc10a410 0x7f45fc1a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:47.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.535+0000 7f4602200700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f45fc10fa50 0x7f45fc1a4a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57808/0 (socket says 172.21.15.72:57808) 2024-04-26T23:44:47.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.535+0000 7f4602200700 1 -- 172.21.15.72:0/2287817190 learned_addr learned my addr 172.21.15.72:0/2287817190 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:44:47.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.535+0000 7f4601800700 1 --2- 172.21.15.72:0/2287817190 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f45fc109a50 0x7f45fc1a0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:47.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.535+0000 7f4602200700 1 -- 172.21.15.72:0/2287817190 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45fc10a410 msgr2=0x7f45fc1a06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:47.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.535+0000 7f4602200700 1 --2- 172.21.15.72:0/2287817190 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45fc10a410 0x7f45fc1a06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:47.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.535+0000 7f4602200700 1 -- 172.21.15.72:0/2287817190 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f45fc109a50 msgr2=0x7f45fc1a0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:47.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.535+0000 7f4602200700 1 --2- 172.21.15.72:0/2287817190 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f45fc109a50 0x7f45fc1a0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:47.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.535+0000 7f4602200700 1 -- 172.21.15.72:0/2287817190 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45fc1a5140 con 0x7f45fc10fa50 2024-04-26T23:44:47.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.535+0000 7f4600e00700 1 --2- 172.21.15.72:0/2287817190 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45fc10a410 0x7f45fc1a06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:44:47.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.536+0000 7f4602200700 1 --2- 172.21.15.72:0/2287817190 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f45fc10fa50 0x7f45fc1a4a00 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f45f400ef30 tx=0x7f45f400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:47.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.536+0000 7f45fa000700 1 -- 172.21.15.72:0/2287817190 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45f40092e0 con 0x7f45fc10fa50 2024-04-26T23:44:47.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.536+0000 7f45fa000700 1 -- 172.21.15.72:0/2287817190 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45f4004bb0 con 0x7f45fc10fa50 2024-04-26T23:44:47.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.536+0000 7f45fa000700 1 -- 172.21.15.72:0/2287817190 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45f4004280 con 0x7f45fc10fa50 2024-04-26T23:44:47.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.536+0000 7f4604200700 1 -- 172.21.15.72:0/2287817190 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45fc1a53d0 con 0x7f45fc10fa50 2024-04-26T23:44:47.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.536+0000 7f4604200700 1 -- 172.21.15.72:0/2287817190 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45fc1accb0 con 0x7f45fc10fa50 2024-04-26T23:44:47.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.537+0000 7f4604200700 1 -- 172.21.15.72:0/2287817190 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45fc06a110 con 0x7f45fc10fa50 2024-04-26T23:44:47.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.539+0000 7f45fa000700 1 -- 172.21.15.72:0/2287817190 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f45f401b030 con 0x7f45fc10fa50 2024-04-26T23:44:47.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.540+0000 7f45fa000700 1 --2- 172.21.15.72:0/2287817190 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f45e40a3cd0 0x7f45e40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:47.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.541+0000 7f45fa000700 1 -- 172.21.15.72:0/2287817190 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f45f40d03f0 con 0x7f45fc10fa50 2024-04-26T23:44:47.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.541+0000 7f4601800700 1 --2- 172.21.15.72:0/2287817190 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f45e40a3cd0 0x7f45e40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:47.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.542+0000 7f4601800700 1 --2- 172.21.15.72:0/2287817190 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f45e40a3cd0 0x7f45e40a6190 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f45f0006010 tx=0x7f45f000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:47.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.546+0000 7f45fa000700 1 -- 172.21.15.72:0/2287817190 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f45f4022020 con 0x7f45fc10fa50 2024-04-26T23:44:47.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:47 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:44:47.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:47 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:44:47.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:47 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/3894489640' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-26T23:44:47.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:47 smithi136 ceph-mon[33050]: pgmap v375: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:44:47.781 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.777+0000 7f4604200700 1 -- 172.21.15.72:0/2287817190 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f45fc02e490 con 0x7f45fc10fa50 2024-04-26T23:44:47.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.778+0000 7f45fa000700 1 -- 172.21.15.72:0/2287817190 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f45f4010040 con 0x7f45fc10fa50 2024-04-26T23:44:47.782 INFO:teuthology.orchestra.run.smithi072.stdout:163208757318 2024-04-26T23:44:47.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.781+0000 7f4604200700 1 -- 172.21.15.72:0/2287817190 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f45e40a3cd0 msgr2=0x7f45e40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:47.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.781+0000 7f4604200700 1 --2- 172.21.15.72:0/2287817190 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f45e40a3cd0 0x7f45e40a6190 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f45f0006010 tx=0x7f45f000b260 comp rx=0 tx=0).stop 2024-04-26T23:44:47.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.781+0000 7f4604200700 1 -- 172.21.15.72:0/2287817190 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f45fc10fa50 msgr2=0x7f45fc1a4a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:47.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.781+0000 7f4604200700 1 --2- 172.21.15.72:0/2287817190 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f45fc10fa50 0x7f45fc1a4a00 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f45f400ef30 tx=0x7f45f400c3a0 comp rx=0 tx=0).stop 2024-04-26T23:44:47.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.781+0000 7f4604200700 1 -- 172.21.15.72:0/2287817190 shutdown_connections 2024-04-26T23:44:47.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.781+0000 7f4604200700 1 --2- 172.21.15.72:0/2287817190 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f45e40a3cd0 0x7f45e40a6190 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:47.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.781+0000 7f4604200700 1 --2- 172.21.15.72:0/2287817190 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f45fc10fa50 0x7f45fc1a4a00 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:47.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.781+0000 7f4604200700 1 --2- 172.21.15.72:0/2287817190 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f45fc109a50 0x7f45fc1a0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:47.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.781+0000 7f4604200700 1 --2- 172.21.15.72:0/2287817190 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45fc10a410 0x7f45fc1a06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:47.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.781+0000 7f4604200700 1 -- 172.21.15.72:0/2287817190 >> 172.21.15.72:0/2287817190 conn(0x7f45fc076bf0 msgr2=0x7f45fc077250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:44:47.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.782+0000 7f4604200700 1 -- 172.21.15.72:0/2287817190 shutdown_connections 2024-04-26T23:44:47.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:47.782+0000 7f4604200700 1 -- 172.21.15.72:0/2287817190 wait complete. 2024-04-26T23:44:47.877 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383707 got 343597383715 for osd.11 2024-04-26T23:44:47.877 DEBUG:teuthology.parallel:result is None 2024-04-26T23:44:48.381 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:48 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/2287817190' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-26T23:44:48.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:48 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/2287817190' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-26T23:44:48.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.448+0000 7ff482800700 1 -- 172.21.15.72:0/1124353423 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff47c102fc0 msgr2=0x7ff47c106f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:48.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.448+0000 7ff482800700 1 --2- 172.21.15.72:0/1124353423 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff47c102fc0 0x7ff47c106f80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff4680098f0 tx=0x7ff468009c00 comp rx=0 tx=0).stop 2024-04-26T23:44:48.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.448+0000 7ff482800700 1 -- 172.21.15.72:0/1124353423 shutdown_connections 2024-04-26T23:44:48.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.448+0000 7ff482800700 1 --2- 172.21.15.72:0/1124353423 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff47c107660 0x7ff47c10f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:48.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.448+0000 7ff482800700 1 --2- 172.21.15.72:0/1124353423 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff47c102610 0x7ff47c1029f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:48.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.448+0000 7ff482800700 1 --2- 172.21.15.72:0/1124353423 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff47c102fc0 0x7ff47c106f80 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:48.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.448+0000 7ff482800700 1 -- 172.21.15.72:0/1124353423 >> 172.21.15.72:0/1124353423 conn(0x7ff47c078370 msgr2=0x7ff47c078780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:44:48.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.448+0000 7ff482800700 1 -- 172.21.15.72:0/1124353423 shutdown_connections 2024-04-26T23:44:48.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.448+0000 7ff482800700 1 -- 172.21.15.72:0/1124353423 wait complete. 2024-04-26T23:44:48.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.449+0000 7ff482800700 1 Processor -- start 2024-04-26T23:44:48.453 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.449+0000 7ff482800700 1 -- start start 2024-04-26T23:44:48.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.450+0000 7ff482800700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff47c102610 0x7ff47c19bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:48.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.450+0000 7ff482800700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff47c102fc0 0x7ff47c19c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:48.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.450+0000 7ff482800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff47c107660 0x7ff47c1a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:48.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.450+0000 7ff482800700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7ff47c1124f0 con 0x7ff47c102fc0 2024-04-26T23:44:48.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.450+0000 7ff482800700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff47c112370 con 0x7ff47c107660 2024-04-26T23:44:48.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.450+0000 7ff482800700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff47c112670 con 0x7ff47c102610 2024-04-26T23:44:48.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.450+0000 7ff47aa00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff47c102fc0 0x7ff47c19c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:48.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.450+0000 7ff47aa00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff47c102fc0 0x7ff47c19c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57816/0 (socket says 172.21.15.72:57816) 2024-04-26T23:44:48.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.450+0000 7ff47aa00700 1 -- 172.21.15.72:0/2265847728 learned_addr learned my addr 172.21.15.72:0/2265847728 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:44:48.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.450+0000 7ff47be00700 1 --2- 172.21.15.72:0/2265847728 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff47c107660 0x7ff47c1a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:48.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.450+0000 7ff47b400700 1 --2- 172.21.15.72:0/2265847728 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff47c102610 0x7ff47c19bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:48.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.451+0000 7ff47b400700 1 -- 172.21.15.72:0/2265847728 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff47c107660 msgr2=0x7ff47c1a0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:48.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.451+0000 7ff47b400700 1 --2- 172.21.15.72:0/2265847728 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff47c107660 0x7ff47c1a0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:48.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.451+0000 7ff47b400700 1 -- 172.21.15.72:0/2265847728 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff47c102fc0 msgr2=0x7ff47c19c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:48.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.451+0000 7ff47b400700 1 --2- 172.21.15.72:0/2265847728 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff47c102fc0 0x7ff47c19c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:48.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.451+0000 7ff47b400700 1 -- 172.21.15.72:0/2265847728 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff47c1a0d90 con 0x7ff47c102610 2024-04-26T23:44:48.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.451+0000 7ff47b400700 1 --2- 172.21.15.72:0/2265847728 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff47c102610 0x7ff47c19bd80 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff46c00e9f0 tx=0x7ff46c00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:48.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.452+0000 7ff473e00700 1 -- 172.21.15.72:0/2265847728 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff46c004220 con 0x7ff47c102610 2024-04-26T23:44:48.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.452+0000 7ff473e00700 1 -- 172.21.15.72:0/2265847728 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff46c0043c0 con 0x7ff47c102610 2024-04-26T23:44:48.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.452+0000 7ff473e00700 1 -- 172.21.15.72:0/2265847728 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff46c005020 con 0x7ff47c102610 2024-04-26T23:44:48.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.452+0000 7ff482800700 1 -- 172.21.15.72:0/2265847728 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff47c1a1080 con 0x7ff47c102610 2024-04-26T23:44:48.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.452+0000 7ff482800700 1 -- 172.21.15.72:0/2265847728 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff47c1a8900 con 0x7ff47c102610 2024-04-26T23:44:48.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.453+0000 7ff482800700 1 -- 172.21.15.72:0/2265847728 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff47c06a110 con 0x7ff47c102610 2024-04-26T23:44:48.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.455+0000 7ff473e00700 1 -- 172.21.15.72:0/2265847728 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff46c004750 con 0x7ff47c102610 2024-04-26T23:44:48.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.456+0000 7ff473e00700 1 --2- 172.21.15.72:0/2265847728 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff4600a3d50 0x7ff4600a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:48.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.457+0000 7ff473e00700 1 -- 172.21.15.72:0/2265847728 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff46c0d09f0 con 0x7ff47c102610 2024-04-26T23:44:48.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.457+0000 7ff47aa00700 1 --2- 172.21.15.72:0/2265847728 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff4600a3d50 0x7ff4600a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:48.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.458+0000 7ff47aa00700 1 --2- 172.21.15.72:0/2265847728 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff4600a3d50 0x7ff4600a6210 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff468009680 tx=0x7ff468019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:48.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.459+0000 7ff473e00700 1 -- 172.21.15.72:0/2265847728 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff46c09ceb0 con 0x7ff47c102610 2024-04-26T23:44:48.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:48 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/2287817190' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-26T23:44:48.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.709+0000 7ff482800700 1 -- 172.21.15.72:0/2265847728 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7ff47c02e490 con 0x7ff47c102610 2024-04-26T23:44:48.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.709+0000 7ff473e00700 1 -- 172.21.15.72:0/2265847728 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7ff46c09c7b0 con 0x7ff47c102610 2024-04-26T23:44:48.713 INFO:teuthology.orchestra.run.smithi072.stdout:107374182479 2024-04-26T23:44:48.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.712+0000 7ff482800700 1 -- 172.21.15.72:0/2265847728 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff4600a3d50 msgr2=0x7ff4600a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:48.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.712+0000 7ff482800700 1 --2- 172.21.15.72:0/2265847728 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff4600a3d50 0x7ff4600a6210 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff468009680 tx=0x7ff468019040 comp rx=0 tx=0).stop 2024-04-26T23:44:48.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.712+0000 7ff482800700 1 -- 172.21.15.72:0/2265847728 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff47c102610 msgr2=0x7ff47c19bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:48.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.712+0000 7ff482800700 1 --2- 172.21.15.72:0/2265847728 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff47c102610 0x7ff47c19bd80 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff46c00e9f0 tx=0x7ff46c00ed00 comp rx=0 tx=0).stop 2024-04-26T23:44:48.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.712+0000 7ff482800700 1 -- 172.21.15.72:0/2265847728 shutdown_connections 2024-04-26T23:44:48.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.712+0000 7ff482800700 1 --2- 172.21.15.72:0/2265847728 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff4600a3d50 0x7ff4600a6210 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:48.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.712+0000 7ff482800700 1 --2- 172.21.15.72:0/2265847728 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff47c107660 0x7ff47c1a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:48.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.712+0000 7ff482800700 1 --2- 172.21.15.72:0/2265847728 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff47c102610 0x7ff47c19bd80 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:48.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.712+0000 7ff482800700 1 --2- 172.21.15.72:0/2265847728 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff47c102fc0 0x7ff47c19c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:48.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.712+0000 7ff482800700 1 -- 172.21.15.72:0/2265847728 >> 172.21.15.72:0/2265847728 conn(0x7ff47c078370 msgr2=0x7ff47c101760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:44:48.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.713+0000 7ff482800700 1 -- 172.21.15.72:0/2265847728 shutdown_connections 2024-04-26T23:44:48.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:48.713+0000 7ff482800700 1 -- 172.21.15.72:0/2265847728 wait complete. 2024-04-26T23:44:49.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:49 smithi076 ceph-mon[32774]: pgmap v376: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:44:49.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:49 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/2265847728' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-26T23:44:49.429 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:49 smithi072 ceph-mon[28184]: pgmap v376: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:44:49.430 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:49 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/2265847728' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-26T23:44:49.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:49 smithi136 ceph-mon[33050]: pgmap v376: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:44:49.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:49 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/2265847728' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-26T23:44:49.593 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:44:49.629 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411407 got 77309411412 for osd.2 2024-04-26T23:44:49.629 DEBUG:teuthology.parallel:result is None 2024-04-26T23:44:50.116 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757312 got 163208757318 for osd.5 2024-04-26T23:44:50.116 DEBUG:teuthology.parallel:result is None 2024-04-26T23:44:50.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:50 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:44:50.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:50 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:44:50.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:50 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:44:50.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:50 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:44:50.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:50 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:44:50.544 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:50 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:44:50.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:50 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:44:50.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:50 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:44:50.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:50 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:44:50.781 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182474 got 107374182479 for osd.3 2024-04-26T23:44:50.782 DEBUG:teuthology.parallel:result is None 2024-04-26T23:44:51.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:51 smithi072 ceph-mon[28184]: pgmap v377: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:44:51.429 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:51 smithi076 ceph-mon[32774]: pgmap v377: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:44:51.550 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:51 smithi136 ceph-mon[33050]: pgmap v377: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:44:52.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.060+0000 7f15ce800700 1 -- 172.21.15.72:0/2992449385 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f15c8102370 msgr2=0x7f15c8102750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:52.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.060+0000 7f15ce800700 1 --2- 172.21.15.72:0/2992449385 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f15c8102370 0x7f15c8102750 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f15bc009990 tx=0x7f15bc009ca0 comp rx=0 tx=0).stop 2024-04-26T23:44:52.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.060+0000 7f15ce800700 1 -- 172.21.15.72:0/2992449385 shutdown_connections 2024-04-26T23:44:52.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.060+0000 7f15ce800700 1 --2- 172.21.15.72:0/2992449385 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15c8107780 0x7f15c8107c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:52.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.060+0000 7f15ce800700 1 --2- 172.21.15.72:0/2992449385 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f15c8102370 0x7f15c8102750 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:52.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.060+0000 7f15ce800700 1 --2- 172.21.15.72:0/2992449385 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f15c8102c90 0x7f15c8107160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:52.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.060+0000 7f15ce800700 1 -- 172.21.15.72:0/2992449385 >> 172.21.15.72:0/2992449385 conn(0x7f15c80fddd0 msgr2=0x7f15c81001f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:44:52.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.061+0000 7f15ce800700 1 -- 172.21.15.72:0/2992449385 shutdown_connections 2024-04-26T23:44:52.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.061+0000 7f15ce800700 1 -- 172.21.15.72:0/2992449385 wait complete. 2024-04-26T23:44:52.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.062+0000 7f15ce800700 1 Processor -- start 2024-04-26T23:44:52.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.062+0000 7f15ce800700 1 -- start start 2024-04-26T23:44:52.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.063+0000 7f15ce800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f15c8102370 0x7f15c81a00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:52.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.063+0000 7f15ce800700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f15c8102c90 0x7f15c81a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:52.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.063+0000 7f15ce800700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15c8107780 0x7f15c81a4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:52.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.063+0000 7f15ce800700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f15c8119bc0 con 0x7f15c8102c90 2024-04-26T23:44:52.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.063+0000 7f15ce800700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f15c8119a40 con 0x7f15c8102370 2024-04-26T23:44:52.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.063+0000 7f15ce800700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f15c8119d40 con 0x7f15c8107780 2024-04-26T23:44:52.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.063+0000 7f15cd400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f15c8102370 0x7f15c81a00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:52.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.063+0000 7f15cde00700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15c8107780 0x7f15c81a4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:52.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.063+0000 7f15cca00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f15c8102c90 0x7f15c81a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:52.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.063+0000 7f15cd400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f15c8102370 0x7f15c81a00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:38202/0 (socket says 172.21.15.72:38202) 2024-04-26T23:44:52.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.063+0000 7f15cde00700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15c8107780 0x7f15c81a4970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.72:38604/0 (socket says 172.21.15.72:38604) 2024-04-26T23:44:52.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.063+0000 7f15cd400700 1 -- 172.21.15.72:0/893389224 learned_addr learned my addr 172.21.15.72:0/893389224 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:44:52.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.064+0000 7f15cd400700 1 -- 172.21.15.72:0/893389224 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15c8107780 msgr2=0x7f15c81a4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:52.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.064+0000 7f15cd400700 1 --2- 172.21.15.72:0/893389224 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15c8107780 0x7f15c81a4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:52.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.064+0000 7f15cd400700 1 -- 172.21.15.72:0/893389224 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f15c8102c90 msgr2=0x7f15c81a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:52.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.064+0000 7f15cd400700 1 --2- 172.21.15.72:0/893389224 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f15c8102c90 0x7f15c81a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:52.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.064+0000 7f15cd400700 1 -- 172.21.15.72:0/893389224 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15c81a50b0 con 0x7f15c8102370 2024-04-26T23:44:52.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.064+0000 7f15cde00700 1 --2- 172.21.15.72:0/893389224 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15c8107780 0x7f15c81a4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:44:52.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.064+0000 7f15cca00700 1 --2- 172.21.15.72:0/893389224 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f15c8102c90 0x7f15c81a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:44:52.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.064+0000 7f15cd400700 1 --2- 172.21.15.72:0/893389224 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f15c8102370 0x7f15c81a00f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f15bc0057f0 tx=0x7f15bc00bfd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:52.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.064+0000 7f15c6000700 1 -- 172.21.15.72:0/893389224 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15bc026070 con 0x7f15c8102370 2024-04-26T23:44:52.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.064+0000 7f15ce800700 1 -- 172.21.15.72:0/893389224 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15c81a5340 con 0x7f15c8102370 2024-04-26T23:44:52.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.065+0000 7f15c6000700 1 -- 172.21.15.72:0/893389224 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15bc021410 con 0x7f15c8102370 2024-04-26T23:44:52.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.065+0000 7f15ce800700 1 -- 172.21.15.72:0/893389224 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15c81acc20 con 0x7f15c8102370 2024-04-26T23:44:52.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.065+0000 7f15c6000700 1 -- 172.21.15.72:0/893389224 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15bc017650 con 0x7f15c8102370 2024-04-26T23:44:52.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.066+0000 7f15ce800700 1 -- 172.21.15.72:0/893389224 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15c806a110 con 0x7f15c8102370 2024-04-26T23:44:52.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.068+0000 7f15c6000700 1 -- 172.21.15.72:0/893389224 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f15bc029070 con 0x7f15c8102370 2024-04-26T23:44:52.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.069+0000 7f15c6000700 1 --2- 172.21.15.72:0/893389224 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f15b00a3d50 0x7f15b00a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:52.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.069+0000 7f15c6000700 1 -- 172.21.15.72:0/893389224 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f15bc0d17b0 con 0x7f15c8102370 2024-04-26T23:44:52.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.069+0000 7f15cca00700 1 --2- 172.21.15.72:0/893389224 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f15b00a3d50 0x7f15b00a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:52.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.070+0000 7f15cca00700 1 --2- 172.21.15.72:0/893389224 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f15b00a3d50 0x7f15b00a6210 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f15b4009680 tx=0x7f15b4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:52.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.071+0000 7f15c6000700 1 -- 172.21.15.72:0/893389224 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f15bc013070 con 0x7f15c8102370 2024-04-26T23:44:52.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.298+0000 7f15ce800700 1 -- 172.21.15.72:0/893389224 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f15c81aceb0 con 0x7f15c8102370 2024-04-26T23:44:52.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.299+0000 7f15c6000700 1 -- 172.21.15.72:0/893389224 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f15bc09dd00 con 0x7f15c8102370 2024-04-26T23:44:52.302 INFO:teuthology.orchestra.run.smithi072.stdout:253403070516 2024-04-26T23:44:52.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.302+0000 7f15ce800700 1 -- 172.21.15.72:0/893389224 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f15b00a3d50 msgr2=0x7f15b00a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:52.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.302+0000 7f15ce800700 1 --2- 172.21.15.72:0/893389224 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f15b00a3d50 0x7f15b00a6210 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f15b4009680 tx=0x7f15b4009260 comp rx=0 tx=0).stop 2024-04-26T23:44:52.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.302+0000 7f15ce800700 1 -- 172.21.15.72:0/893389224 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f15c8102370 msgr2=0x7f15c81a00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:52.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.302+0000 7f15ce800700 1 --2- 172.21.15.72:0/893389224 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f15c8102370 0x7f15c81a00f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f15bc0057f0 tx=0x7f15bc00bfd0 comp rx=0 tx=0).stop 2024-04-26T23:44:52.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.302+0000 7f15ce800700 1 -- 172.21.15.72:0/893389224 shutdown_connections 2024-04-26T23:44:52.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.302+0000 7f15ce800700 1 --2- 172.21.15.72:0/893389224 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f15b00a3d50 0x7f15b00a6210 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:52.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.302+0000 7f15ce800700 1 --2- 172.21.15.72:0/893389224 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15c8107780 0x7f15c81a4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:52.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.302+0000 7f15ce800700 1 --2- 172.21.15.72:0/893389224 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f15c8102370 0x7f15c81a00f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:52.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.302+0000 7f15ce800700 1 --2- 172.21.15.72:0/893389224 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f15c8102c90 0x7f15c81a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:52.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.302+0000 7f15ce800700 1 -- 172.21.15.72:0/893389224 >> 172.21.15.72:0/893389224 conn(0x7f15c80fddd0 msgr2=0x7f15c8110f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:44:52.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.302+0000 7f15ce800700 1 -- 172.21.15.72:0/893389224 shutdown_connections 2024-04-26T23:44:52.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:52.302+0000 7f15ce800700 1 -- 172.21.15.72:0/893389224 wait complete. 2024-04-26T23:44:52.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:52 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/893389224' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-26T23:44:52.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:52 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/893389224' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-26T23:44:52.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:52 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/893389224' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-26T23:44:53.165 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070509 got 253403070516 for osd.8 2024-04-26T23:44:53.165 DEBUG:teuthology.parallel:result is None 2024-04-26T23:44:53.506 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:53 smithi072 ceph-mon[28184]: pgmap v378: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:44:53.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:53 smithi076 ceph-mon[32774]: pgmap v378: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:44:53.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:53 smithi136 ceph-mon[33050]: pgmap v378: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:44:53.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:53.821+0000 7ff397a00700 1 -- 172.21.15.72:0/2871473597 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff390105640 msgr2=0x7ff390105ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:53.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:53.821+0000 7ff397a00700 1 --2- 172.21.15.72:0/2871473597 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff390105640 0x7ff390105ac0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff37c009990 tx=0x7ff37c009ca0 comp rx=0 tx=0).stop 2024-04-26T23:44:53.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:53.821+0000 7ff397a00700 1 -- 172.21.15.72:0/2871473597 shutdown_connections 2024-04-26T23:44:53.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:53.821+0000 7ff397a00700 1 --2- 172.21.15.72:0/2871473597 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff390106000 0x7ff390112510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:53.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:53.821+0000 7ff397a00700 1 --2- 172.21.15.72:0/2871473597 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff39010b640 0x7ff39010ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:53.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:53.821+0000 7ff397a00700 1 --2- 172.21.15.72:0/2871473597 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff390105640 0x7ff390105ac0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:53.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:53.821+0000 7ff397a00700 1 -- 172.21.15.72:0/2871473597 >> 172.21.15.72:0/2871473597 conn(0x7ff3901010f0 msgr2=0x7ff390103510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:44:53.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:53.821+0000 7ff397a00700 1 -- 172.21.15.72:0/2871473597 shutdown_connections 2024-04-26T23:44:53.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:53.821+0000 7ff397a00700 1 -- 172.21.15.72:0/2871473597 wait complete. 2024-04-26T23:44:53.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:53.823+0000 7ff397a00700 1 Processor -- start 2024-04-26T23:44:53.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:53.823+0000 7ff397a00700 1 -- start start 2024-04-26T23:44:53.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:53.823+0000 7ff397a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff390105640 0x7ff3901a1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:53.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:53.823+0000 7ff397a00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff390106000 0x7ff3901a1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:53.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:53.823+0000 7ff397a00700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff39010b640 0x7ff39019bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:53.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:53.823+0000 7ff397a00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7ff390115840 con 0x7ff390106000 2024-04-26T23:44:53.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:53.823+0000 7ff397a00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff3901156c0 con 0x7ff390105640 2024-04-26T23:44:53.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:53.823+0000 7ff397a00700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff3901159c0 con 0x7ff39010b640 2024-04-26T23:44:53.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:53.824+0000 7ff38fe00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff390106000 0x7ff3901a1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:53.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:53.824+0000 7ff395000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff390105640 0x7ff3901a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:53.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:53.824+0000 7ff38fe00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff390106000 0x7ff3901a1e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57864/0 (socket says 172.21.15.72:57864) 2024-04-26T23:44:53.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:53.824+0000 7ff395000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff390105640 0x7ff3901a1910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:38222/0 (socket says 172.21.15.72:38222) 2024-04-26T23:44:53.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:53.824+0000 7ff38fe00700 1 -- 172.21.15.72:0/1817423714 learned_addr learned my addr 172.21.15.72:0/1817423714 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:44:53.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:53.824+0000 7ff38fe00700 1 -- 172.21.15.72:0/1817423714 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff39010b640 msgr2=0x7ff39019bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:53.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:53.824+0000 7ff38fe00700 1 --2- 172.21.15.72:0/1817423714 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff39010b640 0x7ff39019bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:53.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:53.824+0000 7ff38fe00700 1 -- 172.21.15.72:0/1817423714 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff390105640 msgr2=0x7ff3901a1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:53.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:53.824+0000 7ff38fe00700 1 --2- 172.21.15.72:0/1817423714 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff390105640 0x7ff3901a1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:53.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:53.824+0000 7ff38fe00700 1 -- 172.21.15.72:0/1817423714 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff39019c280 con 0x7ff390106000 2024-04-26T23:44:53.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:53.824+0000 7ff395000700 1 --2- 172.21.15.72:0/1817423714 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff390105640 0x7ff3901a1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:44:53.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:53.825+0000 7ff38fe00700 1 --2- 172.21.15.72:0/1817423714 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff390106000 0x7ff3901a1e50 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ff37c000c00 tx=0x7ff37c00bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:53.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:53.825+0000 7ff38d600700 1 -- 172.21.15.72:0/1817423714 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff37c027070 con 0x7ff390106000 2024-04-26T23:44:53.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:53.825+0000 7ff38d600700 1 -- 172.21.15.72:0/1817423714 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff37c022470 con 0x7ff390106000 2024-04-26T23:44:53.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:53.825+0000 7ff397a00700 1 -- 172.21.15.72:0/1817423714 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff39019c510 con 0x7ff390106000 2024-04-26T23:44:53.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:53.825+0000 7ff38d600700 1 -- 172.21.15.72:0/1817423714 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff37c017650 con 0x7ff390106000 2024-04-26T23:44:53.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:53.825+0000 7ff397a00700 1 -- 172.21.15.72:0/1817423714 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3901a87e0 con 0x7ff390106000 2024-04-26T23:44:53.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:53.826+0000 7ff397a00700 1 -- 172.21.15.72:0/1817423714 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff390106d80 con 0x7ff390106000 2024-04-26T23:44:53.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:53.830+0000 7ff38d600700 1 -- 172.21.15.72:0/1817423714 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff37c02a070 con 0x7ff390106000 2024-04-26T23:44:53.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:53.831+0000 7ff38d600700 1 --2- 172.21.15.72:0/1817423714 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff3780a3cd0 0x7ff3780a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:53.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:53.832+0000 7ff38d600700 1 -- 172.21.15.72:0/1817423714 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff37c0d17e0 con 0x7ff390106000 2024-04-26T23:44:53.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:53.832+0000 7ff395000700 1 --2- 172.21.15.72:0/1817423714 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff3780a3cd0 0x7ff3780a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:53.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:53.832+0000 7ff395000700 1 --2- 172.21.15.72:0/1817423714 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff3780a3cd0 0x7ff3780a6190 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff384009fd0 tx=0x7ff384009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:53.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:53.834+0000 7ff38d600700 1 -- 172.21.15.72:0/1817423714 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff37c013070 con 0x7ff390106000 2024-04-26T23:44:54.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:54.065+0000 7ff397a00700 1 -- 172.21.15.72:0/1817423714 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7ff39002e490 con 0x7ff390106000 2024-04-26T23:44:54.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:54.066+0000 7ff38d600700 1 -- 172.21.15.72:0/1817423714 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff37c09dd30 con 0x7ff390106000 2024-04-26T23:44:54.070 INFO:teuthology.orchestra.run.smithi072.stdout:34359738465 2024-04-26T23:44:54.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:54.069+0000 7ff397a00700 1 -- 172.21.15.72:0/1817423714 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff3780a3cd0 msgr2=0x7ff3780a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:54.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:54.069+0000 7ff397a00700 1 --2- 172.21.15.72:0/1817423714 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff3780a3cd0 0x7ff3780a6190 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff384009fd0 tx=0x7ff384009260 comp rx=0 tx=0).stop 2024-04-26T23:44:54.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:54.069+0000 7ff397a00700 1 -- 172.21.15.72:0/1817423714 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff390106000 msgr2=0x7ff3901a1e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:54.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:54.069+0000 7ff397a00700 1 --2- 172.21.15.72:0/1817423714 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff390106000 0x7ff3901a1e50 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ff37c000c00 tx=0x7ff37c00bfd0 comp rx=0 tx=0).stop 2024-04-26T23:44:54.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:54.069+0000 7ff397a00700 1 -- 172.21.15.72:0/1817423714 shutdown_connections 2024-04-26T23:44:54.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:54.069+0000 7ff397a00700 1 --2- 172.21.15.72:0/1817423714 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff3780a3cd0 0x7ff3780a6190 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:54.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:54.069+0000 7ff397a00700 1 --2- 172.21.15.72:0/1817423714 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff39010b640 0x7ff39019bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:54.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:54.069+0000 7ff397a00700 1 --2- 172.21.15.72:0/1817423714 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff390105640 0x7ff3901a1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:54.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:54.069+0000 7ff397a00700 1 --2- 172.21.15.72:0/1817423714 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff390106000 0x7ff3901a1e50 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:54.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:54.069+0000 7ff397a00700 1 -- 172.21.15.72:0/1817423714 >> 172.21.15.72:0/1817423714 conn(0x7ff3901010f0 msgr2=0x7ff390101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:44:54.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:54.069+0000 7ff397a00700 1 -- 172.21.15.72:0/1817423714 shutdown_connections 2024-04-26T23:44:54.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:54.070+0000 7ff397a00700 1 -- 172.21.15.72:0/1817423714 wait complete. 2024-04-26T23:44:54.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:54 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/1817423714' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-26T23:44:54.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:54 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/1817423714' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-26T23:44:54.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:54 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/1817423714' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-26T23:44:54.949 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738459 got 34359738465 for osd.0 2024-04-26T23:44:54.949 DEBUG:teuthology.parallel:result is None 2024-04-26T23:44:54.949 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-26T23:44:54.950 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T23:44:55.401 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:44:55.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:55 smithi076 ceph-mon[32774]: pgmap v379: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:44:55.758 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:55 smithi072 ceph-mon[28184]: pgmap v379: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:44:55.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:55 smithi136 ceph-mon[33050]: pgmap v379: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:44:57.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.692+0000 7fc40e000700 1 -- 172.21.15.72:0/750228672 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc40806b9c0 msgr2=0x7fc40806be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:57.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.692+0000 7fc40e000700 1 --2- 172.21.15.72:0/750228672 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc40806b9c0 0x7fc40806be20 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fc3f4009990 tx=0x7fc3f4009ca0 comp rx=0 tx=0).stop 2024-04-26T23:44:57.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.692+0000 7fc40e000700 1 -- 172.21.15.72:0/750228672 shutdown_connections 2024-04-26T23:44:57.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.692+0000 7fc40e000700 1 --2- 172.21.15.72:0/750228672 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc40806c360 0x7fc4081102d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:57.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.692+0000 7fc40e000700 1 --2- 172.21.15.72:0/750228672 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc408109490 0x7fc408109870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:57.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.692+0000 7fc40e000700 1 --2- 172.21.15.72:0/750228672 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc40806b9c0 0x7fc40806be20 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:57.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.692+0000 7fc40e000700 1 -- 172.21.15.72:0/750228672 >> 172.21.15.72:0/750228672 conn(0x7fc408079290 msgr2=0x7fc4080796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:44:57.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.692+0000 7fc40e000700 1 -- 172.21.15.72:0/750228672 shutdown_connections 2024-04-26T23:44:57.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.692+0000 7fc40e000700 1 -- 172.21.15.72:0/750228672 wait complete. 2024-04-26T23:44:57.698 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:57 smithi072 ceph-mon[28184]: pgmap v380: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:44:57.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.694+0000 7fc40e000700 1 Processor -- start 2024-04-26T23:44:57.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.694+0000 7fc40e000700 1 -- start start 2024-04-26T23:44:57.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.694+0000 7fc40e000700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc40806b9c0 0x7fc4081a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:57.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.695+0000 7fc40e000700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc40806c360 0x7fc4081a1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:57.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.695+0000 7fc40e000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc408109490 0x7fc40819baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:57.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.695+0000 7fc40e000700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fc4081135f0 con 0x7fc40806b9c0 2024-04-26T23:44:57.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.695+0000 7fc40e000700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc408113470 con 0x7fc408109490 2024-04-26T23:44:57.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.695+0000 7fc40e000700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fc408113770 con 0x7fc40806c360 2024-04-26T23:44:57.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.695+0000 7fc407400700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc40806b9c0 0x7fc4081a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:57.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.695+0000 7fc407400700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc40806b9c0 0x7fc4081a1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56106/0 (socket says 172.21.15.72:56106) 2024-04-26T23:44:57.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.695+0000 7fc407400700 1 -- 172.21.15.72:0/2442278710 learned_addr learned my addr 172.21.15.72:0/2442278710 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:44:57.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.695+0000 7fc407e00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc408109490 0x7fc40819baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:57.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.695+0000 7fc406a00700 1 --2- 172.21.15.72:0/2442278710 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc40806c360 0x7fc4081a1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:57.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.695+0000 7fc406a00700 1 -- 172.21.15.72:0/2442278710 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc408109490 msgr2=0x7fc40819baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:57.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.695+0000 7fc406a00700 1 --2- 172.21.15.72:0/2442278710 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc408109490 0x7fc40819baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:57.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.695+0000 7fc406a00700 1 -- 172.21.15.72:0/2442278710 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc40806b9c0 msgr2=0x7fc4081a1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:57.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.695+0000 7fc406a00700 1 --2- 172.21.15.72:0/2442278710 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc40806b9c0 0x7fc4081a1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:57.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.695+0000 7fc406a00700 1 -- 172.21.15.72:0/2442278710 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc40819c250 con 0x7fc40806c360 2024-04-26T23:44:57.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.695+0000 7fc407400700 1 --2- 172.21.15.72:0/2442278710 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc40806b9c0 0x7fc4081a1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:44:57.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.696+0000 7fc406a00700 1 --2- 172.21.15.72:0/2442278710 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc40806c360 0x7fc4081a1e70 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc3f4000c00 tx=0x7fc3f40047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:57.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.696+0000 7fc3ffe00700 1 -- 172.21.15.72:0/2442278710 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3f4026070 con 0x7fc40806c360 2024-04-26T23:44:57.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.696+0000 7fc3ffe00700 1 -- 172.21.15.72:0/2442278710 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3f4021410 con 0x7fc40806c360 2024-04-26T23:44:57.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.696+0000 7fc40e000700 1 -- 172.21.15.72:0/2442278710 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc40819c4e0 con 0x7fc40806c360 2024-04-26T23:44:57.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.696+0000 7fc3ffe00700 1 -- 172.21.15.72:0/2442278710 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3f4017650 con 0x7fc40806c360 2024-04-26T23:44:57.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.696+0000 7fc40e000700 1 -- 172.21.15.72:0/2442278710 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4081a8800 con 0x7fc40806c360 2024-04-26T23:44:57.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.697+0000 7fc40e000700 1 -- 172.21.15.72:0/2442278710 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc40806a110 con 0x7fc40806c360 2024-04-26T23:44:57.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.699+0000 7fc3ffe00700 1 -- 172.21.15.72:0/2442278710 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc3f4029070 con 0x7fc40806c360 2024-04-26T23:44:57.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.701+0000 7fc3ffe00700 1 --2- 172.21.15.72:0/2442278710 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fc3ec0a3d50 0x7fc3ec0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:44:57.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.701+0000 7fc3ffe00700 1 -- 172.21.15.72:0/2442278710 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fc3f40d1570 con 0x7fc40806c360 2024-04-26T23:44:57.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.701+0000 7fc407400700 1 --2- 172.21.15.72:0/2442278710 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fc3ec0a3d50 0x7fc3ec0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:44:57.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.702+0000 7fc407400700 1 --2- 172.21.15.72:0/2442278710 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fc3ec0a3d50 0x7fc3ec0a6210 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc3f8009680 tx=0x7fc3f8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:44:57.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.705+0000 7fc3ffe00700 1 -- 172.21.15.72:0/2442278710 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc3f405e060 con 0x7fc40806c360 2024-04-26T23:44:57.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:57 smithi136 ceph-mon[33050]: pgmap v380: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:44:57.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:57 smithi076 ceph-mon[32774]: pgmap v380: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:44:57.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.926+0000 7fc40e000700 1 -- 172.21.15.72:0/2442278710 --> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc408103140 con 0x7fc3ec0a3d50 2024-04-26T23:44:57.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.941+0000 7fc3ffe00700 1 -- 172.21.15.72:0/2442278710 <== mgr.14152 v2:172.21.15.72:6800/4281177709 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340630 (secure 0 0 0) 0x7fc408103140 con 0x7fc3ec0a3d50 2024-04-26T23:44:57.953 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:44:57.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.948+0000 7fc40e000700 1 -- 172.21.15.72:0/2442278710 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fc3ec0a3d50 msgr2=0x7fc3ec0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:57.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.948+0000 7fc40e000700 1 --2- 172.21.15.72:0/2442278710 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fc3ec0a3d50 0x7fc3ec0a6210 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc3f8009680 tx=0x7fc3f8009260 comp rx=0 tx=0).stop 2024-04-26T23:44:57.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.948+0000 7fc40e000700 1 -- 172.21.15.72:0/2442278710 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc40806c360 msgr2=0x7fc4081a1e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:44:57.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.948+0000 7fc40e000700 1 --2- 172.21.15.72:0/2442278710 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc40806c360 0x7fc4081a1e70 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc3f4000c00 tx=0x7fc3f40047b0 comp rx=0 tx=0).stop 2024-04-26T23:44:57.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.948+0000 7fc40e000700 1 -- 172.21.15.72:0/2442278710 shutdown_connections 2024-04-26T23:44:57.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.948+0000 7fc40e000700 1 --2- 172.21.15.72:0/2442278710 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fc3ec0a3d50 0x7fc3ec0a6210 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:57.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.948+0000 7fc40e000700 1 --2- 172.21.15.72:0/2442278710 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc408109490 0x7fc40819baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:57.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.948+0000 7fc40e000700 1 --2- 172.21.15.72:0/2442278710 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc40806b9c0 0x7fc4081a1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:57.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.948+0000 7fc40e000700 1 --2- 172.21.15.72:0/2442278710 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc40806c360 0x7fc4081a1e70 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:44:57.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.948+0000 7fc40e000700 1 -- 172.21.15.72:0/2442278710 >> 172.21.15.72:0/2442278710 conn(0x7fc408079290 msgr2=0x7fc408100420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:44:57.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.949+0000 7fc40e000700 1 -- 172.21.15.72:0/2442278710 shutdown_connections 2024-04-26T23:44:57.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:44:57.949+0000 7fc40e000700 1 -- 172.21.15.72:0/2442278710 wait complete. 2024-04-26T23:44:57.957 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-04-26T23:44:58.767 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":380,"stamp":"2024-04-26T23:44:56.459354+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":649820,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":657,"num_read_kb":437,"num_write":344,"num_write_kb":3049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":1180848,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":279,"ondisk_log_size":279,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":357184,"kb_used_data":12140,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124584640,"statfs":{"total":1151940427776,"available":1151574671360,"internally_reserved":0,"allocated":12431360,"data_stored":7935362,"data_compressed":354537,"data_compressed_allocated":479232,"data_compressed_original":2457600,"omap_allocated":19139,"internal_metadata":353154365},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":60,"num_read_kb":30,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005969"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.570110+0000","last_change":"2024-04-26T23:43:03.301992+0000","last_active":"2024-04-26T23:43:59.570110+0000","last_peered":"2024-04-26T23:43:59.570110+0000","last_clean":"2024-04-26T23:43:59.570110+0000","last_became_active":"2024-04-26T23:43:03.301498+0000","last_became_peered":"2024-04-26T23:43:03.301498+0000","last_unstale":"2024-04-26T23:43:59.570110+0000","last_undegraded":"2024-04-26T23:43:59.570110+0000","last_fullsized":"2024-04-26T23:43:59.570110+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:40:53.567805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.567766+0000","last_change":"2024-04-26T23:43:03.284964+0000","last_active":"2024-04-26T23:43:59.567766+0000","last_peered":"2024-04-26T23:43:59.567766+0000","last_clean":"2024-04-26T23:43:59.567766+0000","last_became_active":"2024-04-26T23:43:03.284185+0000","last_became_peered":"2024-04-26T23:43:03.284185+0000","last_unstale":"2024-04-26T23:43:59.567766+0000","last_undegraded":"2024-04-26T23:43:59.567766+0000","last_fullsized":"2024-04-26T23:43:59.567766+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:55:35.325775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.567001+0000","last_change":"2024-04-26T23:43:03.300333+0000","last_active":"2024-04-26T23:43:59.567001+0000","last_peered":"2024-04-26T23:43:59.567001+0000","last_clean":"2024-04-26T23:43:59.567001+0000","last_became_active":"2024-04-26T23:43:03.295779+0000","last_became_peered":"2024-04-26T23:43:03.295779+0000","last_unstale":"2024-04-26T23:43:59.567001+0000","last_undegraded":"2024-04-26T23:43:59.567001+0000","last_fullsized":"2024-04-26T23:43:59.567001+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:45:06.598862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.566294+0000","last_change":"2024-04-26T23:43:03.290211+0000","last_active":"2024-04-26T23:43:59.566294+0000","last_peered":"2024-04-26T23:43:59.566294+0000","last_clean":"2024-04-26T23:43:59.566294+0000","last_became_active":"2024-04-26T23:43:03.289853+0000","last_became_peered":"2024-04-26T23:43:03.289853+0000","last_unstale":"2024-04-26T23:43:59.566294+0000","last_undegraded":"2024-04-26T23:43:59.566294+0000","last_fullsized":"2024-04-26T23:43:59.566294+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:14:45.032300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.570830+0000","last_change":"2024-04-26T23:43:03.309667+0000","last_active":"2024-04-26T23:43:59.570830+0000","last_peered":"2024-04-26T23:43:59.570830+0000","last_clean":"2024-04-26T23:43:59.570830+0000","last_became_active":"2024-04-26T23:43:03.309305+0000","last_became_peered":"2024-04-26T23:43:03.309305+0000","last_unstale":"2024-04-26T23:43:59.570830+0000","last_undegraded":"2024-04-26T23:43:59.570830+0000","last_fullsized":"2024-04-26T23:43:59.570830+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:31:56.739866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.867340+0000","last_change":"2024-04-26T23:43:03.292107+0000","last_active":"2024-04-26T23:43:59.867340+0000","last_peered":"2024-04-26T23:43:59.867340+0000","last_clean":"2024-04-26T23:43:59.867340+0000","last_became_active":"2024-04-26T23:43:03.291760+0000","last_became_peered":"2024-04-26T23:43:03.291760+0000","last_unstale":"2024-04-26T23:43:59.867340+0000","last_undegraded":"2024-04-26T23:43:59.867340+0000","last_fullsized":"2024-04-26T23:43:59.867340+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:31:54.654379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.571420+0000","last_change":"2024-04-26T23:43:03.294923+0000","last_active":"2024-04-26T23:43:59.571420+0000","last_peered":"2024-04-26T23:43:59.571420+0000","last_clean":"2024-04-26T23:43:59.571420+0000","last_became_active":"2024-04-26T23:43:03.294523+0000","last_became_peered":"2024-04-26T23:43:03.294523+0000","last_unstale":"2024-04-26T23:43:59.571420+0000","last_undegraded":"2024-04-26T23:43:59.571420+0000","last_fullsized":"2024-04-26T23:43:59.571420+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:36:19.419773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.568934+0000","last_change":"2024-04-26T23:43:03.301335+0000","last_active":"2024-04-26T23:43:59.568934+0000","last_peered":"2024-04-26T23:43:59.568934+0000","last_clean":"2024-04-26T23:43:59.568934+0000","last_became_active":"2024-04-26T23:43:03.300918+0000","last_became_peered":"2024-04-26T23:43:03.300918+0000","last_unstale":"2024-04-26T23:43:59.568934+0000","last_undegraded":"2024-04-26T23:43:59.568934+0000","last_fullsized":"2024-04-26T23:43:59.568934+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:15:05.719452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.574776+0000","last_change":"2024-04-26T23:43:59.574776+0000","last_active":"2024-04-26T23:43:59.574776+0000","last_peered":"2024-04-26T23:43:59.574776+0000","last_clean":"2024-04-26T23:43:59.574776+0000","last_became_active":"2024-04-26T23:43:59.574299+0000","last_became_peered":"2024-04-26T23:43:59.574299+0000","last_unstale":"2024-04-26T23:43:59.574776+0000","last_undegraded":"2024-04-26T23:43:59.574776+0000","last_fullsized":"2024-04-26T23:43:59.574776+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:14:16.089377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,11],"acting":[1,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.867834+0000","last_change":"2024-04-26T23:43:03.287643+0000","last_active":"2024-04-26T23:43:59.867834+0000","last_peered":"2024-04-26T23:43:59.867834+0000","last_clean":"2024-04-26T23:43:59.867834+0000","last_became_active":"2024-04-26T23:43:03.287324+0000","last_became_peered":"2024-04-26T23:43:03.287324+0000","last_unstale":"2024-04-26T23:43:59.867834+0000","last_undegraded":"2024-04-26T23:43:59.867834+0000","last_fullsized":"2024-04-26T23:43:59.867834+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:50:14.816548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.572333+0000","last_change":"2024-04-26T23:43:03.296421+0000","last_active":"2024-04-26T23:43:59.572333+0000","last_peered":"2024-04-26T23:43:59.572333+0000","last_clean":"2024-04-26T23:43:59.572333+0000","last_became_active":"2024-04-26T23:43:03.296060+0000","last_became_peered":"2024-04-26T23:43:03.296060+0000","last_unstale":"2024-04-26T23:43:59.572333+0000","last_undegraded":"2024-04-26T23:43:59.572333+0000","last_fullsized":"2024-04-26T23:43:59.572333+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:50:44.393446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.566081+0000","last_change":"2024-04-26T23:43:03.288562+0000","last_active":"2024-04-26T23:43:59.566081+0000","last_peered":"2024-04-26T23:43:59.566081+0000","last_clean":"2024-04-26T23:43:59.566081+0000","last_became_active":"2024-04-26T23:43:03.288219+0000","last_became_peered":"2024-04-26T23:43:03.288219+0000","last_unstale":"2024-04-26T23:43:59.566081+0000","last_undegraded":"2024-04-26T23:43:59.566081+0000","last_fullsized":"2024-04-26T23:43:59.566081+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:54:32.107146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.868611+0000","last_change":"2024-04-26T23:43:03.296840+0000","last_active":"2024-04-26T23:43:59.868611+0000","last_peered":"2024-04-26T23:43:59.868611+0000","last_clean":"2024-04-26T23:43:59.868611+0000","last_became_active":"2024-04-26T23:43:03.296501+0000","last_became_peered":"2024-04-26T23:43:03.296501+0000","last_unstale":"2024-04-26T23:43:59.868611+0000","last_undegraded":"2024-04-26T23:43:59.868611+0000","last_fullsized":"2024-04-26T23:43:59.868611+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:07:05.293388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.567210+0000","last_change":"2024-04-26T23:43:03.291739+0000","last_active":"2024-04-26T23:43:59.567210+0000","last_peered":"2024-04-26T23:43:59.567210+0000","last_clean":"2024-04-26T23:43:59.567210+0000","last_became_active":"2024-04-26T23:43:03.287839+0000","last_became_peered":"2024-04-26T23:43:03.287839+0000","last_unstale":"2024-04-26T23:43:59.567210+0000","last_undegraded":"2024-04-26T23:43:59.567210+0000","last_fullsized":"2024-04-26T23:43:59.567210+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:10:56.698324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.567020+0000","last_change":"2024-04-26T23:43:03.295767+0000","last_active":"2024-04-26T23:43:59.567020+0000","last_peered":"2024-04-26T23:43:59.567020+0000","last_clean":"2024-04-26T23:43:59.567020+0000","last_became_active":"2024-04-26T23:43:03.295339+0000","last_became_peered":"2024-04-26T23:43:03.295339+0000","last_unstale":"2024-04-26T23:43:59.567020+0000","last_undegraded":"2024-04-26T23:43:59.567020+0000","last_fullsized":"2024-04-26T23:43:59.567020+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:21:44.410953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.571434+0000","last_change":"2024-04-26T23:43:03.301687+0000","last_active":"2024-04-26T23:43:59.571434+0000","last_peered":"2024-04-26T23:43:59.571434+0000","last_clean":"2024-04-26T23:43:59.571434+0000","last_became_active":"2024-04-26T23:43:03.301355+0000","last_became_peered":"2024-04-26T23:43:03.301355+0000","last_unstale":"2024-04-26T23:43:59.571434+0000","last_undegraded":"2024-04-26T23:43:59.571434+0000","last_fullsized":"2024-04-26T23:43:59.571434+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:00:57.991022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.569995+0000","last_change":"2024-04-26T23:43:03.302109+0000","last_active":"2024-04-26T23:43:59.569995+0000","last_peered":"2024-04-26T23:43:59.569995+0000","last_clean":"2024-04-26T23:43:59.569995+0000","last_became_active":"2024-04-26T23:43:03.301725+0000","last_became_peered":"2024-04-26T23:43:03.301725+0000","last_unstale":"2024-04-26T23:43:59.569995+0000","last_undegraded":"2024-04-26T23:43:59.569995+0000","last_fullsized":"2024-04-26T23:43:59.569995+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:36:44.630125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.576171+0000","last_change":"2024-04-26T23:43:03.298880+0000","last_active":"2024-04-26T23:43:59.576171+0000","last_peered":"2024-04-26T23:43:59.576171+0000","last_clean":"2024-04-26T23:43:59.576171+0000","last_became_active":"2024-04-26T23:43:03.298464+0000","last_became_peered":"2024-04-26T23:43:03.298464+0000","last_unstale":"2024-04-26T23:43:59.576171+0000","last_undegraded":"2024-04-26T23:43:59.576171+0000","last_fullsized":"2024-04-26T23:43:59.576171+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:42:54.745002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.567554+0000","last_change":"2024-04-26T23:43:03.309454+0000","last_active":"2024-04-26T23:43:59.567554+0000","last_peered":"2024-04-26T23:43:59.567554+0000","last_clean":"2024-04-26T23:43:59.567554+0000","last_became_active":"2024-04-26T23:43:03.309091+0000","last_became_peered":"2024-04-26T23:43:03.309091+0000","last_unstale":"2024-04-26T23:43:59.567554+0000","last_undegraded":"2024-04-26T23:43:59.567554+0000","last_fullsized":"2024-04-26T23:43:59.567554+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:46:12.333410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.566347+0000","last_change":"2024-04-26T23:43:07.541317+0000","last_active":"2024-04-26T23:43:59.566347+0000","last_peered":"2024-04-26T23:43:59.566347+0000","last_clean":"2024-04-26T23:43:59.566347+0000","last_became_active":"2024-04-26T23:43:07.540760+0000","last_became_peered":"2024-04-26T23:43:07.540760+0000","last_unstale":"2024-04-26T23:43:59.566347+0000","last_undegraded":"2024-04-26T23:43:59.566347+0000","last_fullsized":"2024-04-26T23:43:59.566347+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:28:55.545416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.576255+0000","last_change":"2024-04-26T23:43:02.450956+0000","last_active":"2024-04-26T23:43:59.576255+0000","last_peered":"2024-04-26T23:43:59.576255+0000","last_clean":"2024-04-26T23:43:59.576255+0000","last_became_active":"2024-04-26T23:43:02.446733+0000","last_became_peered":"2024-04-26T23:43:02.446733+0000","last_unstale":"2024-04-26T23:43:59.576255+0000","last_undegraded":"2024-04-26T23:43:59.576255+0000","last_fullsized":"2024-04-26T23:43:59.576255+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:58:47.613575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.571020+0000","last_change":"2024-04-26T23:43:03.311681+0000","last_active":"2024-04-26T23:43:59.571020+0000","last_peered":"2024-04-26T23:43:59.571020+0000","last_clean":"2024-04-26T23:43:59.571020+0000","last_became_active":"2024-04-26T23:43:03.311087+0000","last_became_peered":"2024-04-26T23:43:03.311087+0000","last_unstale":"2024-04-26T23:43:59.571020+0000","last_undegraded":"2024-04-26T23:43:59.571020+0000","last_fullsized":"2024-04-26T23:43:59.571020+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:13:00.744062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.565507+0000","last_change":"2024-04-26T23:43:07.543547+0000","last_active":"2024-04-26T23:43:59.565507+0000","last_peered":"2024-04-26T23:43:59.565507+0000","last_clean":"2024-04-26T23:43:59.565507+0000","last_became_active":"2024-04-26T23:43:07.543073+0000","last_became_peered":"2024-04-26T23:43:07.543073+0000","last_unstale":"2024-04-26T23:43:59.565507+0000","last_undegraded":"2024-04-26T23:43:59.565507+0000","last_fullsized":"2024-04-26T23:43:59.565507+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:30:05.418875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.571145+0000","last_change":"2024-04-26T23:43:02.460459+0000","last_active":"2024-04-26T23:43:59.571145+0000","last_peered":"2024-04-26T23:43:59.571145+0000","last_clean":"2024-04-26T23:43:59.571145+0000","last_became_active":"2024-04-26T23:43:02.459652+0000","last_became_peered":"2024-04-26T23:43:02.459652+0000","last_unstale":"2024-04-26T23:43:59.571145+0000","last_undegraded":"2024-04-26T23:43:59.571145+0000","last_fullsized":"2024-04-26T23:43:59.571145+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:27:19.166748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.568072+0000","last_change":"2024-04-26T23:43:03.300078+0000","last_active":"2024-04-26T23:43:59.568072+0000","last_peered":"2024-04-26T23:43:59.568072+0000","last_clean":"2024-04-26T23:43:59.568072+0000","last_became_active":"2024-04-26T23:43:03.296319+0000","last_became_peered":"2024-04-26T23:43:03.296319+0000","last_unstale":"2024-04-26T23:43:59.568072+0000","last_undegraded":"2024-04-26T23:43:59.568072+0000","last_fullsized":"2024-04-26T23:43:59.568072+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:33:51.859539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.565683+0000","last_change":"2024-04-26T23:43:07.543494+0000","last_active":"2024-04-26T23:43:59.565683+0000","last_peered":"2024-04-26T23:43:59.565683+0000","last_clean":"2024-04-26T23:43:59.565683+0000","last_became_active":"2024-04-26T23:43:07.542806+0000","last_became_peered":"2024-04-26T23:43:07.542806+0000","last_unstale":"2024-04-26T23:43:59.565683+0000","last_undegraded":"2024-04-26T23:43:59.565683+0000","last_fullsized":"2024-04-26T23:43:59.565683+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:54:40.522615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.571618+0000","last_change":"2024-04-26T23:43:02.438943+0000","last_active":"2024-04-26T23:43:59.571618+0000","last_peered":"2024-04-26T23:43:59.571618+0000","last_clean":"2024-04-26T23:43:59.571618+0000","last_became_active":"2024-04-26T23:43:02.438604+0000","last_became_peered":"2024-04-26T23:43:02.438604+0000","last_unstale":"2024-04-26T23:43:59.571618+0000","last_undegraded":"2024-04-26T23:43:59.571618+0000","last_fullsized":"2024-04-26T23:43:59.571618+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:06:51.054342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.573038+0000","last_change":"2024-04-26T23:43:03.293856+0000","last_active":"2024-04-26T23:43:59.573038+0000","last_peered":"2024-04-26T23:43:59.573038+0000","last_clean":"2024-04-26T23:43:59.573038+0000","last_became_active":"2024-04-26T23:43:03.293189+0000","last_became_peered":"2024-04-26T23:43:03.293189+0000","last_unstale":"2024-04-26T23:43:59.573038+0000","last_undegraded":"2024-04-26T23:43:59.573038+0000","last_fullsized":"2024-04-26T23:43:59.573038+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:50:27.545567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.565535+0000","last_change":"2024-04-26T23:43:07.535396+0000","last_active":"2024-04-26T23:43:59.565535+0000","last_peered":"2024-04-26T23:43:59.565535+0000","last_clean":"2024-04-26T23:43:59.565535+0000","last_became_active":"2024-04-26T23:43:07.534914+0000","last_became_peered":"2024-04-26T23:43:07.534914+0000","last_unstale":"2024-04-26T23:43:59.565535+0000","last_undegraded":"2024-04-26T23:43:59.565535+0000","last_fullsized":"2024-04-26T23:43:59.565535+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:27:31.199079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.571839+0000","last_change":"2024-04-26T23:43:02.442935+0000","last_active":"2024-04-26T23:43:59.571839+0000","last_peered":"2024-04-26T23:43:59.571839+0000","last_clean":"2024-04-26T23:43:59.571839+0000","last_became_active":"2024-04-26T23:43:02.442588+0000","last_became_peered":"2024-04-26T23:43:02.442588+0000","last_unstale":"2024-04-26T23:43:59.571839+0000","last_undegraded":"2024-04-26T23:43:59.571839+0000","last_fullsized":"2024-04-26T23:43:59.571839+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:05:26.442001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.569033+0000","last_change":"2024-04-26T23:43:03.289407+0000","last_active":"2024-04-26T23:43:59.569033+0000","last_peered":"2024-04-26T23:43:59.569033+0000","last_clean":"2024-04-26T23:43:59.569033+0000","last_became_active":"2024-04-26T23:43:03.287951+0000","last_became_peered":"2024-04-26T23:43:03.287951+0000","last_unstale":"2024-04-26T23:43:59.569033+0000","last_undegraded":"2024-04-26T23:43:59.569033+0000","last_fullsized":"2024-04-26T23:43:59.569033+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:24:42.687653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.567047+0000","last_change":"2024-04-26T23:43:07.527019+0000","last_active":"2024-04-26T23:43:59.567047+0000","last_peered":"2024-04-26T23:43:59.567047+0000","last_clean":"2024-04-26T23:43:59.567047+0000","last_became_active":"2024-04-26T23:43:07.526516+0000","last_became_peered":"2024-04-26T23:43:07.526516+0000","last_unstale":"2024-04-26T23:43:59.567047+0000","last_undegraded":"2024-04-26T23:43:59.567047+0000","last_fullsized":"2024-04-26T23:43:59.567047+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:44:41.267144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'6","reported_seq":120,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:44:52.327980+0000","last_change":"2024-04-26T23:43:02.445159+0000","last_active":"2024-04-26T23:44:52.327980+0000","last_peered":"2024-04-26T23:44:52.327980+0000","last_clean":"2024-04-26T23:44:52.327980+0000","last_became_active":"2024-04-26T23:43:02.444781+0000","last_became_peered":"2024-04-26T23:43:02.444781+0000","last_unstale":"2024-04-26T23:44:52.327980+0000","last_undegraded":"2024-04-26T23:44:52.327980+0000","last_fullsized":"2024-04-26T23:44:52.327980+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:45:16.509790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":87,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.572128+0000","last_change":"2024-04-26T23:43:03.292717+0000","last_active":"2024-04-26T23:43:59.572128+0000","last_peered":"2024-04-26T23:43:59.572128+0000","last_clean":"2024-04-26T23:43:59.572128+0000","last_became_active":"2024-04-26T23:43:03.292355+0000","last_became_peered":"2024-04-26T23:43:03.292355+0000","last_unstale":"2024-04-26T23:43:59.572128+0000","last_undegraded":"2024-04-26T23:43:59.572128+0000","last_fullsized":"2024-04-26T23:43:59.572128+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:06:02.063165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.566558+0000","last_change":"2024-04-26T23:43:07.540758+0000","last_active":"2024-04-26T23:43:59.566558+0000","last_peered":"2024-04-26T23:43:59.566558+0000","last_clean":"2024-04-26T23:43:59.566558+0000","last_became_active":"2024-04-26T23:43:07.540305+0000","last_became_peered":"2024-04-26T23:43:07.540305+0000","last_unstale":"2024-04-26T23:43:59.566558+0000","last_undegraded":"2024-04-26T23:43:59.566558+0000","last_fullsized":"2024-04-26T23:43:59.566558+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:08:04.491544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.867249+0000","last_change":"2024-04-26T23:43:02.453650+0000","last_active":"2024-04-26T23:43:59.867249+0000","last_peered":"2024-04-26T23:43:59.867249+0000","last_clean":"2024-04-26T23:43:59.867249+0000","last_became_active":"2024-04-26T23:43:02.453319+0000","last_became_peered":"2024-04-26T23:43:02.453319+0000","last_unstale":"2024-04-26T23:43:59.867249+0000","last_undegraded":"2024-04-26T23:43:59.867249+0000","last_fullsized":"2024-04-26T23:43:59.867249+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:23:27.570603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.870417+0000","last_change":"2024-04-26T23:43:59.870417+0000","last_active":"2024-04-26T23:43:59.870417+0000","last_peered":"2024-04-26T23:43:59.870417+0000","last_clean":"2024-04-26T23:43:59.870417+0000","last_became_active":"2024-04-26T23:43:59.870053+0000","last_became_peered":"2024-04-26T23:43:59.870053+0000","last_unstale":"2024-04-26T23:43:59.870417+0000","last_undegraded":"2024-04-26T23:43:59.870417+0000","last_fullsized":"2024-04-26T23:43:59.870417+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:07:31.020611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.569999+0000","last_change":"2024-04-26T23:43:07.546026+0000","last_active":"2024-04-26T23:43:59.569999+0000","last_peered":"2024-04-26T23:43:59.569999+0000","last_clean":"2024-04-26T23:43:59.569999+0000","last_became_active":"2024-04-26T23:43:07.545668+0000","last_became_peered":"2024-04-26T23:43:07.545668+0000","last_unstale":"2024-04-26T23:43:59.569999+0000","last_undegraded":"2024-04-26T23:43:59.569999+0000","last_fullsized":"2024-04-26T23:43:59.569999+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:54:20.580267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.868447+0000","last_change":"2024-04-26T23:43:02.454937+0000","last_active":"2024-04-26T23:43:59.868447+0000","last_peered":"2024-04-26T23:43:59.868447+0000","last_clean":"2024-04-26T23:43:59.868447+0000","last_became_active":"2024-04-26T23:43:02.454585+0000","last_became_peered":"2024-04-26T23:43:02.454585+0000","last_unstale":"2024-04-26T23:43:59.868447+0000","last_undegraded":"2024-04-26T23:43:59.868447+0000","last_fullsized":"2024-04-26T23:43:59.868447+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:55:54.999411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.873013+0000","last_change":"2024-04-26T23:43:59.870821+0000","last_active":"2024-04-26T23:43:59.873013+0000","last_peered":"2024-04-26T23:43:59.873013+0000","last_clean":"2024-04-26T23:43:59.873013+0000","last_became_active":"2024-04-26T23:43:59.870240+0000","last_became_peered":"2024-04-26T23:43:59.870240+0000","last_unstale":"2024-04-26T23:43:59.873013+0000","last_undegraded":"2024-04-26T23:43:59.873013+0000","last_fullsized":"2024-04-26T23:43:59.873013+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:33:00.284232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,6],"acting":[9,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.567737+0000","last_change":"2024-04-26T23:43:07.535682+0000","last_active":"2024-04-26T23:43:59.567737+0000","last_peered":"2024-04-26T23:43:59.567737+0000","last_clean":"2024-04-26T23:43:59.567737+0000","last_became_active":"2024-04-26T23:43:07.535221+0000","last_became_peered":"2024-04-26T23:43:07.535221+0000","last_unstale":"2024-04-26T23:43:59.567737+0000","last_undegraded":"2024-04-26T23:43:59.567737+0000","last_fullsized":"2024-04-26T23:43:59.567737+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:33:54.105329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.569396+0000","last_change":"2024-04-26T23:43:02.443541+0000","last_active":"2024-04-26T23:43:59.569396+0000","last_peered":"2024-04-26T23:43:59.569396+0000","last_clean":"2024-04-26T23:43:59.569396+0000","last_became_active":"2024-04-26T23:43:02.442948+0000","last_became_peered":"2024-04-26T23:43:02.442948+0000","last_unstale":"2024-04-26T23:43:59.569396+0000","last_undegraded":"2024-04-26T23:43:59.569396+0000","last_fullsized":"2024-04-26T23:43:59.569396+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:58:44.595575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.574143+0000","last_change":"2024-04-26T23:43:03.311029+0000","last_active":"2024-04-26T23:43:59.574143+0000","last_peered":"2024-04-26T23:43:59.574143+0000","last_clean":"2024-04-26T23:43:59.574143+0000","last_became_active":"2024-04-26T23:43:03.310722+0000","last_became_peered":"2024-04-26T23:43:03.310722+0000","last_unstale":"2024-04-26T23:43:59.574143+0000","last_undegraded":"2024-04-26T23:43:59.574143+0000","last_fullsized":"2024-04-26T23:43:59.574143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:07:54.449492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.569381+0000","last_change":"2024-04-26T23:43:07.537391+0000","last_active":"2024-04-26T23:43:59.569381+0000","last_peered":"2024-04-26T23:43:59.569381+0000","last_clean":"2024-04-26T23:43:59.569381+0000","last_became_active":"2024-04-26T23:43:07.536216+0000","last_became_peered":"2024-04-26T23:43:07.536216+0000","last_unstale":"2024-04-26T23:43:59.569381+0000","last_undegraded":"2024-04-26T23:43:59.569381+0000","last_fullsized":"2024-04-26T23:43:59.569381+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:45:55.326283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":113,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:44:52.327074+0000","last_change":"2024-04-26T23:43:02.451929+0000","last_active":"2024-04-26T23:44:52.327074+0000","last_peered":"2024-04-26T23:44:52.327074+0000","last_clean":"2024-04-26T23:44:52.327074+0000","last_became_active":"2024-04-26T23:43:02.451700+0000","last_became_peered":"2024-04-26T23:43:02.451700+0000","last_unstale":"2024-04-26T23:44:52.327074+0000","last_undegraded":"2024-04-26T23:44:52.327074+0000","last_fullsized":"2024-04-26T23:44:52.327074+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:21:31.279681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":84,"num_read_kb":84,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.572719+0000","last_change":"2024-04-26T23:43:03.295462+0000","last_active":"2024-04-26T23:43:59.572719+0000","last_peered":"2024-04-26T23:43:59.572719+0000","last_clean":"2024-04-26T23:43:59.572719+0000","last_became_active":"2024-04-26T23:43:03.295044+0000","last_became_peered":"2024-04-26T23:43:03.295044+0000","last_unstale":"2024-04-26T23:43:59.572719+0000","last_undegraded":"2024-04-26T23:43:59.572719+0000","last_fullsized":"2024-04-26T23:43:59.572719+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:56:19.445166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.573518+0000","last_change":"2024-04-26T23:43:07.539397+0000","last_active":"2024-04-26T23:43:59.573518+0000","last_peered":"2024-04-26T23:43:59.573518+0000","last_clean":"2024-04-26T23:43:59.573518+0000","last_became_active":"2024-04-26T23:43:07.533081+0000","last_became_peered":"2024-04-26T23:43:07.533081+0000","last_unstale":"2024-04-26T23:43:59.573518+0000","last_undegraded":"2024-04-26T23:43:59.573518+0000","last_fullsized":"2024-04-26T23:43:59.573518+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:24:39.693721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.571849+0000","last_change":"2024-04-26T23:43:02.439919+0000","last_active":"2024-04-26T23:43:59.571849+0000","last_peered":"2024-04-26T23:43:59.571849+0000","last_clean":"2024-04-26T23:43:59.571849+0000","last_became_active":"2024-04-26T23:43:02.439561+0000","last_became_peered":"2024-04-26T23:43:02.439561+0000","last_unstale":"2024-04-26T23:43:59.571849+0000","last_undegraded":"2024-04-26T23:43:59.571849+0000","last_fullsized":"2024-04-26T23:43:59.571849+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:30:22.905464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.567979+0000","last_change":"2024-04-26T23:43:03.294088+0000","last_active":"2024-04-26T23:43:59.567979+0000","last_peered":"2024-04-26T23:43:59.567979+0000","last_clean":"2024-04-26T23:43:59.567979+0000","last_became_active":"2024-04-26T23:43:03.293726+0000","last_became_peered":"2024-04-26T23:43:03.293726+0000","last_unstale":"2024-04-26T23:43:59.567979+0000","last_undegraded":"2024-04-26T23:43:59.567979+0000","last_fullsized":"2024-04-26T23:43:59.567979+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:19:42.575193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.565828+0000","last_change":"2024-04-26T23:43:07.543056+0000","last_active":"2024-04-26T23:43:59.565828+0000","last_peered":"2024-04-26T23:43:59.565828+0000","last_clean":"2024-04-26T23:43:59.565828+0000","last_became_active":"2024-04-26T23:43:07.542324+0000","last_became_peered":"2024-04-26T23:43:07.542324+0000","last_unstale":"2024-04-26T23:43:59.565828+0000","last_undegraded":"2024-04-26T23:43:59.565828+0000","last_fullsized":"2024-04-26T23:43:59.565828+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:23:46.415932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.568667+0000","last_change":"2024-04-26T23:43:02.454135+0000","last_active":"2024-04-26T23:43:59.568667+0000","last_peered":"2024-04-26T23:43:59.568667+0000","last_clean":"2024-04-26T23:43:59.568667+0000","last_became_active":"2024-04-26T23:43:02.453798+0000","last_became_peered":"2024-04-26T23:43:02.453798+0000","last_unstale":"2024-04-26T23:43:59.568667+0000","last_undegraded":"2024-04-26T23:43:59.568667+0000","last_fullsized":"2024-04-26T23:43:59.568667+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:16:25.286569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.573165+0000","last_change":"2024-04-26T23:43:03.293520+0000","last_active":"2024-04-26T23:43:59.573165+0000","last_peered":"2024-04-26T23:43:59.573165+0000","last_clean":"2024-04-26T23:43:59.573165+0000","last_became_active":"2024-04-26T23:43:03.292537+0000","last_became_peered":"2024-04-26T23:43:03.292537+0000","last_unstale":"2024-04-26T23:43:59.573165+0000","last_undegraded":"2024-04-26T23:43:59.573165+0000","last_fullsized":"2024-04-26T23:43:59.573165+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:19:28.808217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.574638+0000","last_change":"2024-04-26T23:43:07.528116+0000","last_active":"2024-04-26T23:43:59.574638+0000","last_peered":"2024-04-26T23:43:59.574638+0000","last_clean":"2024-04-26T23:43:59.574638+0000","last_became_active":"2024-04-26T23:43:07.527766+0000","last_became_peered":"2024-04-26T23:43:07.527766+0000","last_unstale":"2024-04-26T23:43:59.574638+0000","last_undegraded":"2024-04-26T23:43:59.574638+0000","last_fullsized":"2024-04-26T23:43:59.574638+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:39:23.949964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'33","reported_seq":234,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:44:53.338225+0000","last_change":"2024-04-26T23:43:02.437454+0000","last_active":"2024-04-26T23:44:53.338225+0000","last_peered":"2024-04-26T23:44:53.338225+0000","last_clean":"2024-04-26T23:44:53.338225+0000","last_became_active":"2024-04-26T23:43:02.437142+0000","last_became_peered":"2024-04-26T23:43:02.437142+0000","last_unstale":"2024-04-26T23:44:53.338225+0000","last_undegraded":"2024-04-26T23:44:53.338225+0000","last_fullsized":"2024-04-26T23:44:53.338225+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:06:37.751066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54826,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":175,"num_read_kb":140,"num_write":34,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.571323+0000","last_change":"2024-04-26T23:43:03.287831+0000","last_active":"2024-04-26T23:43:59.571323+0000","last_peered":"2024-04-26T23:43:59.571323+0000","last_clean":"2024-04-26T23:43:59.571323+0000","last_became_active":"2024-04-26T23:43:03.287479+0000","last_became_peered":"2024-04-26T23:43:03.287479+0000","last_unstale":"2024-04-26T23:43:59.571323+0000","last_undegraded":"2024-04-26T23:43:59.571323+0000","last_fullsized":"2024-04-26T23:43:59.571323+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:39:41.726893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.569525+0000","last_change":"2024-04-26T23:43:07.543985+0000","last_active":"2024-04-26T23:43:59.569525+0000","last_peered":"2024-04-26T23:43:59.569525+0000","last_clean":"2024-04-26T23:43:59.569525+0000","last_became_active":"2024-04-26T23:43:07.543327+0000","last_became_peered":"2024-04-26T23:43:07.543327+0000","last_unstale":"2024-04-26T23:43:59.569525+0000","last_undegraded":"2024-04-26T23:43:59.569525+0000","last_fullsized":"2024-04-26T23:43:59.569525+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:24:10.577822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.566396+0000","last_change":"2024-04-26T23:43:02.447627+0000","last_active":"2024-04-26T23:43:59.566396+0000","last_peered":"2024-04-26T23:43:59.566396+0000","last_clean":"2024-04-26T23:43:59.566396+0000","last_became_active":"2024-04-26T23:43:02.447325+0000","last_became_peered":"2024-04-26T23:43:02.447325+0000","last_unstale":"2024-04-26T23:43:59.566396+0000","last_undegraded":"2024-04-26T23:43:59.566396+0000","last_fullsized":"2024-04-26T23:43:59.566396+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:19:03.757202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.869446+0000","last_change":"2024-04-26T23:43:03.313388+0000","last_active":"2024-04-26T23:43:59.869446+0000","last_peered":"2024-04-26T23:43:59.869446+0000","last_clean":"2024-04-26T23:43:59.869446+0000","last_became_active":"2024-04-26T23:43:03.312899+0000","last_became_peered":"2024-04-26T23:43:03.312899+0000","last_unstale":"2024-04-26T23:43:59.869446+0000","last_undegraded":"2024-04-26T23:43:59.869446+0000","last_fullsized":"2024-04-26T23:43:59.869446+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:58:37.126485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.569457+0000","last_change":"2024-04-26T23:43:07.536706+0000","last_active":"2024-04-26T23:43:59.569457+0000","last_peered":"2024-04-26T23:43:59.569457+0000","last_clean":"2024-04-26T23:43:59.569457+0000","last_became_active":"2024-04-26T23:43:07.536316+0000","last_became_peered":"2024-04-26T23:43:07.536316+0000","last_unstale":"2024-04-26T23:43:59.569457+0000","last_undegraded":"2024-04-26T23:43:59.569457+0000","last_fullsized":"2024-04-26T23:43:59.569457+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:36:46.397725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.568997+0000","last_change":"2024-04-26T23:43:02.448647+0000","last_active":"2024-04-26T23:43:59.568997+0000","last_peered":"2024-04-26T23:43:59.568997+0000","last_clean":"2024-04-26T23:43:59.568997+0000","last_became_active":"2024-04-26T23:43:02.448392+0000","last_became_peered":"2024-04-26T23:43:02.448392+0000","last_unstale":"2024-04-26T23:43:59.568997+0000","last_undegraded":"2024-04-26T23:43:59.568997+0000","last_fullsized":"2024-04-26T23:43:59.568997+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:19:01.367361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.571932+0000","last_change":"2024-04-26T23:43:03.302980+0000","last_active":"2024-04-26T23:43:59.571932+0000","last_peered":"2024-04-26T23:43:59.571932+0000","last_clean":"2024-04-26T23:43:59.571932+0000","last_became_active":"2024-04-26T23:43:03.302589+0000","last_became_peered":"2024-04-26T23:43:03.302589+0000","last_unstale":"2024-04-26T23:43:59.571932+0000","last_undegraded":"2024-04-26T23:43:59.571932+0000","last_fullsized":"2024-04-26T23:43:59.571932+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:57:18.641063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.575941+0000","last_change":"2024-04-26T23:43:07.547842+0000","last_active":"2024-04-26T23:43:59.575941+0000","last_peered":"2024-04-26T23:43:59.575941+0000","last_clean":"2024-04-26T23:43:59.575941+0000","last_became_active":"2024-04-26T23:43:07.541774+0000","last_became_peered":"2024-04-26T23:43:07.541774+0000","last_unstale":"2024-04-26T23:43:59.575941+0000","last_undegraded":"2024-04-26T23:43:59.575941+0000","last_fullsized":"2024-04-26T23:43:59.575941+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:25:49.999465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'4","reported_seq":118,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:44:52.375466+0000","last_change":"2024-04-26T23:43:02.437210+0000","last_active":"2024-04-26T23:44:52.375466+0000","last_peered":"2024-04-26T23:44:52.375466+0000","last_clean":"2024-04-26T23:44:52.375466+0000","last_became_active":"2024-04-26T23:43:02.436886+0000","last_became_peered":"2024-04-26T23:43:02.436886+0000","last_unstale":"2024-04-26T23:44:52.375466+0000","last_undegraded":"2024-04-26T23:44:52.375466+0000","last_fullsized":"2024-04-26T23:44:52.375466+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:07:03.663757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":88,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.868608+0000","last_change":"2024-04-26T23:43:59.868608+0000","last_active":"2024-04-26T23:43:59.868608+0000","last_peered":"2024-04-26T23:43:59.868608+0000","last_clean":"2024-04-26T23:43:59.868608+0000","last_became_active":"2024-04-26T23:43:59.868231+0000","last_became_peered":"2024-04-26T23:43:59.868231+0000","last_unstale":"2024-04-26T23:43:59.868608+0000","last_undegraded":"2024-04-26T23:43:59.868608+0000","last_fullsized":"2024-04-26T23:43:59.868608+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:28:42.932733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.869040+0000","last_change":"2024-04-26T23:43:07.545626+0000","last_active":"2024-04-26T23:43:59.869040+0000","last_peered":"2024-04-26T23:43:59.869040+0000","last_clean":"2024-04-26T23:43:59.869040+0000","last_became_active":"2024-04-26T23:43:07.545171+0000","last_became_peered":"2024-04-26T23:43:07.545171+0000","last_unstale":"2024-04-26T23:43:59.869040+0000","last_undegraded":"2024-04-26T23:43:59.869040+0000","last_fullsized":"2024-04-26T23:43:59.869040+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:23:33.829834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.589239+0000","last_change":"2024-04-26T23:43:59.589239+0000","last_active":"2024-04-26T23:43:59.589239+0000","last_peered":"2024-04-26T23:43:59.589239+0000","last_clean":"2024-04-26T23:43:59.589239+0000","last_became_active":"2024-04-26T23:43:59.576239+0000","last_became_peered":"2024-04-26T23:43:59.576239+0000","last_unstale":"2024-04-26T23:43:59.589239+0000","last_undegraded":"2024-04-26T23:43:59.589239+0000","last_fullsized":"2024-04-26T23:43:59.589239+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:28:53.326961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.569408+0000","last_change":"2024-04-26T23:43:03.290567+0000","last_active":"2024-04-26T23:43:59.569408+0000","last_peered":"2024-04-26T23:43:59.569408+0000","last_clean":"2024-04-26T23:43:59.569408+0000","last_became_active":"2024-04-26T23:43:03.287807+0000","last_became_peered":"2024-04-26T23:43:03.287807+0000","last_unstale":"2024-04-26T23:43:59.569408+0000","last_undegraded":"2024-04-26T23:43:59.569408+0000","last_fullsized":"2024-04-26T23:43:59.569408+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:25:24.393978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.568665+0000","last_change":"2024-04-26T23:43:07.535402+0000","last_active":"2024-04-26T23:43:59.568665+0000","last_peered":"2024-04-26T23:43:59.568665+0000","last_clean":"2024-04-26T23:43:59.568665+0000","last_became_active":"2024-04-26T23:43:07.534870+0000","last_became_peered":"2024-04-26T23:43:07.534870+0000","last_unstale":"2024-04-26T23:43:59.568665+0000","last_undegraded":"2024-04-26T23:43:59.568665+0000","last_fullsized":"2024-04-26T23:43:59.568665+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:26:37.703831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.571299+0000","last_change":"2024-04-26T23:43:02.440638+0000","last_active":"2024-04-26T23:43:59.571299+0000","last_peered":"2024-04-26T23:43:59.571299+0000","last_clean":"2024-04-26T23:43:59.571299+0000","last_became_active":"2024-04-26T23:43:02.440294+0000","last_became_peered":"2024-04-26T23:43:02.440294+0000","last_unstale":"2024-04-26T23:43:59.571299+0000","last_undegraded":"2024-04-26T23:43:59.571299+0000","last_fullsized":"2024-04-26T23:43:59.571299+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:21:40.083946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.576561+0000","last_change":"2024-04-26T23:43:59.576561+0000","last_active":"2024-04-26T23:43:59.576561+0000","last_peered":"2024-04-26T23:43:59.576561+0000","last_clean":"2024-04-26T23:43:59.576561+0000","last_became_active":"2024-04-26T23:43:59.574898+0000","last_became_peered":"2024-04-26T23:43:59.574898+0000","last_unstale":"2024-04-26T23:43:59.576561+0000","last_undegraded":"2024-04-26T23:43:59.576561+0000","last_fullsized":"2024-04-26T23:43:59.576561+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:57:04.569103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.572008+0000","last_change":"2024-04-26T23:43:07.855931+0000","last_active":"2024-04-26T23:43:59.572008+0000","last_peered":"2024-04-26T23:43:59.572008+0000","last_clean":"2024-04-26T23:43:59.572008+0000","last_became_active":"2024-04-26T23:43:07.855407+0000","last_became_peered":"2024-04-26T23:43:07.855407+0000","last_unstale":"2024-04-26T23:43:59.572008+0000","last_undegraded":"2024-04-26T23:43:59.572008+0000","last_fullsized":"2024-04-26T23:43:59.572008+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:39:25.707837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.575392+0000","last_change":"2024-04-26T23:43:02.457971+0000","last_active":"2024-04-26T23:43:59.575392+0000","last_peered":"2024-04-26T23:43:59.575392+0000","last_clean":"2024-04-26T23:43:59.575392+0000","last_became_active":"2024-04-26T23:43:02.457627+0000","last_became_peered":"2024-04-26T23:43:02.457627+0000","last_unstale":"2024-04-26T23:43:59.575392+0000","last_undegraded":"2024-04-26T23:43:59.575392+0000","last_fullsized":"2024-04-26T23:43:59.575392+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:25:51.749774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.867642+0000","last_change":"2024-04-26T23:43:03.305569+0000","last_active":"2024-04-26T23:43:59.867642+0000","last_peered":"2024-04-26T23:43:59.867642+0000","last_clean":"2024-04-26T23:43:59.867642+0000","last_became_active":"2024-04-26T23:43:03.305211+0000","last_became_peered":"2024-04-26T23:43:03.305211+0000","last_unstale":"2024-04-26T23:43:59.867642+0000","last_undegraded":"2024-04-26T23:43:59.867642+0000","last_fullsized":"2024-04-26T23:43:59.867642+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:54:55.847680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.868402+0000","last_change":"2024-04-26T23:43:07.535408+0000","last_active":"2024-04-26T23:43:59.868402+0000","last_peered":"2024-04-26T23:43:59.868402+0000","last_clean":"2024-04-26T23:43:59.868402+0000","last_became_active":"2024-04-26T23:43:07.534957+0000","last_became_peered":"2024-04-26T23:43:07.534957+0000","last_unstale":"2024-04-26T23:43:59.868402+0000","last_undegraded":"2024-04-26T23:43:59.868402+0000","last_fullsized":"2024-04-26T23:43:59.868402+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:35:31.103532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.571993+0000","last_change":"2024-04-26T23:43:02.453636+0000","last_active":"2024-04-26T23:43:59.571993+0000","last_peered":"2024-04-26T23:43:59.571993+0000","last_clean":"2024-04-26T23:43:59.571993+0000","last_became_active":"2024-04-26T23:43:02.453310+0000","last_became_peered":"2024-04-26T23:43:02.453310+0000","last_unstale":"2024-04-26T23:43:59.571993+0000","last_undegraded":"2024-04-26T23:43:59.571993+0000","last_fullsized":"2024-04-26T23:43:59.571993+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:42:29.295303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.867445+0000","last_change":"2024-04-26T23:43:03.283184+0000","last_active":"2024-04-26T23:43:59.867445+0000","last_peered":"2024-04-26T23:43:59.867445+0000","last_clean":"2024-04-26T23:43:59.867445+0000","last_became_active":"2024-04-26T23:43:03.282780+0000","last_became_peered":"2024-04-26T23:43:03.282780+0000","last_unstale":"2024-04-26T23:43:59.867445+0000","last_undegraded":"2024-04-26T23:43:59.867445+0000","last_fullsized":"2024-04-26T23:43:59.867445+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:00:13.907101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.868602+0000","last_change":"2024-04-26T23:43:07.859107+0000","last_active":"2024-04-26T23:43:59.868602+0000","last_peered":"2024-04-26T23:43:59.868602+0000","last_clean":"2024-04-26T23:43:59.868602+0000","last_became_active":"2024-04-26T23:43:07.856492+0000","last_became_peered":"2024-04-26T23:43:07.856492+0000","last_unstale":"2024-04-26T23:43:59.868602+0000","last_undegraded":"2024-04-26T23:43:59.868602+0000","last_fullsized":"2024-04-26T23:43:59.868602+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:13:36.570398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.569605+0000","last_change":"2024-04-26T23:43:03.289078+0000","last_active":"2024-04-26T23:43:59.569605+0000","last_peered":"2024-04-26T23:43:59.569605+0000","last_clean":"2024-04-26T23:43:59.569605+0000","last_became_active":"2024-04-26T23:43:03.288729+0000","last_became_peered":"2024-04-26T23:43:03.288729+0000","last_unstale":"2024-04-26T23:43:59.569605+0000","last_undegraded":"2024-04-26T23:43:59.569605+0000","last_fullsized":"2024-04-26T23:43:59.569605+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:32:36.346472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":115,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:44:52.374746+0000","last_change":"2024-04-26T23:43:02.463355+0000","last_active":"2024-04-26T23:44:52.374746+0000","last_peered":"2024-04-26T23:44:52.374746+0000","last_clean":"2024-04-26T23:44:52.374746+0000","last_became_active":"2024-04-26T23:43:02.455832+0000","last_became_peered":"2024-04-26T23:43:02.455832+0000","last_unstale":"2024-04-26T23:44:52.374746+0000","last_undegraded":"2024-04-26T23:44:52.374746+0000","last_fullsized":"2024-04-26T23:44:52.374746+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:33:33.837999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":84,"num_read_kb":84,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.572048+0000","last_change":"2024-04-26T23:43:07.540777+0000","last_active":"2024-04-26T23:43:59.572048+0000","last_peered":"2024-04-26T23:43:59.572048+0000","last_clean":"2024-04-26T23:43:59.572048+0000","last_became_active":"2024-04-26T23:43:07.540389+0000","last_became_peered":"2024-04-26T23:43:07.540389+0000","last_unstale":"2024-04-26T23:43:59.572048+0000","last_undegraded":"2024-04-26T23:43:59.572048+0000","last_fullsized":"2024-04-26T23:43:59.572048+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:57:28.674051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.569836+0000","last_change":"2024-04-26T23:43:03.308391+0000","last_active":"2024-04-26T23:43:59.569836+0000","last_peered":"2024-04-26T23:43:59.569836+0000","last_clean":"2024-04-26T23:43:59.569836+0000","last_became_active":"2024-04-26T23:43:03.306762+0000","last_became_peered":"2024-04-26T23:43:03.306762+0000","last_unstale":"2024-04-26T23:43:59.569836+0000","last_undegraded":"2024-04-26T23:43:59.569836+0000","last_fullsized":"2024-04-26T23:43:59.569836+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:34:01.083036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.565972+0000","last_change":"2024-04-26T23:43:02.448037+0000","last_active":"2024-04-26T23:43:59.565972+0000","last_peered":"2024-04-26T23:43:59.565972+0000","last_clean":"2024-04-26T23:43:59.565972+0000","last_became_active":"2024-04-26T23:43:02.447706+0000","last_became_peered":"2024-04-26T23:43:02.447706+0000","last_unstale":"2024-04-26T23:43:59.565972+0000","last_undegraded":"2024-04-26T23:43:59.565972+0000","last_fullsized":"2024-04-26T23:43:59.565972+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:43:56.295191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":348,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.571505+0000","last_change":"2024-04-26T23:41:42.266448+0000","last_active":"2024-04-26T23:43:59.571505+0000","last_peered":"2024-04-26T23:43:59.571505+0000","last_clean":"2024-04-26T23:43:59.571505+0000","last_became_active":"2024-04-26T23:41:42.265898+0000","last_became_peered":"2024-04-26T23:41:42.265898+0000","last_unstale":"2024-04-26T23:43:59.571505+0000","last_undegraded":"2024-04-26T23:43:59.571505+0000","last_fullsized":"2024-04-26T23:43:59.571505+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:37:44.460409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:37:44.460409+0000","last_clean_scrub_stamp":"2024-04-26T23:37:44.460409+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:40:48.177191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.568743+0000","last_change":"2024-04-26T23:43:07.856089+0000","last_active":"2024-04-26T23:43:59.568743+0000","last_peered":"2024-04-26T23:43:59.568743+0000","last_clean":"2024-04-26T23:43:59.568743+0000","last_became_active":"2024-04-26T23:43:07.855542+0000","last_became_peered":"2024-04-26T23:43:07.855542+0000","last_unstale":"2024-04-26T23:43:59.568743+0000","last_undegraded":"2024-04-26T23:43:59.568743+0000","last_fullsized":"2024-04-26T23:43:59.568743+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:48:22.799242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.868119+0000","last_change":"2024-04-26T23:43:03.301989+0000","last_active":"2024-04-26T23:43:59.868119+0000","last_peered":"2024-04-26T23:43:59.868119+0000","last_clean":"2024-04-26T23:43:59.868119+0000","last_became_active":"2024-04-26T23:43:03.301179+0000","last_became_peered":"2024-04-26T23:43:03.301179+0000","last_unstale":"2024-04-26T23:43:59.868119+0000","last_undegraded":"2024-04-26T23:43:59.868119+0000","last_fullsized":"2024-04-26T23:43:59.868119+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:29:33.012009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.571573+0000","last_change":"2024-04-26T23:43:02.439648+0000","last_active":"2024-04-26T23:43:59.571573+0000","last_peered":"2024-04-26T23:43:59.571573+0000","last_clean":"2024-04-26T23:43:59.571573+0000","last_became_active":"2024-04-26T23:43:02.439340+0000","last_became_peered":"2024-04-26T23:43:02.439340+0000","last_unstale":"2024-04-26T23:43:59.571573+0000","last_undegraded":"2024-04-26T23:43:59.571573+0000","last_fullsized":"2024-04-26T23:43:59.571573+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:10:01.096657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.569055+0000","last_change":"2024-04-26T23:43:07.534317+0000","last_active":"2024-04-26T23:43:59.569055+0000","last_peered":"2024-04-26T23:43:59.569055+0000","last_clean":"2024-04-26T23:43:59.569055+0000","last_became_active":"2024-04-26T23:43:07.533808+0000","last_became_peered":"2024-04-26T23:43:07.533808+0000","last_unstale":"2024-04-26T23:43:59.569055+0000","last_undegraded":"2024-04-26T23:43:59.569055+0000","last_fullsized":"2024-04-26T23:43:59.569055+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:54:29.952089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.571438+0000","last_change":"2024-04-26T23:43:02.457501+0000","last_active":"2024-04-26T23:43:59.571438+0000","last_peered":"2024-04-26T23:43:59.571438+0000","last_clean":"2024-04-26T23:43:59.571438+0000","last_became_active":"2024-04-26T23:43:02.457024+0000","last_became_peered":"2024-04-26T23:43:02.457024+0000","last_unstale":"2024-04-26T23:43:59.571438+0000","last_undegraded":"2024-04-26T23:43:59.571438+0000","last_fullsized":"2024-04-26T23:43:59.571438+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:21:58.542186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.868599+0000","last_change":"2024-04-26T23:43:03.303434+0000","last_active":"2024-04-26T23:43:59.868599+0000","last_peered":"2024-04-26T23:43:59.868599+0000","last_clean":"2024-04-26T23:43:59.868599+0000","last_became_active":"2024-04-26T23:43:03.302455+0000","last_became_peered":"2024-04-26T23:43:03.302455+0000","last_unstale":"2024-04-26T23:43:59.868599+0000","last_undegraded":"2024-04-26T23:43:59.868599+0000","last_fullsized":"2024-04-26T23:43:59.868599+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:57:05.530635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.568792+0000","last_change":"2024-04-26T23:43:07.856932+0000","last_active":"2024-04-26T23:43:59.568792+0000","last_peered":"2024-04-26T23:43:59.568792+0000","last_clean":"2024-04-26T23:43:59.568792+0000","last_became_active":"2024-04-26T23:43:07.855699+0000","last_became_peered":"2024-04-26T23:43:07.855699+0000","last_unstale":"2024-04-26T23:43:59.568792+0000","last_undegraded":"2024-04-26T23:43:59.568792+0000","last_fullsized":"2024-04-26T23:43:59.568792+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:20:28.994301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.576206+0000","last_change":"2024-04-26T23:43:02.449852+0000","last_active":"2024-04-26T23:43:59.576206+0000","last_peered":"2024-04-26T23:43:59.576206+0000","last_clean":"2024-04-26T23:43:59.576206+0000","last_became_active":"2024-04-26T23:43:02.449422+0000","last_became_peered":"2024-04-26T23:43:02.449422+0000","last_unstale":"2024-04-26T23:43:59.576206+0000","last_undegraded":"2024-04-26T23:43:59.576206+0000","last_fullsized":"2024-04-26T23:43:59.576206+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:07:25.453872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.570144+0000","last_change":"2024-04-26T23:43:03.304733+0000","last_active":"2024-04-26T23:43:59.570144+0000","last_peered":"2024-04-26T23:43:59.570144+0000","last_clean":"2024-04-26T23:43:59.570144+0000","last_became_active":"2024-04-26T23:43:03.304383+0000","last_became_peered":"2024-04-26T23:43:03.304383+0000","last_unstale":"2024-04-26T23:43:59.570144+0000","last_undegraded":"2024-04-26T23:43:59.570144+0000","last_fullsized":"2024-04-26T23:43:59.570144+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:49:28.679255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.566122+0000","last_change":"2024-04-26T23:43:07.525310+0000","last_active":"2024-04-26T23:43:59.566122+0000","last_peered":"2024-04-26T23:43:59.566122+0000","last_clean":"2024-04-26T23:43:59.566122+0000","last_became_active":"2024-04-26T23:43:07.524502+0000","last_became_peered":"2024-04-26T23:43:07.524502+0000","last_unstale":"2024-04-26T23:43:59.566122+0000","last_undegraded":"2024-04-26T23:43:59.566122+0000","last_fullsized":"2024-04-26T23:43:59.566122+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:07:45.927837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.566013+0000","last_change":"2024-04-26T23:43:02.446238+0000","last_active":"2024-04-26T23:43:59.566013+0000","last_peered":"2024-04-26T23:43:59.566013+0000","last_clean":"2024-04-26T23:43:59.566013+0000","last_became_active":"2024-04-26T23:43:02.445959+0000","last_became_peered":"2024-04-26T23:43:02.445959+0000","last_unstale":"2024-04-26T23:43:59.566013+0000","last_undegraded":"2024-04-26T23:43:59.566013+0000","last_fullsized":"2024-04-26T23:43:59.566013+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:25:10.007481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.567831+0000","last_change":"2024-04-26T23:43:03.306035+0000","last_active":"2024-04-26T23:43:59.567831+0000","last_peered":"2024-04-26T23:43:59.567831+0000","last_clean":"2024-04-26T23:43:59.567831+0000","last_became_active":"2024-04-26T23:43:03.305604+0000","last_became_peered":"2024-04-26T23:43:03.305604+0000","last_unstale":"2024-04-26T23:43:59.567831+0000","last_undegraded":"2024-04-26T23:43:59.567831+0000","last_fullsized":"2024-04-26T23:43:59.567831+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:02:38.567754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.565884+0000","last_change":"2024-04-26T23:43:07.855808+0000","last_active":"2024-04-26T23:43:59.565884+0000","last_peered":"2024-04-26T23:43:59.565884+0000","last_clean":"2024-04-26T23:43:59.565884+0000","last_became_active":"2024-04-26T23:43:07.855266+0000","last_became_peered":"2024-04-26T23:43:07.855266+0000","last_unstale":"2024-04-26T23:43:59.565884+0000","last_undegraded":"2024-04-26T23:43:59.565884+0000","last_fullsized":"2024-04-26T23:43:59.565884+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:06:03.016504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.571965+0000","last_change":"2024-04-26T23:43:02.443464+0000","last_active":"2024-04-26T23:43:59.571965+0000","last_peered":"2024-04-26T23:43:59.571965+0000","last_clean":"2024-04-26T23:43:59.571965+0000","last_became_active":"2024-04-26T23:43:02.443170+0000","last_became_peered":"2024-04-26T23:43:02.443170+0000","last_unstale":"2024-04-26T23:43:59.571965+0000","last_undegraded":"2024-04-26T23:43:59.571965+0000","last_fullsized":"2024-04-26T23:43:59.571965+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:45:51.431082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.572506+0000","last_change":"2024-04-26T23:43:03.282714+0000","last_active":"2024-04-26T23:43:59.572506+0000","last_peered":"2024-04-26T23:43:59.572506+0000","last_clean":"2024-04-26T23:43:59.572506+0000","last_became_active":"2024-04-26T23:43:03.282353+0000","last_became_peered":"2024-04-26T23:43:03.282353+0000","last_unstale":"2024-04-26T23:43:59.572506+0000","last_undegraded":"2024-04-26T23:43:59.572506+0000","last_fullsized":"2024-04-26T23:43:59.572506+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:18:24.082946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.576174+0000","last_change":"2024-04-26T23:43:07.543301+0000","last_active":"2024-04-26T23:43:59.576174+0000","last_peered":"2024-04-26T23:43:59.576174+0000","last_clean":"2024-04-26T23:43:59.576174+0000","last_became_active":"2024-04-26T23:43:07.540434+0000","last_became_peered":"2024-04-26T23:43:07.540434+0000","last_unstale":"2024-04-26T23:43:59.576174+0000","last_undegraded":"2024-04-26T23:43:59.576174+0000","last_fullsized":"2024-04-26T23:43:59.576174+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:15:18.542595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.568745+0000","last_change":"2024-04-26T23:43:02.451391+0000","last_active":"2024-04-26T23:43:59.568745+0000","last_peered":"2024-04-26T23:43:59.568745+0000","last_clean":"2024-04-26T23:43:59.568745+0000","last_became_active":"2024-04-26T23:43:02.450735+0000","last_became_peered":"2024-04-26T23:43:02.450735+0000","last_unstale":"2024-04-26T23:43:59.568745+0000","last_undegraded":"2024-04-26T23:43:59.568745+0000","last_fullsized":"2024-04-26T23:43:59.568745+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:57:48.658671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.569744+0000","last_change":"2024-04-26T23:43:03.301858+0000","last_active":"2024-04-26T23:43:59.569744+0000","last_peered":"2024-04-26T23:43:59.569744+0000","last_clean":"2024-04-26T23:43:59.569744+0000","last_became_active":"2024-04-26T23:43:03.301474+0000","last_became_peered":"2024-04-26T23:43:03.301474+0000","last_unstale":"2024-04-26T23:43:59.569744+0000","last_undegraded":"2024-04-26T23:43:59.569744+0000","last_fullsized":"2024-04-26T23:43:59.569744+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:24:56.386127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.566096+0000","last_change":"2024-04-26T23:43:07.544512+0000","last_active":"2024-04-26T23:43:59.566096+0000","last_peered":"2024-04-26T23:43:59.566096+0000","last_clean":"2024-04-26T23:43:59.566096+0000","last_became_active":"2024-04-26T23:43:07.544029+0000","last_became_peered":"2024-04-26T23:43:07.544029+0000","last_unstale":"2024-04-26T23:43:59.566096+0000","last_undegraded":"2024-04-26T23:43:59.566096+0000","last_fullsized":"2024-04-26T23:43:59.566096+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:38:08.149123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":15,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.580107+0000","last_change":"2024-04-26T23:43:59.579721+0000","last_active":"2024-04-26T23:43:59.580107+0000","last_peered":"2024-04-26T23:43:59.580107+0000","last_clean":"2024-04-26T23:43:59.580107+0000","last_became_active":"2024-04-26T23:43:59.576675+0000","last_became_peered":"2024-04-26T23:43:59.576675+0000","last_unstale":"2024-04-26T23:43:59.580107+0000","last_undegraded":"2024-04-26T23:43:59.580107+0000","last_fullsized":"2024-04-26T23:43:59.580107+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:59:30.124137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.571481+0000","last_change":"2024-04-26T23:43:03.307895+0000","last_active":"2024-04-26T23:43:59.571481+0000","last_peered":"2024-04-26T23:43:59.571481+0000","last_clean":"2024-04-26T23:43:59.571481+0000","last_became_active":"2024-04-26T23:43:03.307478+0000","last_became_peered":"2024-04-26T23:43:03.307478+0000","last_unstale":"2024-04-26T23:43:59.571481+0000","last_undegraded":"2024-04-26T23:43:59.571481+0000","last_fullsized":"2024-04-26T23:43:59.571481+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:57:14.356234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.565736+0000","last_change":"2024-04-26T23:43:07.855803+0000","last_active":"2024-04-26T23:43:59.565736+0000","last_peered":"2024-04-26T23:43:59.565736+0000","last_clean":"2024-04-26T23:43:59.565736+0000","last_became_active":"2024-04-26T23:43:07.855200+0000","last_became_peered":"2024-04-26T23:43:07.855200+0000","last_unstale":"2024-04-26T23:43:59.565736+0000","last_undegraded":"2024-04-26T23:43:59.565736+0000","last_fullsized":"2024-04-26T23:43:59.565736+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:02:46.223330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.571815+0000","last_change":"2024-04-26T23:43:02.437513+0000","last_active":"2024-04-26T23:43:59.571815+0000","last_peered":"2024-04-26T23:43:59.571815+0000","last_clean":"2024-04-26T23:43:59.571815+0000","last_became_active":"2024-04-26T23:43:02.437153+0000","last_became_peered":"2024-04-26T23:43:02.437153+0000","last_unstale":"2024-04-26T23:43:59.571815+0000","last_undegraded":"2024-04-26T23:43:59.571815+0000","last_fullsized":"2024-04-26T23:43:59.571815+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:11:45.477579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.572239+0000","last_change":"2024-04-26T23:43:03.297835+0000","last_active":"2024-04-26T23:43:59.572239+0000","last_peered":"2024-04-26T23:43:59.572239+0000","last_clean":"2024-04-26T23:43:59.572239+0000","last_became_active":"2024-04-26T23:43:03.297481+0000","last_became_peered":"2024-04-26T23:43:03.297481+0000","last_unstale":"2024-04-26T23:43:59.572239+0000","last_undegraded":"2024-04-26T23:43:59.572239+0000","last_fullsized":"2024-04-26T23:43:59.572239+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:52:54.757461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.571776+0000","last_change":"2024-04-26T23:43:07.536849+0000","last_active":"2024-04-26T23:43:59.571776+0000","last_peered":"2024-04-26T23:43:59.571776+0000","last_clean":"2024-04-26T23:43:59.571776+0000","last_became_active":"2024-04-26T23:43:07.536214+0000","last_became_peered":"2024-04-26T23:43:07.536214+0000","last_unstale":"2024-04-26T23:43:59.571776+0000","last_undegraded":"2024-04-26T23:43:59.571776+0000","last_fullsized":"2024-04-26T23:43:59.571776+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:33:16.782368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.566803+0000","last_change":"2024-04-26T23:43:02.464181+0000","last_active":"2024-04-26T23:43:59.566803+0000","last_peered":"2024-04-26T23:43:59.566803+0000","last_clean":"2024-04-26T23:43:59.566803+0000","last_became_active":"2024-04-26T23:43:02.463830+0000","last_became_peered":"2024-04-26T23:43:02.463830+0000","last_unstale":"2024-04-26T23:43:59.566803+0000","last_undegraded":"2024-04-26T23:43:59.566803+0000","last_fullsized":"2024-04-26T23:43:59.566803+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:14:52.831302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.572417+0000","last_change":"2024-04-26T23:43:03.298497+0000","last_active":"2024-04-26T23:43:59.572417+0000","last_peered":"2024-04-26T23:43:59.572417+0000","last_clean":"2024-04-26T23:43:59.572417+0000","last_became_active":"2024-04-26T23:43:03.298194+0000","last_became_peered":"2024-04-26T23:43:03.298194+0000","last_unstale":"2024-04-26T23:43:59.572417+0000","last_undegraded":"2024-04-26T23:43:59.572417+0000","last_fullsized":"2024-04-26T23:43:59.572417+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:39:19.007294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.567343+0000","last_change":"2024-04-26T23:43:07.527962+0000","last_active":"2024-04-26T23:43:59.567343+0000","last_peered":"2024-04-26T23:43:59.567343+0000","last_clean":"2024-04-26T23:43:59.567343+0000","last_became_active":"2024-04-26T23:43:07.527386+0000","last_became_peered":"2024-04-26T23:43:07.527386+0000","last_unstale":"2024-04-26T23:43:59.567343+0000","last_undegraded":"2024-04-26T23:43:59.567343+0000","last_fullsized":"2024-04-26T23:43:59.567343+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:57:21.711670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.569154+0000","last_change":"2024-04-26T23:43:02.453988+0000","last_active":"2024-04-26T23:43:59.569154+0000","last_peered":"2024-04-26T23:43:59.569154+0000","last_clean":"2024-04-26T23:43:59.569154+0000","last_became_active":"2024-04-26T23:43:02.453654+0000","last_became_peered":"2024-04-26T23:43:02.453654+0000","last_unstale":"2024-04-26T23:43:59.569154+0000","last_undegraded":"2024-04-26T23:43:59.569154+0000","last_fullsized":"2024-04-26T23:43:59.569154+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:46:08.718705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.571441+0000","last_change":"2024-04-26T23:43:03.305861+0000","last_active":"2024-04-26T23:43:59.571441+0000","last_peered":"2024-04-26T23:43:59.571441+0000","last_clean":"2024-04-26T23:43:59.571441+0000","last_became_active":"2024-04-26T23:43:03.305301+0000","last_became_peered":"2024-04-26T23:43:03.305301+0000","last_unstale":"2024-04-26T23:43:59.571441+0000","last_undegraded":"2024-04-26T23:43:59.571441+0000","last_fullsized":"2024-04-26T23:43:59.571441+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:08:24.174443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.568906+0000","last_change":"2024-04-26T23:43:07.529914+0000","last_active":"2024-04-26T23:43:59.568906+0000","last_peered":"2024-04-26T23:43:59.568906+0000","last_clean":"2024-04-26T23:43:59.568906+0000","last_became_active":"2024-04-26T23:43:07.529509+0000","last_became_peered":"2024-04-26T23:43:07.529509+0000","last_unstale":"2024-04-26T23:43:59.568906+0000","last_undegraded":"2024-04-26T23:43:59.568906+0000","last_fullsized":"2024-04-26T23:43:59.568906+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:50:23.293672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.868745+0000","last_change":"2024-04-26T23:43:02.452096+0000","last_active":"2024-04-26T23:43:59.868745+0000","last_peered":"2024-04-26T23:43:59.868745+0000","last_clean":"2024-04-26T23:43:59.868745+0000","last_became_active":"2024-04-26T23:43:02.450039+0000","last_became_peered":"2024-04-26T23:43:02.450039+0000","last_unstale":"2024-04-26T23:43:59.868745+0000","last_undegraded":"2024-04-26T23:43:59.868745+0000","last_fullsized":"2024-04-26T23:43:59.868745+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:47:03.062575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.572594+0000","last_change":"2024-04-26T23:43:03.294790+0000","last_active":"2024-04-26T23:43:59.572594+0000","last_peered":"2024-04-26T23:43:59.572594+0000","last_clean":"2024-04-26T23:43:59.572594+0000","last_became_active":"2024-04-26T23:43:03.294460+0000","last_became_peered":"2024-04-26T23:43:03.294460+0000","last_unstale":"2024-04-26T23:43:59.572594+0000","last_undegraded":"2024-04-26T23:43:59.572594+0000","last_fullsized":"2024-04-26T23:43:59.572594+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:57:40.879315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.569838+0000","last_change":"2024-04-26T23:43:03.284344+0000","last_active":"2024-04-26T23:43:59.569838+0000","last_peered":"2024-04-26T23:43:59.569838+0000","last_clean":"2024-04-26T23:43:59.569838+0000","last_became_active":"2024-04-26T23:43:03.283954+0000","last_became_peered":"2024-04-26T23:43:03.283954+0000","last_unstale":"2024-04-26T23:43:59.569838+0000","last_undegraded":"2024-04-26T23:43:59.569838+0000","last_fullsized":"2024-04-26T23:43:59.569838+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:12:56.516067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.574959+0000","last_change":"2024-04-26T23:43:59.574959+0000","last_active":"2024-04-26T23:43:59.574959+0000","last_peered":"2024-04-26T23:43:59.574959+0000","last_clean":"2024-04-26T23:43:59.574959+0000","last_became_active":"2024-04-26T23:43:59.574588+0000","last_became_peered":"2024-04-26T23:43:59.574588+0000","last_unstale":"2024-04-26T23:43:59.574959+0000","last_undegraded":"2024-04-26T23:43:59.574959+0000","last_fullsized":"2024-04-26T23:43:59.574959+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:52:56.689439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.574946+0000","last_change":"2024-04-26T23:43:59.574946+0000","last_active":"2024-04-26T23:43:59.574946+0000","last_peered":"2024-04-26T23:43:59.574946+0000","last_clean":"2024-04-26T23:43:59.574946+0000","last_became_active":"2024-04-26T23:43:59.574238+0000","last_became_peered":"2024-04-26T23:43:59.574238+0000","last_unstale":"2024-04-26T23:43:59.574946+0000","last_undegraded":"2024-04-26T23:43:59.574946+0000","last_fullsized":"2024-04-26T23:43:59.574946+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:58:41.619868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,0],"acting":[2,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.567529+0000","last_change":"2024-04-26T23:43:03.304580+0000","last_active":"2024-04-26T23:43:59.567529+0000","last_peered":"2024-04-26T23:43:59.567529+0000","last_clean":"2024-04-26T23:43:59.567529+0000","last_became_active":"2024-04-26T23:43:03.303128+0000","last_became_peered":"2024-04-26T23:43:03.303128+0000","last_unstale":"2024-04-26T23:43:59.567529+0000","last_undegraded":"2024-04-26T23:43:59.567529+0000","last_fullsized":"2024-04-26T23:43:59.567529+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:11:36.868298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.868705+0000","last_change":"2024-04-26T23:43:03.306441+0000","last_active":"2024-04-26T23:43:59.868705+0000","last_peered":"2024-04-26T23:43:59.868705+0000","last_clean":"2024-04-26T23:43:59.868705+0000","last_became_active":"2024-04-26T23:43:03.305209+0000","last_became_peered":"2024-04-26T23:43:03.305209+0000","last_unstale":"2024-04-26T23:43:59.868705+0000","last_undegraded":"2024-04-26T23:43:59.868705+0000","last_fullsized":"2024-04-26T23:43:59.868705+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:41:39.757001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.569953+0000","last_change":"2024-04-26T23:43:03.284291+0000","last_active":"2024-04-26T23:43:59.569953+0000","last_peered":"2024-04-26T23:43:59.569953+0000","last_clean":"2024-04-26T23:43:59.569953+0000","last_became_active":"2024-04-26T23:43:03.284006+0000","last_became_peered":"2024-04-26T23:43:03.284006+0000","last_unstale":"2024-04-26T23:43:59.569953+0000","last_undegraded":"2024-04-26T23:43:59.569953+0000","last_fullsized":"2024-04-26T23:43:59.569953+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:24:01.386322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.567715+0000","last_change":"2024-04-26T23:43:03.286478+0000","last_active":"2024-04-26T23:43:59.567715+0000","last_peered":"2024-04-26T23:43:59.567715+0000","last_clean":"2024-04-26T23:43:59.567715+0000","last_became_active":"2024-04-26T23:43:03.286115+0000","last_became_peered":"2024-04-26T23:43:03.286115+0000","last_unstale":"2024-04-26T23:43:59.567715+0000","last_undegraded":"2024-04-26T23:43:59.567715+0000","last_fullsized":"2024-04-26T23:43:59.567715+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:34:57.534124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.872042+0000","last_change":"2024-04-26T23:43:59.872042+0000","last_active":"2024-04-26T23:43:59.872042+0000","last_peered":"2024-04-26T23:43:59.872042+0000","last_clean":"2024-04-26T23:43:59.872042+0000","last_became_active":"2024-04-26T23:43:59.870288+0000","last_became_peered":"2024-04-26T23:43:59.870288+0000","last_unstale":"2024-04-26T23:43:59.872042+0000","last_undegraded":"2024-04-26T23:43:59.872042+0000","last_fullsized":"2024-04-26T23:43:59.872042+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:16:38.518370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.569276+0000","last_change":"2024-04-26T23:43:03.300602+0000","last_active":"2024-04-26T23:43:59.569276+0000","last_peered":"2024-04-26T23:43:59.569276+0000","last_clean":"2024-04-26T23:43:59.569276+0000","last_became_active":"2024-04-26T23:43:03.300225+0000","last_became_peered":"2024-04-26T23:43:03.300225+0000","last_unstale":"2024-04-26T23:43:59.569276+0000","last_undegraded":"2024-04-26T23:43:59.569276+0000","last_fullsized":"2024-04-26T23:43:59.569276+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:52:36.638153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.569164+0000","last_change":"2024-04-26T23:43:03.305631+0000","last_active":"2024-04-26T23:43:59.569164+0000","last_peered":"2024-04-26T23:43:59.569164+0000","last_clean":"2024-04-26T23:43:59.569164+0000","last_became_active":"2024-04-26T23:43:03.300193+0000","last_became_peered":"2024-04-26T23:43:03.300193+0000","last_unstale":"2024-04-26T23:43:59.569164+0000","last_undegraded":"2024-04-26T23:43:59.569164+0000","last_fullsized":"2024-04-26T23:43:59.569164+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:27:08.355753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.572761+0000","last_change":"2024-04-26T23:43:03.309514+0000","last_active":"2024-04-26T23:43:59.572761+0000","last_peered":"2024-04-26T23:43:59.572761+0000","last_clean":"2024-04-26T23:43:59.572761+0000","last_became_active":"2024-04-26T23:43:03.309085+0000","last_became_peered":"2024-04-26T23:43:03.309085+0000","last_unstale":"2024-04-26T23:43:59.572761+0000","last_undegraded":"2024-04-26T23:43:59.572761+0000","last_fullsized":"2024-04-26T23:43:59.572761+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:08:45.649513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.869023+0000","last_change":"2024-04-26T23:43:03.303553+0000","last_active":"2024-04-26T23:43:59.869023+0000","last_peered":"2024-04-26T23:43:59.869023+0000","last_clean":"2024-04-26T23:43:59.869023+0000","last_became_active":"2024-04-26T23:43:03.302979+0000","last_became_peered":"2024-04-26T23:43:03.302979+0000","last_unstale":"2024-04-26T23:43:59.869023+0000","last_undegraded":"2024-04-26T23:43:59.869023+0000","last_fullsized":"2024-04-26T23:43:59.869023+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:44:25.809138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.569584+0000","last_change":"2024-04-26T23:43:03.288433+0000","last_active":"2024-04-26T23:43:59.569584+0000","last_peered":"2024-04-26T23:43:59.569584+0000","last_clean":"2024-04-26T23:43:59.569584+0000","last_became_active":"2024-04-26T23:43:03.288125+0000","last_became_peered":"2024-04-26T23:43:03.288125+0000","last_unstale":"2024-04-26T23:43:59.569584+0000","last_undegraded":"2024-04-26T23:43:59.569584+0000","last_fullsized":"2024-04-26T23:43:59.569584+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:53:21.673755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":282161,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":59218,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":531,"num_read_kb":328,"num_write":161,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":177654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140,"ondisk_log_size":140,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383718,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":995328,"data_stored":500831,"data_compressed":20573,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645354,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32288,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712864,"statfs":{"total":95995035648,"available":95961972736,"internally_reserved":0,"allocated":1134592,"data_stored":1143316,"data_compressed":56460,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808880,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1015808,"data_stored":505163,"data_compressed":20573,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070518,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1011712,"data_stored":500987,"data_compressed":20573,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332156,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32220,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712932,"statfs":{"total":95995035648,"available":95962042368,"internally_reserved":0,"allocated":1060864,"data_stored":1097287,"data_compressed":56460,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495681,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":991232,"data_stored":496243,"data_compressed":20573,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1052672,"data_stored":550979,"data_compressed":20573,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738466,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1015808,"data_stored":497631,"data_compressed":20573,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411417,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":552608,"data_compressed":20573,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182481,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1003520,"data_stored":496437,"data_compressed":20573,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920845,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1011712,"data_stored":501396,"data_compressed":20573,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757320,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32220,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712932,"statfs":{"total":95995035648,"available":95962042368,"internally_reserved":0,"allocated":1069056,"data_stored":1092484,"data_compressed":56460,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34242,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34242,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34242,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35887,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35887,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35887,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35887,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35887,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":54826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":56455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":56795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T23:44:58.773 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T23:44:59.212 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:44:59.709 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:59 smithi072 ceph-mon[28184]: from='client.25024 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T23:44:59.709 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:59 smithi072 ceph-mon[28184]: pgmap v381: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:44:59.709 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:59 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-04-26T23:44:59.710 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:59 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]: dispatch 2024-04-26T23:44:59.710 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:59 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]: dispatch 2024-04-26T23:44:59.710 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:59 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [5, 8]}]: dispatch 2024-04-26T23:44:59.710 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:44:59 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]: dispatch 2024-04-26T23:44:59.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:59 smithi136 ceph-mon[33050]: from='client.25024 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T23:44:59.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:59 smithi136 ceph-mon[33050]: pgmap v381: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:44:59.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:59 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-04-26T23:44:59.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:59 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]: dispatch 2024-04-26T23:44:59.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:59 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]: dispatch 2024-04-26T23:44:59.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:59 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [5, 8]}]: dispatch 2024-04-26T23:44:59.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:44:59 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]: dispatch 2024-04-26T23:44:59.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:59 smithi076 ceph-mon[32774]: from='client.25024 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T23:44:59.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:59 smithi076 ceph-mon[32774]: pgmap v381: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:44:59.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:59 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-04-26T23:44:59.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:59 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]: dispatch 2024-04-26T23:44:59.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:59 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]: dispatch 2024-04-26T23:44:59.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:59 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [5, 8]}]: dispatch 2024-04-26T23:44:59.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:44:59 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]: dispatch 2024-04-26T23:45:00.919 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:00 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-04-26T23:45:00.919 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:00 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]': finished 2024-04-26T23:45:00.919 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:00 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]': finished 2024-04-26T23:45:00.919 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:00 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [5, 8]}]': finished 2024-04-26T23:45:00.920 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:00 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]': finished 2024-04-26T23:45:00.920 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:00 smithi072 ceph-mon[28184]: osdmap e92: 12 total, 12 up, 12 in 2024-04-26T23:45:00.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:00 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-04-26T23:45:00.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:00 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]': finished 2024-04-26T23:45:00.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:00 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]': finished 2024-04-26T23:45:00.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:00 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [5, 8]}]': finished 2024-04-26T23:45:00.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:00 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]': finished 2024-04-26T23:45:00.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:00 smithi076 ceph-mon[32774]: osdmap e92: 12 total, 12 up, 12 in 2024-04-26T23:45:01.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:00 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-04-26T23:45:01.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:00 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]': finished 2024-04-26T23:45:01.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:00 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]': finished 2024-04-26T23:45:01.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:00 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [5, 8]}]': finished 2024-04-26T23:45:01.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:00 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]': finished 2024-04-26T23:45:01.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:00 smithi136 ceph-mon[33050]: osdmap e92: 12 total, 12 up, 12 in 2024-04-26T23:45:01.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.434+0000 7f9248a00700 1 -- 172.21.15.72:0/3679775639 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f92441094c0 msgr2=0x7f92441098a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:01.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.434+0000 7f9248a00700 1 --2- 172.21.15.72:0/3679775639 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f92441094c0 0x7f92441098a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f92340098f0 tx=0x7f9234009c00 comp rx=0 tx=0).stop 2024-04-26T23:45:01.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.434+0000 7f9248a00700 1 -- 172.21.15.72:0/3679775639 shutdown_connections 2024-04-26T23:45:01.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.434+0000 7f9248a00700 1 --2- 172.21.15.72:0/3679775639 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9244103e30 0x7f9244110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:01.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.434+0000 7f9248a00700 1 --2- 172.21.15.72:0/3679775639 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f92441094c0 0x7f92441098a0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:01.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.434+0000 7f9248a00700 1 --2- 172.21.15.72:0/3679775639 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9244103470 0x7f92441038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:01.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.434+0000 7f9248a00700 1 -- 172.21.15.72:0/3679775639 >> 172.21.15.72:0/3679775639 conn(0x7f924407a580 msgr2=0x7f924407a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:01.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.434+0000 7f9248a00700 1 -- 172.21.15.72:0/3679775639 shutdown_connections 2024-04-26T23:45:01.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.434+0000 7f9248a00700 1 -- 172.21.15.72:0/3679775639 wait complete. 2024-04-26T23:45:01.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.435+0000 7f9248a00700 1 Processor -- start 2024-04-26T23:45:01.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.435+0000 7f9248a00700 1 -- start start 2024-04-26T23:45:01.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.436+0000 7f9248a00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9244103470 0x7f924419bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:01.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.436+0000 7f9248a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9244103e30 0x7f924419c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:01.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.436+0000 7f9248a00700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f92441094c0 0x7f92441a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:01.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.436+0000 7f9248a00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f92441135f0 con 0x7f9244103470 2024-04-26T23:45:01.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.436+0000 7f9248a00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f9244113470 con 0x7f9244103e30 2024-04-26T23:45:01.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.436+0000 7f9248a00700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f9244113770 con 0x7f92441094c0 2024-04-26T23:45:01.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.436+0000 7f9241e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9244103470 0x7f924419bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:01.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.436+0000 7f9239400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9244103e30 0x7f924419c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:01.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.436+0000 7f9241e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9244103470 0x7f924419bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56122/0 (socket says 172.21.15.72:56122) 2024-04-26T23:45:01.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.436+0000 7f9241e00700 1 -- 172.21.15.72:0/48945506 learned_addr learned my addr 172.21.15.72:0/48945506 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:45:01.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.436+0000 7f9242800700 1 --2- 172.21.15.72:0/48945506 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f92441094c0 0x7f92441a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:01.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.436+0000 7f9239400700 1 -- 172.21.15.72:0/48945506 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f92441094c0 msgr2=0x7f92441a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:01.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.436+0000 7f9239400700 1 --2- 172.21.15.72:0/48945506 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f92441094c0 0x7f92441a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:01.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.436+0000 7f9239400700 1 -- 172.21.15.72:0/48945506 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9244103470 msgr2=0x7f924419bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:01.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.436+0000 7f9239400700 1 --2- 172.21.15.72:0/48945506 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9244103470 0x7f924419bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:01.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.437+0000 7f9239400700 1 -- 172.21.15.72:0/48945506 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92441a0d80 con 0x7f9244103e30 2024-04-26T23:45:01.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.437+0000 7f9239400700 1 --2- 172.21.15.72:0/48945506 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9244103e30 0x7f924419c2b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f922c00eb90 tx=0x7f922c00eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:01.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.437+0000 7f923b400700 1 -- 172.21.15.72:0/48945506 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f922c00ccf0 con 0x7f9244103e30 2024-04-26T23:45:01.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.437+0000 7f923b400700 1 -- 172.21.15.72:0/48945506 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f922c0044d0 con 0x7f9244103e30 2024-04-26T23:45:01.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.437+0000 7f923b400700 1 -- 172.21.15.72:0/48945506 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f922c010430 con 0x7f9244103e30 2024-04-26T23:45:01.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.437+0000 7f9248a00700 1 -- 172.21.15.72:0/48945506 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92441a1070 con 0x7f9244103e30 2024-04-26T23:45:01.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.437+0000 7f9248a00700 1 -- 172.21.15.72:0/48945506 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92441a88f0 con 0x7f9244103e30 2024-04-26T23:45:01.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.438+0000 7f9248a00700 1 -- 172.21.15.72:0/48945506 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f924406a110 con 0x7f9244103e30 2024-04-26T23:45:01.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.441+0000 7f923b400700 1 -- 172.21.15.72:0/48945506 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f922c003710 con 0x7f9244103e30 2024-04-26T23:45:01.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.442+0000 7f923b400700 1 --2- 172.21.15.72:0/48945506 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f92280a3d50 0x7f92280a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:01.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.442+0000 7f923b400700 1 -- 172.21.15.72:0/48945506 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f922c0d0d80 con 0x7f9244103e30 2024-04-26T23:45:01.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.442+0000 7f9241e00700 1 --2- 172.21.15.72:0/48945506 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f92280a3d50 0x7f92280a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:01.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.443+0000 7f9241e00700 1 --2- 172.21.15.72:0/48945506 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f92280a3d50 0x7f92280a6210 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9234009680 tx=0x7f9234019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:01.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.444+0000 7f923b400700 1 -- 172.21.15.72:0/48945506 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f922c01f340 con 0x7f9244103e30 2024-04-26T23:45:01.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.667+0000 7f9248a00700 1 -- 172.21.15.72:0/48945506 --> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9244063860 con 0x7f92280a3d50 2024-04-26T23:45:01.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.696+0000 7f923b400700 1 -- 172.21.15.72:0/48945506 <== mgr.14152 v2:172.21.15.72:6800/4281177709 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340591 (secure 0 0 0) 0x7f923407a280 con 0x7f92280a3d50 2024-04-26T23:45:01.702 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:45:01.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.704+0000 7f9248a00700 1 -- 172.21.15.72:0/48945506 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f92280a3d50 msgr2=0x7f92280a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:01.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.704+0000 7f9248a00700 1 --2- 172.21.15.72:0/48945506 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f92280a3d50 0x7f92280a6210 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9234009680 tx=0x7f9234019040 comp rx=0 tx=0).stop 2024-04-26T23:45:01.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.704+0000 7f9248a00700 1 -- 172.21.15.72:0/48945506 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9244103e30 msgr2=0x7f924419c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:01.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.704+0000 7f9248a00700 1 --2- 172.21.15.72:0/48945506 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9244103e30 0x7f924419c2b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f922c00eb90 tx=0x7f922c00eea0 comp rx=0 tx=0).stop 2024-04-26T23:45:01.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.704+0000 7f9248a00700 1 -- 172.21.15.72:0/48945506 shutdown_connections 2024-04-26T23:45:01.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.704+0000 7f9248a00700 1 --2- 172.21.15.72:0/48945506 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f92280a3d50 0x7f92280a6210 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:01.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.704+0000 7f9248a00700 1 --2- 172.21.15.72:0/48945506 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f92441094c0 0x7f92441a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:01.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.704+0000 7f9248a00700 1 --2- 172.21.15.72:0/48945506 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9244103470 0x7f924419bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:01.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.704+0000 7f9248a00700 1 --2- 172.21.15.72:0/48945506 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9244103e30 0x7f924419c2b0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:01.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.704+0000 7f9248a00700 1 -- 172.21.15.72:0/48945506 >> 172.21.15.72:0/48945506 conn(0x7f924407a580 msgr2=0x7f924410e950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:01.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.705+0000 7f9248a00700 1 -- 172.21.15.72:0/48945506 shutdown_connections 2024-04-26T23:45:01.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:01.705+0000 7f9248a00700 1 -- 172.21.15.72:0/48945506 wait complete. 2024-04-26T23:45:01.711 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-04-26T23:45:01.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:01 smithi076 ceph-mon[32774]: pgmap v383: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-26T23:45:01.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:01 smithi076 ceph-mon[32774]: osdmap e93: 12 total, 12 up, 12 in 2024-04-26T23:45:02.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:01 smithi072 ceph-mon[28184]: pgmap v383: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-26T23:45:02.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:01 smithi072 ceph-mon[28184]: osdmap e93: 12 total, 12 up, 12 in 2024-04-26T23:45:02.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:01 smithi136 ceph-mon[33050]: pgmap v383: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-26T23:45:02.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:01 smithi136 ceph-mon[33050]: osdmap e93: 12 total, 12 up, 12 in 2024-04-26T23:45:02.554 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":384,"stamp":"2024-04-26T23:45:00.536684+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":649820,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":686,"num_read_kb":451,"num_write":344,"num_write_kb":3049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":1180848,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":279,"ondisk_log_size":279,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":420,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":357208,"kb_used_data":12164,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124584616,"statfs":{"total":1151940427776,"available":1151574646784,"internally_reserved":0,"allocated":12455936,"data_stored":7954526,"data_compressed":360485,"data_compressed_allocated":487424,"data_compressed_original":2473984,"omap_allocated":19139,"internal_metadata":353154365},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":59,"num_read_kb":29,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.079954"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-26T23:44:59.538966+0000","last_change":"2024-04-26T23:43:03.301992+0000","last_active":"2024-04-26T23:44:59.538966+0000","last_peered":"2024-04-26T23:44:59.538966+0000","last_clean":"2024-04-26T23:44:59.538966+0000","last_became_active":"2024-04-26T23:43:03.301498+0000","last_became_peered":"2024-04-26T23:43:03.301498+0000","last_unstale":"2024-04-26T23:44:59.538966+0000","last_undegraded":"2024-04-26T23:44:59.538966+0000","last_fullsized":"2024-04-26T23:44:59.538966+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:40:53.567805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.567766+0000","last_change":"2024-04-26T23:43:03.284964+0000","last_active":"2024-04-26T23:43:59.567766+0000","last_peered":"2024-04-26T23:43:59.567766+0000","last_clean":"2024-04-26T23:43:59.567766+0000","last_became_active":"2024-04-26T23:43:03.284185+0000","last_became_peered":"2024-04-26T23:43:03.284185+0000","last_unstale":"2024-04-26T23:43:59.567766+0000","last_undegraded":"2024-04-26T23:43:59.567766+0000","last_fullsized":"2024-04-26T23:43:59.567766+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:55:35.325775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.567001+0000","last_change":"2024-04-26T23:43:03.300333+0000","last_active":"2024-04-26T23:43:59.567001+0000","last_peered":"2024-04-26T23:43:59.567001+0000","last_clean":"2024-04-26T23:43:59.567001+0000","last_became_active":"2024-04-26T23:43:03.295779+0000","last_became_peered":"2024-04-26T23:43:03.295779+0000","last_unstale":"2024-04-26T23:43:59.567001+0000","last_undegraded":"2024-04-26T23:43:59.567001+0000","last_fullsized":"2024-04-26T23:43:59.567001+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:45:06.598862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.566294+0000","last_change":"2024-04-26T23:43:03.290211+0000","last_active":"2024-04-26T23:43:59.566294+0000","last_peered":"2024-04-26T23:43:59.566294+0000","last_clean":"2024-04-26T23:43:59.566294+0000","last_became_active":"2024-04-26T23:43:03.289853+0000","last_became_peered":"2024-04-26T23:43:03.289853+0000","last_unstale":"2024-04-26T23:43:59.566294+0000","last_undegraded":"2024-04-26T23:43:59.566294+0000","last_fullsized":"2024-04-26T23:43:59.566294+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:14:45.032300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.570830+0000","last_change":"2024-04-26T23:43:03.309667+0000","last_active":"2024-04-26T23:43:59.570830+0000","last_peered":"2024-04-26T23:43:59.570830+0000","last_clean":"2024-04-26T23:43:59.570830+0000","last_became_active":"2024-04-26T23:43:03.309305+0000","last_became_peered":"2024-04-26T23:43:03.309305+0000","last_unstale":"2024-04-26T23:43:59.570830+0000","last_undegraded":"2024-04-26T23:43:59.570830+0000","last_fullsized":"2024-04-26T23:43:59.570830+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:31:56.739866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.867340+0000","last_change":"2024-04-26T23:43:03.292107+0000","last_active":"2024-04-26T23:43:59.867340+0000","last_peered":"2024-04-26T23:43:59.867340+0000","last_clean":"2024-04-26T23:43:59.867340+0000","last_became_active":"2024-04-26T23:43:03.291760+0000","last_became_peered":"2024-04-26T23:43:03.291760+0000","last_unstale":"2024-04-26T23:43:59.867340+0000","last_undegraded":"2024-04-26T23:43:59.867340+0000","last_fullsized":"2024-04-26T23:43:59.867340+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:31:54.654379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.571420+0000","last_change":"2024-04-26T23:43:03.294923+0000","last_active":"2024-04-26T23:43:59.571420+0000","last_peered":"2024-04-26T23:43:59.571420+0000","last_clean":"2024-04-26T23:43:59.571420+0000","last_became_active":"2024-04-26T23:43:03.294523+0000","last_became_peered":"2024-04-26T23:43:03.294523+0000","last_unstale":"2024-04-26T23:43:59.571420+0000","last_undegraded":"2024-04-26T23:43:59.571420+0000","last_fullsized":"2024-04-26T23:43:59.571420+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:36:19.419773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.568934+0000","last_change":"2024-04-26T23:43:03.301335+0000","last_active":"2024-04-26T23:43:59.568934+0000","last_peered":"2024-04-26T23:43:59.568934+0000","last_clean":"2024-04-26T23:43:59.568934+0000","last_became_active":"2024-04-26T23:43:03.300918+0000","last_became_peered":"2024-04-26T23:43:03.300918+0000","last_unstale":"2024-04-26T23:43:59.568934+0000","last_undegraded":"2024-04-26T23:43:59.568934+0000","last_fullsized":"2024-04-26T23:43:59.568934+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:15:05.719452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-26T23:44:59.539520+0000","last_change":"2024-04-26T23:43:59.574776+0000","last_active":"2024-04-26T23:44:59.539520+0000","last_peered":"2024-04-26T23:44:59.539520+0000","last_clean":"2024-04-26T23:44:59.539520+0000","last_became_active":"2024-04-26T23:43:59.574299+0000","last_became_peered":"2024-04-26T23:43:59.574299+0000","last_unstale":"2024-04-26T23:44:59.539520+0000","last_undegraded":"2024-04-26T23:44:59.539520+0000","last_fullsized":"2024-04-26T23:44:59.539520+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:14:16.089377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,11],"acting":[1,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.867834+0000","last_change":"2024-04-26T23:43:03.287643+0000","last_active":"2024-04-26T23:43:59.867834+0000","last_peered":"2024-04-26T23:43:59.867834+0000","last_clean":"2024-04-26T23:43:59.867834+0000","last_became_active":"2024-04-26T23:43:03.287324+0000","last_became_peered":"2024-04-26T23:43:03.287324+0000","last_unstale":"2024-04-26T23:43:59.867834+0000","last_undegraded":"2024-04-26T23:43:59.867834+0000","last_fullsized":"2024-04-26T23:43:59.867834+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:50:14.816548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.572333+0000","last_change":"2024-04-26T23:43:03.296421+0000","last_active":"2024-04-26T23:43:59.572333+0000","last_peered":"2024-04-26T23:43:59.572333+0000","last_clean":"2024-04-26T23:43:59.572333+0000","last_became_active":"2024-04-26T23:43:03.296060+0000","last_became_peered":"2024-04-26T23:43:03.296060+0000","last_unstale":"2024-04-26T23:43:59.572333+0000","last_undegraded":"2024-04-26T23:43:59.572333+0000","last_fullsized":"2024-04-26T23:43:59.572333+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:50:44.393446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.566081+0000","last_change":"2024-04-26T23:43:03.288562+0000","last_active":"2024-04-26T23:43:59.566081+0000","last_peered":"2024-04-26T23:43:59.566081+0000","last_clean":"2024-04-26T23:43:59.566081+0000","last_became_active":"2024-04-26T23:43:03.288219+0000","last_became_peered":"2024-04-26T23:43:03.288219+0000","last_unstale":"2024-04-26T23:43:59.566081+0000","last_undegraded":"2024-04-26T23:43:59.566081+0000","last_fullsized":"2024-04-26T23:43:59.566081+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:54:32.107146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.868611+0000","last_change":"2024-04-26T23:43:03.296840+0000","last_active":"2024-04-26T23:43:59.868611+0000","last_peered":"2024-04-26T23:43:59.868611+0000","last_clean":"2024-04-26T23:43:59.868611+0000","last_became_active":"2024-04-26T23:43:03.296501+0000","last_became_peered":"2024-04-26T23:43:03.296501+0000","last_unstale":"2024-04-26T23:43:59.868611+0000","last_undegraded":"2024-04-26T23:43:59.868611+0000","last_fullsized":"2024-04-26T23:43:59.868611+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:07:05.293388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.567210+0000","last_change":"2024-04-26T23:43:03.291739+0000","last_active":"2024-04-26T23:43:59.567210+0000","last_peered":"2024-04-26T23:43:59.567210+0000","last_clean":"2024-04-26T23:43:59.567210+0000","last_became_active":"2024-04-26T23:43:03.287839+0000","last_became_peered":"2024-04-26T23:43:03.287839+0000","last_unstale":"2024-04-26T23:43:59.567210+0000","last_undegraded":"2024-04-26T23:43:59.567210+0000","last_fullsized":"2024-04-26T23:43:59.567210+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:10:56.698324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.567020+0000","last_change":"2024-04-26T23:43:03.295767+0000","last_active":"2024-04-26T23:43:59.567020+0000","last_peered":"2024-04-26T23:43:59.567020+0000","last_clean":"2024-04-26T23:43:59.567020+0000","last_became_active":"2024-04-26T23:43:03.295339+0000","last_became_peered":"2024-04-26T23:43:03.295339+0000","last_unstale":"2024-04-26T23:43:59.567020+0000","last_undegraded":"2024-04-26T23:43:59.567020+0000","last_fullsized":"2024-04-26T23:43:59.567020+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:21:44.410953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-26T23:44:59.537359+0000","last_change":"2024-04-26T23:43:03.301687+0000","last_active":"2024-04-26T23:44:59.537359+0000","last_peered":"2024-04-26T23:44:59.537359+0000","last_clean":"2024-04-26T23:44:59.537359+0000","last_became_active":"2024-04-26T23:43:03.301355+0000","last_became_peered":"2024-04-26T23:43:03.301355+0000","last_unstale":"2024-04-26T23:44:59.537359+0000","last_undegraded":"2024-04-26T23:44:59.537359+0000","last_fullsized":"2024-04-26T23:44:59.537359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:00:57.991022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-26T23:44:59.538842+0000","last_change":"2024-04-26T23:43:03.302109+0000","last_active":"2024-04-26T23:44:59.538842+0000","last_peered":"2024-04-26T23:44:59.538842+0000","last_clean":"2024-04-26T23:44:59.538842+0000","last_became_active":"2024-04-26T23:43:03.301725+0000","last_became_peered":"2024-04-26T23:43:03.301725+0000","last_unstale":"2024-04-26T23:44:59.538842+0000","last_undegraded":"2024-04-26T23:44:59.538842+0000","last_fullsized":"2024-04-26T23:44:59.538842+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:36:44.630125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.576171+0000","last_change":"2024-04-26T23:43:03.298880+0000","last_active":"2024-04-26T23:43:59.576171+0000","last_peered":"2024-04-26T23:43:59.576171+0000","last_clean":"2024-04-26T23:43:59.576171+0000","last_became_active":"2024-04-26T23:43:03.298464+0000","last_became_peered":"2024-04-26T23:43:03.298464+0000","last_unstale":"2024-04-26T23:43:59.576171+0000","last_undegraded":"2024-04-26T23:43:59.576171+0000","last_fullsized":"2024-04-26T23:43:59.576171+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:42:54.745002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.567554+0000","last_change":"2024-04-26T23:43:03.309454+0000","last_active":"2024-04-26T23:43:59.567554+0000","last_peered":"2024-04-26T23:43:59.567554+0000","last_clean":"2024-04-26T23:43:59.567554+0000","last_became_active":"2024-04-26T23:43:03.309091+0000","last_became_peered":"2024-04-26T23:43:03.309091+0000","last_unstale":"2024-04-26T23:43:59.567554+0000","last_undegraded":"2024-04-26T23:43:59.567554+0000","last_fullsized":"2024-04-26T23:43:59.567554+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:46:12.333410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.566347+0000","last_change":"2024-04-26T23:43:07.541317+0000","last_active":"2024-04-26T23:43:59.566347+0000","last_peered":"2024-04-26T23:43:59.566347+0000","last_clean":"2024-04-26T23:43:59.566347+0000","last_became_active":"2024-04-26T23:43:07.540760+0000","last_became_peered":"2024-04-26T23:43:07.540760+0000","last_unstale":"2024-04-26T23:43:59.566347+0000","last_undegraded":"2024-04-26T23:43:59.566347+0000","last_fullsized":"2024-04-26T23:43:59.566347+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:28:55.545416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.576255+0000","last_change":"2024-04-26T23:43:02.450956+0000","last_active":"2024-04-26T23:43:59.576255+0000","last_peered":"2024-04-26T23:43:59.576255+0000","last_clean":"2024-04-26T23:43:59.576255+0000","last_became_active":"2024-04-26T23:43:02.446733+0000","last_became_peered":"2024-04-26T23:43:02.446733+0000","last_unstale":"2024-04-26T23:43:59.576255+0000","last_undegraded":"2024-04-26T23:43:59.576255+0000","last_fullsized":"2024-04-26T23:43:59.576255+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:58:47.613575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.571020+0000","last_change":"2024-04-26T23:43:03.311681+0000","last_active":"2024-04-26T23:43:59.571020+0000","last_peered":"2024-04-26T23:43:59.571020+0000","last_clean":"2024-04-26T23:43:59.571020+0000","last_became_active":"2024-04-26T23:43:03.311087+0000","last_became_peered":"2024-04-26T23:43:03.311087+0000","last_unstale":"2024-04-26T23:43:59.571020+0000","last_undegraded":"2024-04-26T23:43:59.571020+0000","last_fullsized":"2024-04-26T23:43:59.571020+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:13:00.744062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.565507+0000","last_change":"2024-04-26T23:43:07.543547+0000","last_active":"2024-04-26T23:43:59.565507+0000","last_peered":"2024-04-26T23:43:59.565507+0000","last_clean":"2024-04-26T23:43:59.565507+0000","last_became_active":"2024-04-26T23:43:07.543073+0000","last_became_peered":"2024-04-26T23:43:07.543073+0000","last_unstale":"2024-04-26T23:43:59.565507+0000","last_undegraded":"2024-04-26T23:43:59.565507+0000","last_fullsized":"2024-04-26T23:43:59.565507+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:30:05.418875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.571145+0000","last_change":"2024-04-26T23:43:02.460459+0000","last_active":"2024-04-26T23:43:59.571145+0000","last_peered":"2024-04-26T23:43:59.571145+0000","last_clean":"2024-04-26T23:43:59.571145+0000","last_became_active":"2024-04-26T23:43:02.459652+0000","last_became_peered":"2024-04-26T23:43:02.459652+0000","last_unstale":"2024-04-26T23:43:59.571145+0000","last_undegraded":"2024-04-26T23:43:59.571145+0000","last_fullsized":"2024-04-26T23:43:59.571145+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:27:19.166748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.568072+0000","last_change":"2024-04-26T23:43:03.300078+0000","last_active":"2024-04-26T23:43:59.568072+0000","last_peered":"2024-04-26T23:43:59.568072+0000","last_clean":"2024-04-26T23:43:59.568072+0000","last_became_active":"2024-04-26T23:43:03.296319+0000","last_became_peered":"2024-04-26T23:43:03.296319+0000","last_unstale":"2024-04-26T23:43:59.568072+0000","last_undegraded":"2024-04-26T23:43:59.568072+0000","last_fullsized":"2024-04-26T23:43:59.568072+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:33:51.859539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.565683+0000","last_change":"2024-04-26T23:43:07.543494+0000","last_active":"2024-04-26T23:43:59.565683+0000","last_peered":"2024-04-26T23:43:59.565683+0000","last_clean":"2024-04-26T23:43:59.565683+0000","last_became_active":"2024-04-26T23:43:07.542806+0000","last_became_peered":"2024-04-26T23:43:07.542806+0000","last_unstale":"2024-04-26T23:43:59.565683+0000","last_undegraded":"2024-04-26T23:43:59.565683+0000","last_fullsized":"2024-04-26T23:43:59.565683+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:54:40.522615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.571618+0000","last_change":"2024-04-26T23:43:02.438943+0000","last_active":"2024-04-26T23:43:59.571618+0000","last_peered":"2024-04-26T23:43:59.571618+0000","last_clean":"2024-04-26T23:43:59.571618+0000","last_became_active":"2024-04-26T23:43:02.438604+0000","last_became_peered":"2024-04-26T23:43:02.438604+0000","last_unstale":"2024-04-26T23:43:59.571618+0000","last_undegraded":"2024-04-26T23:43:59.571618+0000","last_fullsized":"2024-04-26T23:43:59.571618+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:06:51.054342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-26T23:44:59.538833+0000","last_change":"2024-04-26T23:43:03.293856+0000","last_active":"2024-04-26T23:44:59.538833+0000","last_peered":"2024-04-26T23:44:59.538833+0000","last_clean":"2024-04-26T23:44:59.538833+0000","last_became_active":"2024-04-26T23:43:03.293189+0000","last_became_peered":"2024-04-26T23:43:03.293189+0000","last_unstale":"2024-04-26T23:44:59.538833+0000","last_undegraded":"2024-04-26T23:44:59.538833+0000","last_fullsized":"2024-04-26T23:44:59.538833+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:50:27.545567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.565535+0000","last_change":"2024-04-26T23:43:07.535396+0000","last_active":"2024-04-26T23:43:59.565535+0000","last_peered":"2024-04-26T23:43:59.565535+0000","last_clean":"2024-04-26T23:43:59.565535+0000","last_became_active":"2024-04-26T23:43:07.534914+0000","last_became_peered":"2024-04-26T23:43:07.534914+0000","last_unstale":"2024-04-26T23:43:59.565535+0000","last_undegraded":"2024-04-26T23:43:59.565535+0000","last_fullsized":"2024-04-26T23:43:59.565535+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:27:31.199079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.571839+0000","last_change":"2024-04-26T23:43:02.442935+0000","last_active":"2024-04-26T23:43:59.571839+0000","last_peered":"2024-04-26T23:43:59.571839+0000","last_clean":"2024-04-26T23:43:59.571839+0000","last_became_active":"2024-04-26T23:43:02.442588+0000","last_became_peered":"2024-04-26T23:43:02.442588+0000","last_unstale":"2024-04-26T23:43:59.571839+0000","last_undegraded":"2024-04-26T23:43:59.571839+0000","last_fullsized":"2024-04-26T23:43:59.571839+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:05:26.442001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.569033+0000","last_change":"2024-04-26T23:43:03.289407+0000","last_active":"2024-04-26T23:43:59.569033+0000","last_peered":"2024-04-26T23:43:59.569033+0000","last_clean":"2024-04-26T23:43:59.569033+0000","last_became_active":"2024-04-26T23:43:03.287951+0000","last_became_peered":"2024-04-26T23:43:03.287951+0000","last_unstale":"2024-04-26T23:43:59.569033+0000","last_undegraded":"2024-04-26T23:43:59.569033+0000","last_fullsized":"2024-04-26T23:43:59.569033+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:24:42.687653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.567047+0000","last_change":"2024-04-26T23:43:07.527019+0000","last_active":"2024-04-26T23:43:59.567047+0000","last_peered":"2024-04-26T23:43:59.567047+0000","last_clean":"2024-04-26T23:43:59.567047+0000","last_became_active":"2024-04-26T23:43:07.526516+0000","last_became_peered":"2024-04-26T23:43:07.526516+0000","last_unstale":"2024-04-26T23:43:59.567047+0000","last_undegraded":"2024-04-26T23:43:59.567047+0000","last_fullsized":"2024-04-26T23:43:59.567047+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:44:41.267144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'6","reported_seq":125,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:44:57.335620+0000","last_change":"2024-04-26T23:43:02.445159+0000","last_active":"2024-04-26T23:44:57.335620+0000","last_peered":"2024-04-26T23:44:57.335620+0000","last_clean":"2024-04-26T23:44:57.335620+0000","last_became_active":"2024-04-26T23:43:02.444781+0000","last_became_peered":"2024-04-26T23:43:02.444781+0000","last_unstale":"2024-04-26T23:44:57.335620+0000","last_undegraded":"2024-04-26T23:44:57.335620+0000","last_fullsized":"2024-04-26T23:44:57.335620+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:45:16.509790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.572128+0000","last_change":"2024-04-26T23:43:03.292717+0000","last_active":"2024-04-26T23:43:59.572128+0000","last_peered":"2024-04-26T23:43:59.572128+0000","last_clean":"2024-04-26T23:43:59.572128+0000","last_became_active":"2024-04-26T23:43:03.292355+0000","last_became_peered":"2024-04-26T23:43:03.292355+0000","last_unstale":"2024-04-26T23:43:59.572128+0000","last_undegraded":"2024-04-26T23:43:59.572128+0000","last_fullsized":"2024-04-26T23:43:59.572128+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:06:02.063165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.566558+0000","last_change":"2024-04-26T23:43:07.540758+0000","last_active":"2024-04-26T23:43:59.566558+0000","last_peered":"2024-04-26T23:43:59.566558+0000","last_clean":"2024-04-26T23:43:59.566558+0000","last_became_active":"2024-04-26T23:43:07.540305+0000","last_became_peered":"2024-04-26T23:43:07.540305+0000","last_unstale":"2024-04-26T23:43:59.566558+0000","last_undegraded":"2024-04-26T23:43:59.566558+0000","last_fullsized":"2024-04-26T23:43:59.566558+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:08:04.491544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.867249+0000","last_change":"2024-04-26T23:43:02.453650+0000","last_active":"2024-04-26T23:43:59.867249+0000","last_peered":"2024-04-26T23:43:59.867249+0000","last_clean":"2024-04-26T23:43:59.867249+0000","last_became_active":"2024-04-26T23:43:02.453319+0000","last_became_peered":"2024-04-26T23:43:02.453319+0000","last_unstale":"2024-04-26T23:43:59.867249+0000","last_undegraded":"2024-04-26T23:43:59.867249+0000","last_fullsized":"2024-04-26T23:43:59.867249+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:23:27.570603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.870417+0000","last_change":"2024-04-26T23:43:59.870417+0000","last_active":"2024-04-26T23:43:59.870417+0000","last_peered":"2024-04-26T23:43:59.870417+0000","last_clean":"2024-04-26T23:43:59.870417+0000","last_became_active":"2024-04-26T23:43:59.870053+0000","last_became_peered":"2024-04-26T23:43:59.870053+0000","last_unstale":"2024-04-26T23:43:59.870417+0000","last_undegraded":"2024-04-26T23:43:59.870417+0000","last_fullsized":"2024-04-26T23:43:59.870417+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:07:31.020611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-26T23:44:59.538672+0000","last_change":"2024-04-26T23:43:07.546026+0000","last_active":"2024-04-26T23:44:59.538672+0000","last_peered":"2024-04-26T23:44:59.538672+0000","last_clean":"2024-04-26T23:44:59.538672+0000","last_became_active":"2024-04-26T23:43:07.545668+0000","last_became_peered":"2024-04-26T23:43:07.545668+0000","last_unstale":"2024-04-26T23:44:59.538672+0000","last_undegraded":"2024-04-26T23:44:59.538672+0000","last_fullsized":"2024-04-26T23:44:59.538672+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:54:20.580267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.868447+0000","last_change":"2024-04-26T23:43:02.454937+0000","last_active":"2024-04-26T23:43:59.868447+0000","last_peered":"2024-04-26T23:43:59.868447+0000","last_clean":"2024-04-26T23:43:59.868447+0000","last_became_active":"2024-04-26T23:43:02.454585+0000","last_became_peered":"2024-04-26T23:43:02.454585+0000","last_unstale":"2024-04-26T23:43:59.868447+0000","last_undegraded":"2024-04-26T23:43:59.868447+0000","last_fullsized":"2024-04-26T23:43:59.868447+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:55:54.999411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.873013+0000","last_change":"2024-04-26T23:43:59.870821+0000","last_active":"2024-04-26T23:43:59.873013+0000","last_peered":"2024-04-26T23:43:59.873013+0000","last_clean":"2024-04-26T23:43:59.873013+0000","last_became_active":"2024-04-26T23:43:59.870240+0000","last_became_peered":"2024-04-26T23:43:59.870240+0000","last_unstale":"2024-04-26T23:43:59.873013+0000","last_undegraded":"2024-04-26T23:43:59.873013+0000","last_fullsized":"2024-04-26T23:43:59.873013+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:33:00.284232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,6],"acting":[9,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.567737+0000","last_change":"2024-04-26T23:43:07.535682+0000","last_active":"2024-04-26T23:43:59.567737+0000","last_peered":"2024-04-26T23:43:59.567737+0000","last_clean":"2024-04-26T23:43:59.567737+0000","last_became_active":"2024-04-26T23:43:07.535221+0000","last_became_peered":"2024-04-26T23:43:07.535221+0000","last_unstale":"2024-04-26T23:43:59.567737+0000","last_undegraded":"2024-04-26T23:43:59.567737+0000","last_fullsized":"2024-04-26T23:43:59.567737+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:33:54.105329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.569396+0000","last_change":"2024-04-26T23:43:02.443541+0000","last_active":"2024-04-26T23:43:59.569396+0000","last_peered":"2024-04-26T23:43:59.569396+0000","last_clean":"2024-04-26T23:43:59.569396+0000","last_became_active":"2024-04-26T23:43:02.442948+0000","last_became_peered":"2024-04-26T23:43:02.442948+0000","last_unstale":"2024-04-26T23:43:59.569396+0000","last_undegraded":"2024-04-26T23:43:59.569396+0000","last_fullsized":"2024-04-26T23:43:59.569396+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:58:44.595575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.574143+0000","last_change":"2024-04-26T23:43:03.311029+0000","last_active":"2024-04-26T23:43:59.574143+0000","last_peered":"2024-04-26T23:43:59.574143+0000","last_clean":"2024-04-26T23:43:59.574143+0000","last_became_active":"2024-04-26T23:43:03.310722+0000","last_became_peered":"2024-04-26T23:43:03.310722+0000","last_unstale":"2024-04-26T23:43:59.574143+0000","last_undegraded":"2024-04-26T23:43:59.574143+0000","last_fullsized":"2024-04-26T23:43:59.574143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:07:54.449492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.569381+0000","last_change":"2024-04-26T23:43:07.537391+0000","last_active":"2024-04-26T23:43:59.569381+0000","last_peered":"2024-04-26T23:43:59.569381+0000","last_clean":"2024-04-26T23:43:59.569381+0000","last_became_active":"2024-04-26T23:43:07.536216+0000","last_became_peered":"2024-04-26T23:43:07.536216+0000","last_unstale":"2024-04-26T23:43:59.569381+0000","last_undegraded":"2024-04-26T23:43:59.569381+0000","last_fullsized":"2024-04-26T23:43:59.569381+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:45:55.326283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":118,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:44:57.335011+0000","last_change":"2024-04-26T23:43:02.451929+0000","last_active":"2024-04-26T23:44:57.335011+0000","last_peered":"2024-04-26T23:44:57.335011+0000","last_clean":"2024-04-26T23:44:57.335011+0000","last_became_active":"2024-04-26T23:43:02.451700+0000","last_became_peered":"2024-04-26T23:43:02.451700+0000","last_unstale":"2024-04-26T23:44:57.335011+0000","last_undegraded":"2024-04-26T23:44:57.335011+0000","last_fullsized":"2024-04-26T23:44:57.335011+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:21:31.279681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":89,"num_read_kb":89,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-26T23:44:59.539275+0000","last_change":"2024-04-26T23:43:03.295462+0000","last_active":"2024-04-26T23:44:59.539275+0000","last_peered":"2024-04-26T23:44:59.539275+0000","last_clean":"2024-04-26T23:44:59.539275+0000","last_became_active":"2024-04-26T23:43:03.295044+0000","last_became_peered":"2024-04-26T23:43:03.295044+0000","last_unstale":"2024-04-26T23:44:59.539275+0000","last_undegraded":"2024-04-26T23:44:59.539275+0000","last_fullsized":"2024-04-26T23:44:59.539275+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:56:19.445166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.573518+0000","last_change":"2024-04-26T23:43:07.539397+0000","last_active":"2024-04-26T23:43:59.573518+0000","last_peered":"2024-04-26T23:43:59.573518+0000","last_clean":"2024-04-26T23:43:59.573518+0000","last_became_active":"2024-04-26T23:43:07.533081+0000","last_became_peered":"2024-04-26T23:43:07.533081+0000","last_unstale":"2024-04-26T23:43:59.573518+0000","last_undegraded":"2024-04-26T23:43:59.573518+0000","last_fullsized":"2024-04-26T23:43:59.573518+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:24:39.693721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.571849+0000","last_change":"2024-04-26T23:43:02.439919+0000","last_active":"2024-04-26T23:43:59.571849+0000","last_peered":"2024-04-26T23:43:59.571849+0000","last_clean":"2024-04-26T23:43:59.571849+0000","last_became_active":"2024-04-26T23:43:02.439561+0000","last_became_peered":"2024-04-26T23:43:02.439561+0000","last_unstale":"2024-04-26T23:43:59.571849+0000","last_undegraded":"2024-04-26T23:43:59.571849+0000","last_fullsized":"2024-04-26T23:43:59.571849+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:30:22.905464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.567979+0000","last_change":"2024-04-26T23:43:03.294088+0000","last_active":"2024-04-26T23:43:59.567979+0000","last_peered":"2024-04-26T23:43:59.567979+0000","last_clean":"2024-04-26T23:43:59.567979+0000","last_became_active":"2024-04-26T23:43:03.293726+0000","last_became_peered":"2024-04-26T23:43:03.293726+0000","last_unstale":"2024-04-26T23:43:59.567979+0000","last_undegraded":"2024-04-26T23:43:59.567979+0000","last_fullsized":"2024-04-26T23:43:59.567979+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:19:42.575193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.565828+0000","last_change":"2024-04-26T23:43:07.543056+0000","last_active":"2024-04-26T23:43:59.565828+0000","last_peered":"2024-04-26T23:43:59.565828+0000","last_clean":"2024-04-26T23:43:59.565828+0000","last_became_active":"2024-04-26T23:43:07.542324+0000","last_became_peered":"2024-04-26T23:43:07.542324+0000","last_unstale":"2024-04-26T23:43:59.565828+0000","last_undegraded":"2024-04-26T23:43:59.565828+0000","last_fullsized":"2024-04-26T23:43:59.565828+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:23:46.415932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.568667+0000","last_change":"2024-04-26T23:43:02.454135+0000","last_active":"2024-04-26T23:43:59.568667+0000","last_peered":"2024-04-26T23:43:59.568667+0000","last_clean":"2024-04-26T23:43:59.568667+0000","last_became_active":"2024-04-26T23:43:02.453798+0000","last_became_peered":"2024-04-26T23:43:02.453798+0000","last_unstale":"2024-04-26T23:43:59.568667+0000","last_undegraded":"2024-04-26T23:43:59.568667+0000","last_fullsized":"2024-04-26T23:43:59.568667+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:16:25.286569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-26T23:44:59.539446+0000","last_change":"2024-04-26T23:43:03.293520+0000","last_active":"2024-04-26T23:44:59.539446+0000","last_peered":"2024-04-26T23:44:59.539446+0000","last_clean":"2024-04-26T23:44:59.539446+0000","last_became_active":"2024-04-26T23:43:03.292537+0000","last_became_peered":"2024-04-26T23:43:03.292537+0000","last_unstale":"2024-04-26T23:44:59.539446+0000","last_undegraded":"2024-04-26T23:44:59.539446+0000","last_fullsized":"2024-04-26T23:44:59.539446+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:19:28.808217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.574638+0000","last_change":"2024-04-26T23:43:07.528116+0000","last_active":"2024-04-26T23:43:59.574638+0000","last_peered":"2024-04-26T23:43:59.574638+0000","last_clean":"2024-04-26T23:43:59.574638+0000","last_became_active":"2024-04-26T23:43:07.527766+0000","last_became_peered":"2024-04-26T23:43:07.527766+0000","last_unstale":"2024-04-26T23:43:59.574638+0000","last_undegraded":"2024-04-26T23:43:59.574638+0000","last_fullsized":"2024-04-26T23:43:59.574638+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:39:23.949964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'33","reported_seq":244,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:44:58.346898+0000","last_change":"2024-04-26T23:43:02.437454+0000","last_active":"2024-04-26T23:44:58.346898+0000","last_peered":"2024-04-26T23:44:58.346898+0000","last_clean":"2024-04-26T23:44:58.346898+0000","last_became_active":"2024-04-26T23:43:02.437142+0000","last_became_peered":"2024-04-26T23:43:02.437142+0000","last_unstale":"2024-04-26T23:44:58.346898+0000","last_undegraded":"2024-04-26T23:44:58.346898+0000","last_fullsized":"2024-04-26T23:44:58.346898+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:06:37.751066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54826,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":185,"num_read_kb":145,"num_write":34,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.571323+0000","last_change":"2024-04-26T23:43:03.287831+0000","last_active":"2024-04-26T23:43:59.571323+0000","last_peered":"2024-04-26T23:43:59.571323+0000","last_clean":"2024-04-26T23:43:59.571323+0000","last_became_active":"2024-04-26T23:43:03.287479+0000","last_became_peered":"2024-04-26T23:43:03.287479+0000","last_unstale":"2024-04-26T23:43:59.571323+0000","last_undegraded":"2024-04-26T23:43:59.571323+0000","last_fullsized":"2024-04-26T23:43:59.571323+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:39:41.726893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.569525+0000","last_change":"2024-04-26T23:43:07.543985+0000","last_active":"2024-04-26T23:43:59.569525+0000","last_peered":"2024-04-26T23:43:59.569525+0000","last_clean":"2024-04-26T23:43:59.569525+0000","last_became_active":"2024-04-26T23:43:07.543327+0000","last_became_peered":"2024-04-26T23:43:07.543327+0000","last_unstale":"2024-04-26T23:43:59.569525+0000","last_undegraded":"2024-04-26T23:43:59.569525+0000","last_fullsized":"2024-04-26T23:43:59.569525+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:24:10.577822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.566396+0000","last_change":"2024-04-26T23:43:02.447627+0000","last_active":"2024-04-26T23:43:59.566396+0000","last_peered":"2024-04-26T23:43:59.566396+0000","last_clean":"2024-04-26T23:43:59.566396+0000","last_became_active":"2024-04-26T23:43:02.447325+0000","last_became_peered":"2024-04-26T23:43:02.447325+0000","last_unstale":"2024-04-26T23:43:59.566396+0000","last_undegraded":"2024-04-26T23:43:59.566396+0000","last_fullsized":"2024-04-26T23:43:59.566396+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:19:03.757202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.869446+0000","last_change":"2024-04-26T23:43:03.313388+0000","last_active":"2024-04-26T23:43:59.869446+0000","last_peered":"2024-04-26T23:43:59.869446+0000","last_clean":"2024-04-26T23:43:59.869446+0000","last_became_active":"2024-04-26T23:43:03.312899+0000","last_became_peered":"2024-04-26T23:43:03.312899+0000","last_unstale":"2024-04-26T23:43:59.869446+0000","last_undegraded":"2024-04-26T23:43:59.869446+0000","last_fullsized":"2024-04-26T23:43:59.869446+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:58:37.126485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-26T23:44:59.538190+0000","last_change":"2024-04-26T23:43:07.536706+0000","last_active":"2024-04-26T23:44:59.538190+0000","last_peered":"2024-04-26T23:44:59.538190+0000","last_clean":"2024-04-26T23:44:59.538190+0000","last_became_active":"2024-04-26T23:43:07.536316+0000","last_became_peered":"2024-04-26T23:43:07.536316+0000","last_unstale":"2024-04-26T23:44:59.538190+0000","last_undegraded":"2024-04-26T23:44:59.538190+0000","last_fullsized":"2024-04-26T23:44:59.538190+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:36:46.397725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.568997+0000","last_change":"2024-04-26T23:43:02.448647+0000","last_active":"2024-04-26T23:43:59.568997+0000","last_peered":"2024-04-26T23:43:59.568997+0000","last_clean":"2024-04-26T23:43:59.568997+0000","last_became_active":"2024-04-26T23:43:02.448392+0000","last_became_peered":"2024-04-26T23:43:02.448392+0000","last_unstale":"2024-04-26T23:43:59.568997+0000","last_undegraded":"2024-04-26T23:43:59.568997+0000","last_fullsized":"2024-04-26T23:43:59.568997+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:19:01.367361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.571932+0000","last_change":"2024-04-26T23:43:03.302980+0000","last_active":"2024-04-26T23:43:59.571932+0000","last_peered":"2024-04-26T23:43:59.571932+0000","last_clean":"2024-04-26T23:43:59.571932+0000","last_became_active":"2024-04-26T23:43:03.302589+0000","last_became_peered":"2024-04-26T23:43:03.302589+0000","last_unstale":"2024-04-26T23:43:59.571932+0000","last_undegraded":"2024-04-26T23:43:59.571932+0000","last_fullsized":"2024-04-26T23:43:59.571932+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:57:18.641063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.575941+0000","last_change":"2024-04-26T23:43:07.547842+0000","last_active":"2024-04-26T23:43:59.575941+0000","last_peered":"2024-04-26T23:43:59.575941+0000","last_clean":"2024-04-26T23:43:59.575941+0000","last_became_active":"2024-04-26T23:43:07.541774+0000","last_became_peered":"2024-04-26T23:43:07.541774+0000","last_unstale":"2024-04-26T23:43:59.575941+0000","last_undegraded":"2024-04-26T23:43:59.575941+0000","last_fullsized":"2024-04-26T23:43:59.575941+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:25:49.999465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'4","reported_seq":123,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:44:57.386294+0000","last_change":"2024-04-26T23:43:02.437210+0000","last_active":"2024-04-26T23:44:57.386294+0000","last_peered":"2024-04-26T23:44:57.386294+0000","last_clean":"2024-04-26T23:44:57.386294+0000","last_became_active":"2024-04-26T23:43:02.436886+0000","last_became_peered":"2024-04-26T23:43:02.436886+0000","last_unstale":"2024-04-26T23:44:57.386294+0000","last_undegraded":"2024-04-26T23:44:57.386294+0000","last_fullsized":"2024-04-26T23:44:57.386294+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:07:03.663757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":93,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.868608+0000","last_change":"2024-04-26T23:43:59.868608+0000","last_active":"2024-04-26T23:43:59.868608+0000","last_peered":"2024-04-26T23:43:59.868608+0000","last_clean":"2024-04-26T23:43:59.868608+0000","last_became_active":"2024-04-26T23:43:59.868231+0000","last_became_peered":"2024-04-26T23:43:59.868231+0000","last_unstale":"2024-04-26T23:43:59.868608+0000","last_undegraded":"2024-04-26T23:43:59.868608+0000","last_fullsized":"2024-04-26T23:43:59.868608+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:28:42.932733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.869040+0000","last_change":"2024-04-26T23:43:07.545626+0000","last_active":"2024-04-26T23:43:59.869040+0000","last_peered":"2024-04-26T23:43:59.869040+0000","last_clean":"2024-04-26T23:43:59.869040+0000","last_became_active":"2024-04-26T23:43:07.545171+0000","last_became_peered":"2024-04-26T23:43:07.545171+0000","last_unstale":"2024-04-26T23:43:59.869040+0000","last_undegraded":"2024-04-26T23:43:59.869040+0000","last_fullsized":"2024-04-26T23:43:59.869040+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:23:33.829834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.589239+0000","last_change":"2024-04-26T23:43:59.589239+0000","last_active":"2024-04-26T23:43:59.589239+0000","last_peered":"2024-04-26T23:43:59.589239+0000","last_clean":"2024-04-26T23:43:59.589239+0000","last_became_active":"2024-04-26T23:43:59.576239+0000","last_became_peered":"2024-04-26T23:43:59.576239+0000","last_unstale":"2024-04-26T23:43:59.589239+0000","last_undegraded":"2024-04-26T23:43:59.589239+0000","last_fullsized":"2024-04-26T23:43:59.589239+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:28:53.326961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.569408+0000","last_change":"2024-04-26T23:43:03.290567+0000","last_active":"2024-04-26T23:43:59.569408+0000","last_peered":"2024-04-26T23:43:59.569408+0000","last_clean":"2024-04-26T23:43:59.569408+0000","last_became_active":"2024-04-26T23:43:03.287807+0000","last_became_peered":"2024-04-26T23:43:03.287807+0000","last_unstale":"2024-04-26T23:43:59.569408+0000","last_undegraded":"2024-04-26T23:43:59.569408+0000","last_fullsized":"2024-04-26T23:43:59.569408+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:25:24.393978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.568665+0000","last_change":"2024-04-26T23:43:07.535402+0000","last_active":"2024-04-26T23:43:59.568665+0000","last_peered":"2024-04-26T23:43:59.568665+0000","last_clean":"2024-04-26T23:43:59.568665+0000","last_became_active":"2024-04-26T23:43:07.534870+0000","last_became_peered":"2024-04-26T23:43:07.534870+0000","last_unstale":"2024-04-26T23:43:59.568665+0000","last_undegraded":"2024-04-26T23:43:59.568665+0000","last_fullsized":"2024-04-26T23:43:59.568665+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:26:37.703831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":2,"reported_epoch":92,"state":"peering","last_fresh":"2024-04-26T23:43:08.418042+0000","last_change":"2024-04-26T23:45:00.495897+0000","last_active":"2024-04-26T23:45:00.495897+0000","last_peered":"2024-04-26T23:43:08.418042+0000","last_clean":"2024-04-26T23:43:08.418042+0000","last_became_active":"2024-04-26T23:43:02.440294+0000","last_became_peered":"2024-04-26T23:43:02.440294+0000","last_unstale":"2024-04-26T23:43:08.418042+0000","last_undegraded":"2024-04-26T23:43:08.418042+0000","last_fullsized":"2024-04-26T23:43:08.418042+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-26T23:44:59.538568+0000","last_change":"2024-04-26T23:43:59.576561+0000","last_active":"2024-04-26T23:44:59.538568+0000","last_peered":"2024-04-26T23:44:59.538568+0000","last_clean":"2024-04-26T23:44:59.538568+0000","last_became_active":"2024-04-26T23:43:59.574898+0000","last_became_peered":"2024-04-26T23:43:59.574898+0000","last_unstale":"2024-04-26T23:44:59.538568+0000","last_undegraded":"2024-04-26T23:44:59.538568+0000","last_fullsized":"2024-04-26T23:44:59.538568+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:57:04.569103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.572008+0000","last_change":"2024-04-26T23:43:07.855931+0000","last_active":"2024-04-26T23:43:59.572008+0000","last_peered":"2024-04-26T23:43:59.572008+0000","last_clean":"2024-04-26T23:43:59.572008+0000","last_became_active":"2024-04-26T23:43:07.855407+0000","last_became_peered":"2024-04-26T23:43:07.855407+0000","last_unstale":"2024-04-26T23:43:59.572008+0000","last_undegraded":"2024-04-26T23:43:59.572008+0000","last_fullsized":"2024-04-26T23:43:59.572008+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:39:25.707837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.575392+0000","last_change":"2024-04-26T23:43:02.457971+0000","last_active":"2024-04-26T23:43:59.575392+0000","last_peered":"2024-04-26T23:43:59.575392+0000","last_clean":"2024-04-26T23:43:59.575392+0000","last_became_active":"2024-04-26T23:43:02.457627+0000","last_became_peered":"2024-04-26T23:43:02.457627+0000","last_unstale":"2024-04-26T23:43:59.575392+0000","last_undegraded":"2024-04-26T23:43:59.575392+0000","last_fullsized":"2024-04-26T23:43:59.575392+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:25:51.749774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.867642+0000","last_change":"2024-04-26T23:43:03.305569+0000","last_active":"2024-04-26T23:43:59.867642+0000","last_peered":"2024-04-26T23:43:59.867642+0000","last_clean":"2024-04-26T23:43:59.867642+0000","last_became_active":"2024-04-26T23:43:03.305211+0000","last_became_peered":"2024-04-26T23:43:03.305211+0000","last_unstale":"2024-04-26T23:43:59.867642+0000","last_undegraded":"2024-04-26T23:43:59.867642+0000","last_fullsized":"2024-04-26T23:43:59.867642+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:54:55.847680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.868402+0000","last_change":"2024-04-26T23:43:07.535408+0000","last_active":"2024-04-26T23:43:59.868402+0000","last_peered":"2024-04-26T23:43:59.868402+0000","last_clean":"2024-04-26T23:43:59.868402+0000","last_became_active":"2024-04-26T23:43:07.534957+0000","last_became_peered":"2024-04-26T23:43:07.534957+0000","last_unstale":"2024-04-26T23:43:59.868402+0000","last_undegraded":"2024-04-26T23:43:59.868402+0000","last_fullsized":"2024-04-26T23:43:59.868402+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:35:31.103532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.571993+0000","last_change":"2024-04-26T23:43:02.453636+0000","last_active":"2024-04-26T23:43:59.571993+0000","last_peered":"2024-04-26T23:43:59.571993+0000","last_clean":"2024-04-26T23:43:59.571993+0000","last_became_active":"2024-04-26T23:43:02.453310+0000","last_became_peered":"2024-04-26T23:43:02.453310+0000","last_unstale":"2024-04-26T23:43:59.571993+0000","last_undegraded":"2024-04-26T23:43:59.571993+0000","last_fullsized":"2024-04-26T23:43:59.571993+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:42:29.295303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.867445+0000","last_change":"2024-04-26T23:43:03.283184+0000","last_active":"2024-04-26T23:43:59.867445+0000","last_peered":"2024-04-26T23:43:59.867445+0000","last_clean":"2024-04-26T23:43:59.867445+0000","last_became_active":"2024-04-26T23:43:03.282780+0000","last_became_peered":"2024-04-26T23:43:03.282780+0000","last_unstale":"2024-04-26T23:43:59.867445+0000","last_undegraded":"2024-04-26T23:43:59.867445+0000","last_fullsized":"2024-04-26T23:43:59.867445+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:00:13.907101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.868602+0000","last_change":"2024-04-26T23:43:07.859107+0000","last_active":"2024-04-26T23:43:59.868602+0000","last_peered":"2024-04-26T23:43:59.868602+0000","last_clean":"2024-04-26T23:43:59.868602+0000","last_became_active":"2024-04-26T23:43:07.856492+0000","last_became_peered":"2024-04-26T23:43:07.856492+0000","last_unstale":"2024-04-26T23:43:59.868602+0000","last_undegraded":"2024-04-26T23:43:59.868602+0000","last_fullsized":"2024-04-26T23:43:59.868602+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:13:36.570398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.569605+0000","last_change":"2024-04-26T23:43:03.289078+0000","last_active":"2024-04-26T23:43:59.569605+0000","last_peered":"2024-04-26T23:43:59.569605+0000","last_clean":"2024-04-26T23:43:59.569605+0000","last_became_active":"2024-04-26T23:43:03.288729+0000","last_became_peered":"2024-04-26T23:43:03.288729+0000","last_unstale":"2024-04-26T23:43:59.569605+0000","last_undegraded":"2024-04-26T23:43:59.569605+0000","last_fullsized":"2024-04-26T23:43:59.569605+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:32:36.346472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":120,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:44:57.385600+0000","last_change":"2024-04-26T23:43:02.463355+0000","last_active":"2024-04-26T23:44:57.385600+0000","last_peered":"2024-04-26T23:44:57.385600+0000","last_clean":"2024-04-26T23:44:57.385600+0000","last_became_active":"2024-04-26T23:43:02.455832+0000","last_became_peered":"2024-04-26T23:43:02.455832+0000","last_unstale":"2024-04-26T23:44:57.385600+0000","last_undegraded":"2024-04-26T23:44:57.385600+0000","last_fullsized":"2024-04-26T23:44:57.385600+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:33:33.837999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":89,"num_read_kb":89,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.572048+0000","last_change":"2024-04-26T23:43:07.540777+0000","last_active":"2024-04-26T23:43:59.572048+0000","last_peered":"2024-04-26T23:43:59.572048+0000","last_clean":"2024-04-26T23:43:59.572048+0000","last_became_active":"2024-04-26T23:43:07.540389+0000","last_became_peered":"2024-04-26T23:43:07.540389+0000","last_unstale":"2024-04-26T23:43:59.572048+0000","last_undegraded":"2024-04-26T23:43:59.572048+0000","last_fullsized":"2024-04-26T23:43:59.572048+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:57:28.674051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.569836+0000","last_change":"2024-04-26T23:43:03.308391+0000","last_active":"2024-04-26T23:43:59.569836+0000","last_peered":"2024-04-26T23:43:59.569836+0000","last_clean":"2024-04-26T23:43:59.569836+0000","last_became_active":"2024-04-26T23:43:03.306762+0000","last_became_peered":"2024-04-26T23:43:03.306762+0000","last_unstale":"2024-04-26T23:43:59.569836+0000","last_undegraded":"2024-04-26T23:43:59.569836+0000","last_fullsized":"2024-04-26T23:43:59.569836+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:34:01.083036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.565972+0000","last_change":"2024-04-26T23:43:02.448037+0000","last_active":"2024-04-26T23:43:59.565972+0000","last_peered":"2024-04-26T23:43:59.565972+0000","last_clean":"2024-04-26T23:43:59.565972+0000","last_became_active":"2024-04-26T23:43:02.447706+0000","last_became_peered":"2024-04-26T23:43:02.447706+0000","last_unstale":"2024-04-26T23:43:59.565972+0000","last_undegraded":"2024-04-26T23:43:59.565972+0000","last_fullsized":"2024-04-26T23:43:59.565972+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:43:56.295191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":348,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.571505+0000","last_change":"2024-04-26T23:41:42.266448+0000","last_active":"2024-04-26T23:43:59.571505+0000","last_peered":"2024-04-26T23:43:59.571505+0000","last_clean":"2024-04-26T23:43:59.571505+0000","last_became_active":"2024-04-26T23:41:42.265898+0000","last_became_peered":"2024-04-26T23:41:42.265898+0000","last_unstale":"2024-04-26T23:43:59.571505+0000","last_undegraded":"2024-04-26T23:43:59.571505+0000","last_fullsized":"2024-04-26T23:43:59.571505+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:37:44.460409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:37:44.460409+0000","last_clean_scrub_stamp":"2024-04-26T23:37:44.460409+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:40:48.177191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.568743+0000","last_change":"2024-04-26T23:43:07.856089+0000","last_active":"2024-04-26T23:43:59.568743+0000","last_peered":"2024-04-26T23:43:59.568743+0000","last_clean":"2024-04-26T23:43:59.568743+0000","last_became_active":"2024-04-26T23:43:07.855542+0000","last_became_peered":"2024-04-26T23:43:07.855542+0000","last_unstale":"2024-04-26T23:43:59.568743+0000","last_undegraded":"2024-04-26T23:43:59.568743+0000","last_fullsized":"2024-04-26T23:43:59.568743+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:48:22.799242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.868119+0000","last_change":"2024-04-26T23:43:03.301989+0000","last_active":"2024-04-26T23:43:59.868119+0000","last_peered":"2024-04-26T23:43:59.868119+0000","last_clean":"2024-04-26T23:43:59.868119+0000","last_became_active":"2024-04-26T23:43:03.301179+0000","last_became_peered":"2024-04-26T23:43:03.301179+0000","last_unstale":"2024-04-26T23:43:59.868119+0000","last_undegraded":"2024-04-26T23:43:59.868119+0000","last_fullsized":"2024-04-26T23:43:59.868119+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:29:33.012009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.571573+0000","last_change":"2024-04-26T23:43:02.439648+0000","last_active":"2024-04-26T23:43:59.571573+0000","last_peered":"2024-04-26T23:43:59.571573+0000","last_clean":"2024-04-26T23:43:59.571573+0000","last_became_active":"2024-04-26T23:43:02.439340+0000","last_became_peered":"2024-04-26T23:43:02.439340+0000","last_unstale":"2024-04-26T23:43:59.571573+0000","last_undegraded":"2024-04-26T23:43:59.571573+0000","last_fullsized":"2024-04-26T23:43:59.571573+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:10:01.096657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-26T23:44:59.538058+0000","last_change":"2024-04-26T23:43:07.534317+0000","last_active":"2024-04-26T23:44:59.538058+0000","last_peered":"2024-04-26T23:44:59.538058+0000","last_clean":"2024-04-26T23:44:59.538058+0000","last_became_active":"2024-04-26T23:43:07.533808+0000","last_became_peered":"2024-04-26T23:43:07.533808+0000","last_unstale":"2024-04-26T23:44:59.538058+0000","last_undegraded":"2024-04-26T23:44:59.538058+0000","last_fullsized":"2024-04-26T23:44:59.538058+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:54:29.952089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.571438+0000","last_change":"2024-04-26T23:43:02.457501+0000","last_active":"2024-04-26T23:43:59.571438+0000","last_peered":"2024-04-26T23:43:59.571438+0000","last_clean":"2024-04-26T23:43:59.571438+0000","last_became_active":"2024-04-26T23:43:02.457024+0000","last_became_peered":"2024-04-26T23:43:02.457024+0000","last_unstale":"2024-04-26T23:43:59.571438+0000","last_undegraded":"2024-04-26T23:43:59.571438+0000","last_fullsized":"2024-04-26T23:43:59.571438+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:21:58.542186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.868599+0000","last_change":"2024-04-26T23:43:03.303434+0000","last_active":"2024-04-26T23:43:59.868599+0000","last_peered":"2024-04-26T23:43:59.868599+0000","last_clean":"2024-04-26T23:43:59.868599+0000","last_became_active":"2024-04-26T23:43:03.302455+0000","last_became_peered":"2024-04-26T23:43:03.302455+0000","last_unstale":"2024-04-26T23:43:59.868599+0000","last_undegraded":"2024-04-26T23:43:59.868599+0000","last_fullsized":"2024-04-26T23:43:59.868599+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:57:05.530635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.568792+0000","last_change":"2024-04-26T23:43:07.856932+0000","last_active":"2024-04-26T23:43:59.568792+0000","last_peered":"2024-04-26T23:43:59.568792+0000","last_clean":"2024-04-26T23:43:59.568792+0000","last_became_active":"2024-04-26T23:43:07.855699+0000","last_became_peered":"2024-04-26T23:43:07.855699+0000","last_unstale":"2024-04-26T23:43:59.568792+0000","last_undegraded":"2024-04-26T23:43:59.568792+0000","last_fullsized":"2024-04-26T23:43:59.568792+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:20:28.994301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'7","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.576206+0000","last_change":"2024-04-26T23:43:02.449852+0000","last_active":"2024-04-26T23:43:59.576206+0000","last_peered":"2024-04-26T23:43:59.576206+0000","last_clean":"2024-04-26T23:43:59.576206+0000","last_became_active":"2024-04-26T23:43:02.449422+0000","last_became_peered":"2024-04-26T23:43:02.449422+0000","last_unstale":"2024-04-26T23:43:59.576206+0000","last_undegraded":"2024-04-26T23:43:59.576206+0000","last_fullsized":"2024-04-26T23:43:59.576206+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:07:25.453872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.570144+0000","last_change":"2024-04-26T23:43:03.304733+0000","last_active":"2024-04-26T23:43:59.570144+0000","last_peered":"2024-04-26T23:43:59.570144+0000","last_clean":"2024-04-26T23:43:59.570144+0000","last_became_active":"2024-04-26T23:43:03.304383+0000","last_became_peered":"2024-04-26T23:43:03.304383+0000","last_unstale":"2024-04-26T23:43:59.570144+0000","last_undegraded":"2024-04-26T23:43:59.570144+0000","last_fullsized":"2024-04-26T23:43:59.570144+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:49:28.679255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.566122+0000","last_change":"2024-04-26T23:43:07.525310+0000","last_active":"2024-04-26T23:43:59.566122+0000","last_peered":"2024-04-26T23:43:59.566122+0000","last_clean":"2024-04-26T23:43:59.566122+0000","last_became_active":"2024-04-26T23:43:07.524502+0000","last_became_peered":"2024-04-26T23:43:07.524502+0000","last_unstale":"2024-04-26T23:43:59.566122+0000","last_undegraded":"2024-04-26T23:43:59.566122+0000","last_fullsized":"2024-04-26T23:43:59.566122+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:07:45.927837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.566013+0000","last_change":"2024-04-26T23:43:02.446238+0000","last_active":"2024-04-26T23:43:59.566013+0000","last_peered":"2024-04-26T23:43:59.566013+0000","last_clean":"2024-04-26T23:43:59.566013+0000","last_became_active":"2024-04-26T23:43:02.445959+0000","last_became_peered":"2024-04-26T23:43:02.445959+0000","last_unstale":"2024-04-26T23:43:59.566013+0000","last_undegraded":"2024-04-26T23:43:59.566013+0000","last_fullsized":"2024-04-26T23:43:59.566013+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:25:10.007481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.567831+0000","last_change":"2024-04-26T23:43:03.306035+0000","last_active":"2024-04-26T23:43:59.567831+0000","last_peered":"2024-04-26T23:43:59.567831+0000","last_clean":"2024-04-26T23:43:59.567831+0000","last_became_active":"2024-04-26T23:43:03.305604+0000","last_became_peered":"2024-04-26T23:43:03.305604+0000","last_unstale":"2024-04-26T23:43:59.567831+0000","last_undegraded":"2024-04-26T23:43:59.567831+0000","last_fullsized":"2024-04-26T23:43:59.567831+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:02:38.567754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.565884+0000","last_change":"2024-04-26T23:43:07.855808+0000","last_active":"2024-04-26T23:43:59.565884+0000","last_peered":"2024-04-26T23:43:59.565884+0000","last_clean":"2024-04-26T23:43:59.565884+0000","last_became_active":"2024-04-26T23:43:07.855266+0000","last_became_peered":"2024-04-26T23:43:07.855266+0000","last_unstale":"2024-04-26T23:43:59.565884+0000","last_undegraded":"2024-04-26T23:43:59.565884+0000","last_fullsized":"2024-04-26T23:43:59.565884+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:06:03.016504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-26T23:44:59.537488+0000","last_change":"2024-04-26T23:43:02.443464+0000","last_active":"2024-04-26T23:44:59.537488+0000","last_peered":"2024-04-26T23:44:59.537488+0000","last_clean":"2024-04-26T23:44:59.537488+0000","last_became_active":"2024-04-26T23:43:02.443170+0000","last_became_peered":"2024-04-26T23:43:02.443170+0000","last_unstale":"2024-04-26T23:44:59.537488+0000","last_undegraded":"2024-04-26T23:44:59.537488+0000","last_fullsized":"2024-04-26T23:44:59.537488+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:45:51.431082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.572506+0000","last_change":"2024-04-26T23:43:03.282714+0000","last_active":"2024-04-26T23:43:59.572506+0000","last_peered":"2024-04-26T23:43:59.572506+0000","last_clean":"2024-04-26T23:43:59.572506+0000","last_became_active":"2024-04-26T23:43:03.282353+0000","last_became_peered":"2024-04-26T23:43:03.282353+0000","last_unstale":"2024-04-26T23:43:59.572506+0000","last_undegraded":"2024-04-26T23:43:59.572506+0000","last_fullsized":"2024-04-26T23:43:59.572506+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:18:24.082946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.576174+0000","last_change":"2024-04-26T23:43:07.543301+0000","last_active":"2024-04-26T23:43:59.576174+0000","last_peered":"2024-04-26T23:43:59.576174+0000","last_clean":"2024-04-26T23:43:59.576174+0000","last_became_active":"2024-04-26T23:43:07.540434+0000","last_became_peered":"2024-04-26T23:43:07.540434+0000","last_unstale":"2024-04-26T23:43:59.576174+0000","last_undegraded":"2024-04-26T23:43:59.576174+0000","last_fullsized":"2024-04-26T23:43:59.576174+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:15:18.542595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.568745+0000","last_change":"2024-04-26T23:43:02.451391+0000","last_active":"2024-04-26T23:43:59.568745+0000","last_peered":"2024-04-26T23:43:59.568745+0000","last_clean":"2024-04-26T23:43:59.568745+0000","last_became_active":"2024-04-26T23:43:02.450735+0000","last_became_peered":"2024-04-26T23:43:02.450735+0000","last_unstale":"2024-04-26T23:43:59.568745+0000","last_undegraded":"2024-04-26T23:43:59.568745+0000","last_fullsized":"2024-04-26T23:43:59.568745+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:57:48.658671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-26T23:44:59.537799+0000","last_change":"2024-04-26T23:43:03.301858+0000","last_active":"2024-04-26T23:44:59.537799+0000","last_peered":"2024-04-26T23:44:59.537799+0000","last_clean":"2024-04-26T23:44:59.537799+0000","last_became_active":"2024-04-26T23:43:03.301474+0000","last_became_peered":"2024-04-26T23:43:03.301474+0000","last_unstale":"2024-04-26T23:44:59.537799+0000","last_undegraded":"2024-04-26T23:44:59.537799+0000","last_fullsized":"2024-04-26T23:44:59.537799+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:24:56.386127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.566096+0000","last_change":"2024-04-26T23:43:07.544512+0000","last_active":"2024-04-26T23:43:59.566096+0000","last_peered":"2024-04-26T23:43:59.566096+0000","last_clean":"2024-04-26T23:43:59.566096+0000","last_became_active":"2024-04-26T23:43:07.544029+0000","last_became_peered":"2024-04-26T23:43:07.544029+0000","last_unstale":"2024-04-26T23:43:59.566096+0000","last_undegraded":"2024-04-26T23:43:59.566096+0000","last_fullsized":"2024-04-26T23:43:59.566096+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:38:08.149123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-26T23:44:59.538555+0000","last_change":"2024-04-26T23:43:59.579721+0000","last_active":"2024-04-26T23:44:59.538555+0000","last_peered":"2024-04-26T23:44:59.538555+0000","last_clean":"2024-04-26T23:44:59.538555+0000","last_became_active":"2024-04-26T23:43:59.576675+0000","last_became_peered":"2024-04-26T23:43:59.576675+0000","last_unstale":"2024-04-26T23:44:59.538555+0000","last_undegraded":"2024-04-26T23:44:59.538555+0000","last_fullsized":"2024-04-26T23:44:59.538555+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:59:30.124137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.571481+0000","last_change":"2024-04-26T23:43:03.307895+0000","last_active":"2024-04-26T23:43:59.571481+0000","last_peered":"2024-04-26T23:43:59.571481+0000","last_clean":"2024-04-26T23:43:59.571481+0000","last_became_active":"2024-04-26T23:43:03.307478+0000","last_became_peered":"2024-04-26T23:43:03.307478+0000","last_unstale":"2024-04-26T23:43:59.571481+0000","last_undegraded":"2024-04-26T23:43:59.571481+0000","last_fullsized":"2024-04-26T23:43:59.571481+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:57:14.356234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.565736+0000","last_change":"2024-04-26T23:43:07.855803+0000","last_active":"2024-04-26T23:43:59.565736+0000","last_peered":"2024-04-26T23:43:59.565736+0000","last_clean":"2024-04-26T23:43:59.565736+0000","last_became_active":"2024-04-26T23:43:07.855200+0000","last_became_peered":"2024-04-26T23:43:07.855200+0000","last_unstale":"2024-04-26T23:43:59.565736+0000","last_undegraded":"2024-04-26T23:43:59.565736+0000","last_fullsized":"2024-04-26T23:43:59.565736+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:02:46.223330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.571815+0000","last_change":"2024-04-26T23:43:02.437513+0000","last_active":"2024-04-26T23:43:59.571815+0000","last_peered":"2024-04-26T23:43:59.571815+0000","last_clean":"2024-04-26T23:43:59.571815+0000","last_became_active":"2024-04-26T23:43:02.437153+0000","last_became_peered":"2024-04-26T23:43:02.437153+0000","last_unstale":"2024-04-26T23:43:59.571815+0000","last_undegraded":"2024-04-26T23:43:59.571815+0000","last_fullsized":"2024-04-26T23:43:59.571815+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:11:45.477579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.572239+0000","last_change":"2024-04-26T23:43:03.297835+0000","last_active":"2024-04-26T23:43:59.572239+0000","last_peered":"2024-04-26T23:43:59.572239+0000","last_clean":"2024-04-26T23:43:59.572239+0000","last_became_active":"2024-04-26T23:43:03.297481+0000","last_became_peered":"2024-04-26T23:43:03.297481+0000","last_unstale":"2024-04-26T23:43:59.572239+0000","last_undegraded":"2024-04-26T23:43:59.572239+0000","last_fullsized":"2024-04-26T23:43:59.572239+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:52:54.757461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-26T23:44:59.537979+0000","last_change":"2024-04-26T23:43:07.536849+0000","last_active":"2024-04-26T23:44:59.537979+0000","last_peered":"2024-04-26T23:44:59.537979+0000","last_clean":"2024-04-26T23:44:59.537979+0000","last_became_active":"2024-04-26T23:43:07.536214+0000","last_became_peered":"2024-04-26T23:43:07.536214+0000","last_unstale":"2024-04-26T23:44:59.537979+0000","last_undegraded":"2024-04-26T23:44:59.537979+0000","last_fullsized":"2024-04-26T23:44:59.537979+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:33:16.782368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.566803+0000","last_change":"2024-04-26T23:43:02.464181+0000","last_active":"2024-04-26T23:43:59.566803+0000","last_peered":"2024-04-26T23:43:59.566803+0000","last_clean":"2024-04-26T23:43:59.566803+0000","last_became_active":"2024-04-26T23:43:02.463830+0000","last_became_peered":"2024-04-26T23:43:02.463830+0000","last_unstale":"2024-04-26T23:43:59.566803+0000","last_undegraded":"2024-04-26T23:43:59.566803+0000","last_fullsized":"2024-04-26T23:43:59.566803+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:14:52.831302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.572417+0000","last_change":"2024-04-26T23:43:03.298497+0000","last_active":"2024-04-26T23:43:59.572417+0000","last_peered":"2024-04-26T23:43:59.572417+0000","last_clean":"2024-04-26T23:43:59.572417+0000","last_became_active":"2024-04-26T23:43:03.298194+0000","last_became_peered":"2024-04-26T23:43:03.298194+0000","last_unstale":"2024-04-26T23:43:59.572417+0000","last_undegraded":"2024-04-26T23:43:59.572417+0000","last_fullsized":"2024-04-26T23:43:59.572417+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:39:19.007294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.567343+0000","last_change":"2024-04-26T23:43:07.527962+0000","last_active":"2024-04-26T23:43:59.567343+0000","last_peered":"2024-04-26T23:43:59.567343+0000","last_clean":"2024-04-26T23:43:59.567343+0000","last_became_active":"2024-04-26T23:43:07.527386+0000","last_became_peered":"2024-04-26T23:43:07.527386+0000","last_unstale":"2024-04-26T23:43:59.567343+0000","last_undegraded":"2024-04-26T23:43:59.567343+0000","last_fullsized":"2024-04-26T23:43:59.567343+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:57:21.711670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.569154+0000","last_change":"2024-04-26T23:43:02.453988+0000","last_active":"2024-04-26T23:43:59.569154+0000","last_peered":"2024-04-26T23:43:59.569154+0000","last_clean":"2024-04-26T23:43:59.569154+0000","last_became_active":"2024-04-26T23:43:02.453654+0000","last_became_peered":"2024-04-26T23:43:02.453654+0000","last_unstale":"2024-04-26T23:43:59.569154+0000","last_undegraded":"2024-04-26T23:43:59.569154+0000","last_fullsized":"2024-04-26T23:43:59.569154+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:46:08.718705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.571441+0000","last_change":"2024-04-26T23:43:03.305861+0000","last_active":"2024-04-26T23:43:59.571441+0000","last_peered":"2024-04-26T23:43:59.571441+0000","last_clean":"2024-04-26T23:43:59.571441+0000","last_became_active":"2024-04-26T23:43:03.305301+0000","last_became_peered":"2024-04-26T23:43:03.305301+0000","last_unstale":"2024-04-26T23:43:59.571441+0000","last_undegraded":"2024-04-26T23:43:59.571441+0000","last_fullsized":"2024-04-26T23:43:59.571441+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:08:24.174443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.568906+0000","last_change":"2024-04-26T23:43:07.529914+0000","last_active":"2024-04-26T23:43:59.568906+0000","last_peered":"2024-04-26T23:43:59.568906+0000","last_clean":"2024-04-26T23:43:59.568906+0000","last_became_active":"2024-04-26T23:43:07.529509+0000","last_became_peered":"2024-04-26T23:43:07.529509+0000","last_unstale":"2024-04-26T23:43:59.568906+0000","last_undegraded":"2024-04-26T23:43:59.568906+0000","last_fullsized":"2024-04-26T23:43:59.568906+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:50:23.293672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.868745+0000","last_change":"2024-04-26T23:43:02.452096+0000","last_active":"2024-04-26T23:43:59.868745+0000","last_peered":"2024-04-26T23:43:59.868745+0000","last_clean":"2024-04-26T23:43:59.868745+0000","last_became_active":"2024-04-26T23:43:02.450039+0000","last_became_peered":"2024-04-26T23:43:02.450039+0000","last_unstale":"2024-04-26T23:43:59.868745+0000","last_undegraded":"2024-04-26T23:43:59.868745+0000","last_fullsized":"2024-04-26T23:43:59.868745+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:47:03.062575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.572594+0000","last_change":"2024-04-26T23:43:03.294790+0000","last_active":"2024-04-26T23:43:59.572594+0000","last_peered":"2024-04-26T23:43:59.572594+0000","last_clean":"2024-04-26T23:43:59.572594+0000","last_became_active":"2024-04-26T23:43:03.294460+0000","last_became_peered":"2024-04-26T23:43:03.294460+0000","last_unstale":"2024-04-26T23:43:59.572594+0000","last_undegraded":"2024-04-26T23:43:59.572594+0000","last_fullsized":"2024-04-26T23:43:59.572594+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:57:40.879315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-26T23:44:59.537779+0000","last_change":"2024-04-26T23:43:03.284344+0000","last_active":"2024-04-26T23:44:59.537779+0000","last_peered":"2024-04-26T23:44:59.537779+0000","last_clean":"2024-04-26T23:44:59.537779+0000","last_became_active":"2024-04-26T23:43:03.283954+0000","last_became_peered":"2024-04-26T23:43:03.283954+0000","last_unstale":"2024-04-26T23:44:59.537779+0000","last_undegraded":"2024-04-26T23:44:59.537779+0000","last_fullsized":"2024-04-26T23:44:59.537779+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:12:56.516067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.574959+0000","last_change":"2024-04-26T23:43:59.574959+0000","last_active":"2024-04-26T23:43:59.574959+0000","last_peered":"2024-04-26T23:43:59.574959+0000","last_clean":"2024-04-26T23:43:59.574959+0000","last_became_active":"2024-04-26T23:43:59.574588+0000","last_became_peered":"2024-04-26T23:43:59.574588+0000","last_unstale":"2024-04-26T23:43:59.574959+0000","last_undegraded":"2024-04-26T23:43:59.574959+0000","last_fullsized":"2024-04-26T23:43:59.574959+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:52:56.689439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.574946+0000","last_change":"2024-04-26T23:43:59.574946+0000","last_active":"2024-04-26T23:43:59.574946+0000","last_peered":"2024-04-26T23:43:59.574946+0000","last_clean":"2024-04-26T23:43:59.574946+0000","last_became_active":"2024-04-26T23:43:59.574238+0000","last_became_peered":"2024-04-26T23:43:59.574238+0000","last_unstale":"2024-04-26T23:43:59.574946+0000","last_undegraded":"2024-04-26T23:43:59.574946+0000","last_fullsized":"2024-04-26T23:43:59.574946+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:58:41.619868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,0],"acting":[2,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.567529+0000","last_change":"2024-04-26T23:43:03.304580+0000","last_active":"2024-04-26T23:43:59.567529+0000","last_peered":"2024-04-26T23:43:59.567529+0000","last_clean":"2024-04-26T23:43:59.567529+0000","last_became_active":"2024-04-26T23:43:03.303128+0000","last_became_peered":"2024-04-26T23:43:03.303128+0000","last_unstale":"2024-04-26T23:43:59.567529+0000","last_undegraded":"2024-04-26T23:43:59.567529+0000","last_fullsized":"2024-04-26T23:43:59.567529+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:11:36.868298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.868705+0000","last_change":"2024-04-26T23:43:03.306441+0000","last_active":"2024-04-26T23:43:59.868705+0000","last_peered":"2024-04-26T23:43:59.868705+0000","last_clean":"2024-04-26T23:43:59.868705+0000","last_became_active":"2024-04-26T23:43:03.305209+0000","last_became_peered":"2024-04-26T23:43:03.305209+0000","last_unstale":"2024-04-26T23:43:59.868705+0000","last_undegraded":"2024-04-26T23:43:59.868705+0000","last_fullsized":"2024-04-26T23:43:59.868705+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:41:39.757001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-26T23:44:59.538411+0000","last_change":"2024-04-26T23:43:03.284291+0000","last_active":"2024-04-26T23:44:59.538411+0000","last_peered":"2024-04-26T23:44:59.538411+0000","last_clean":"2024-04-26T23:44:59.538411+0000","last_became_active":"2024-04-26T23:43:03.284006+0000","last_became_peered":"2024-04-26T23:43:03.284006+0000","last_unstale":"2024-04-26T23:44:59.538411+0000","last_undegraded":"2024-04-26T23:44:59.538411+0000","last_fullsized":"2024-04-26T23:44:59.538411+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:24:01.386322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.567715+0000","last_change":"2024-04-26T23:43:03.286478+0000","last_active":"2024-04-26T23:43:59.567715+0000","last_peered":"2024-04-26T23:43:59.567715+0000","last_clean":"2024-04-26T23:43:59.567715+0000","last_became_active":"2024-04-26T23:43:03.286115+0000","last_became_peered":"2024-04-26T23:43:03.286115+0000","last_unstale":"2024-04-26T23:43:59.567715+0000","last_undegraded":"2024-04-26T23:43:59.567715+0000","last_fullsized":"2024-04-26T23:43:59.567715+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:34:57.534124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.872042+0000","last_change":"2024-04-26T23:43:59.872042+0000","last_active":"2024-04-26T23:43:59.872042+0000","last_peered":"2024-04-26T23:43:59.872042+0000","last_clean":"2024-04-26T23:43:59.872042+0000","last_became_active":"2024-04-26T23:43:59.870288+0000","last_became_peered":"2024-04-26T23:43:59.870288+0000","last_unstale":"2024-04-26T23:43:59.872042+0000","last_undegraded":"2024-04-26T23:43:59.872042+0000","last_fullsized":"2024-04-26T23:43:59.872042+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:16:38.518370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.569276+0000","last_change":"2024-04-26T23:43:03.300602+0000","last_active":"2024-04-26T23:43:59.569276+0000","last_peered":"2024-04-26T23:43:59.569276+0000","last_clean":"2024-04-26T23:43:59.569276+0000","last_became_active":"2024-04-26T23:43:03.300225+0000","last_became_peered":"2024-04-26T23:43:03.300225+0000","last_unstale":"2024-04-26T23:43:59.569276+0000","last_undegraded":"2024-04-26T23:43:59.569276+0000","last_fullsized":"2024-04-26T23:43:59.569276+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:52:36.638153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.569164+0000","last_change":"2024-04-26T23:43:03.305631+0000","last_active":"2024-04-26T23:43:59.569164+0000","last_peered":"2024-04-26T23:43:59.569164+0000","last_clean":"2024-04-26T23:43:59.569164+0000","last_became_active":"2024-04-26T23:43:03.300193+0000","last_became_peered":"2024-04-26T23:43:03.300193+0000","last_unstale":"2024-04-26T23:43:59.569164+0000","last_undegraded":"2024-04-26T23:43:59.569164+0000","last_fullsized":"2024-04-26T23:43:59.569164+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:27:08.355753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-26T23:44:59.538596+0000","last_change":"2024-04-26T23:43:03.309514+0000","last_active":"2024-04-26T23:44:59.538596+0000","last_peered":"2024-04-26T23:44:59.538596+0000","last_clean":"2024-04-26T23:44:59.538596+0000","last_became_active":"2024-04-26T23:43:03.309085+0000","last_became_peered":"2024-04-26T23:43:03.309085+0000","last_unstale":"2024-04-26T23:44:59.538596+0000","last_undegraded":"2024-04-26T23:44:59.538596+0000","last_fullsized":"2024-04-26T23:44:59.538596+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:08:45.649513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.869023+0000","last_change":"2024-04-26T23:43:03.303553+0000","last_active":"2024-04-26T23:43:59.869023+0000","last_peered":"2024-04-26T23:43:59.869023+0000","last_clean":"2024-04-26T23:43:59.869023+0000","last_became_active":"2024-04-26T23:43:03.302979+0000","last_became_peered":"2024-04-26T23:43:03.302979+0000","last_unstale":"2024-04-26T23:43:59.869023+0000","last_undegraded":"2024-04-26T23:43:59.869023+0000","last_fullsized":"2024-04-26T23:43:59.869023+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:44:25.809138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-26T23:44:59.538194+0000","last_change":"2024-04-26T23:43:03.288433+0000","last_active":"2024-04-26T23:44:59.538194+0000","last_peered":"2024-04-26T23:44:59.538194+0000","last_clean":"2024-04-26T23:44:59.538194+0000","last_became_active":"2024-04-26T23:43:03.288125+0000","last_became_peered":"2024-04-26T23:43:03.288125+0000","last_unstale":"2024-04-26T23:44:59.538194+0000","last_undegraded":"2024-04-26T23:44:59.538194+0000","last_fullsized":"2024-04-26T23:44:59.538194+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:53:21.673755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":282161,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":59218,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":560,"num_read_kb":342,"num_write":161,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":177654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140,"ondisk_log_size":140,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383719,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1007616,"data_stored":510413,"data_compressed":23547,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645355,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32288,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712864,"statfs":{"total":95995035648,"available":95961972736,"internally_reserved":0,"allocated":1134592,"data_stored":1143316,"data_compressed":56460,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808880,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1015808,"data_stored":505163,"data_compressed":20573,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070518,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1011712,"data_stored":500987,"data_compressed":20573,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332157,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32220,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712932,"statfs":{"total":95995035648,"available":95962042368,"internally_reserved":0,"allocated":1060864,"data_stored":1097287,"data_compressed":56460,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495682,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":991232,"data_stored":496243,"data_compressed":20573,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574943,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1064960,"data_stored":560561,"data_compressed":23547,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738467,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1015808,"data_stored":497631,"data_compressed":20573,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411418,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":552608,"data_compressed":20573,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182482,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1003520,"data_stored":496437,"data_compressed":20573,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920846,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1011712,"data_stored":501396,"data_compressed":20573,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757321,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32220,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712932,"statfs":{"total":95995035648,"available":95962042368,"internally_reserved":0,"allocated":1069056,"data_stored":1092484,"data_compressed":56460,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34242,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34242,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34242,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35887,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35887,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35887,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35887,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35887,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":54826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":56455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":56795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T23:45:02.560 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T23:45:02.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:02 smithi072 ceph-mon[28184]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-26T23:45:02.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:02 smithi076 ceph-mon[32774]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-26T23:45:03.043 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:45:03.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:02 smithi136 ceph-mon[33050]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-26T23:45:03.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:03 smithi072 ceph-mon[28184]: from='client.25058 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T23:45:03.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:03 smithi072 ceph-mon[28184]: pgmap v385: 129 pgs: 1 peering, 128 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 127 B/s wr, 3 op/s 2024-04-26T23:45:03.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:03 smithi076 ceph-mon[32774]: from='client.25058 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T23:45:03.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:03 smithi076 ceph-mon[32774]: pgmap v385: 129 pgs: 1 peering, 128 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 127 B/s wr, 3 op/s 2024-04-26T23:45:04.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:03 smithi136 ceph-mon[33050]: from='client.25058 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T23:45:04.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:03 smithi136 ceph-mon[33050]: pgmap v385: 129 pgs: 1 peering, 128 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 127 B/s wr, 3 op/s 2024-04-26T23:45:05.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.252+0000 7f7da1a00700 1 -- 172.21.15.72:0/87904625 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d9c1094c0 msgr2=0x7f7d9c1098a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:05.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.252+0000 7f7da1a00700 1 --2- 172.21.15.72:0/87904625 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d9c1094c0 0x7f7d9c1098a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c009990 tx=0x7f7d8c009ca0 comp rx=0 tx=0).stop 2024-04-26T23:45:05.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.252+0000 7f7da1a00700 1 -- 172.21.15.72:0/87904625 shutdown_connections 2024-04-26T23:45:05.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.252+0000 7f7da1a00700 1 --2- 172.21.15.72:0/87904625 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d9c103e30 0x7f7d9c110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:05.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.252+0000 7f7da1a00700 1 --2- 172.21.15.72:0/87904625 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d9c1094c0 0x7f7d9c1098a0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:05.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.252+0000 7f7da1a00700 1 --2- 172.21.15.72:0/87904625 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7d9c103470 0x7f7d9c1038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:05.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.252+0000 7f7da1a00700 1 -- 172.21.15.72:0/87904625 >> 172.21.15.72:0/87904625 conn(0x7f7d9c07a580 msgr2=0x7f7d9c07a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:05.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.252+0000 7f7da1a00700 1 -- 172.21.15.72:0/87904625 shutdown_connections 2024-04-26T23:45:05.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.252+0000 7f7da1a00700 1 -- 172.21.15.72:0/87904625 wait complete. 2024-04-26T23:45:05.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.253+0000 7f7da1a00700 1 Processor -- start 2024-04-26T23:45:05.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.253+0000 7f7da1a00700 1 -- start start 2024-04-26T23:45:05.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.254+0000 7f7da1a00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d9c103470 0x7f7d9c19bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:05.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.254+0000 7f7da1a00700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d9c103e30 0x7f7d9c19c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:05.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.254+0000 7f7da1a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7d9c1094c0 0x7f7d9c1a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:05.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.254+0000 7f7da1a00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f7d9c113610 con 0x7f7d9c103470 2024-04-26T23:45:05.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.254+0000 7f7da1a00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f7d9c113490 con 0x7f7d9c1094c0 2024-04-26T23:45:05.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.254+0000 7f7da1a00700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f7d9c113790 con 0x7f7d9c103e30 2024-04-26T23:45:05.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.254+0000 7f7d9aa00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d9c103470 0x7f7d9c19bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:05.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.254+0000 7f7d9aa00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d9c103470 0x7f7d9c19bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56148/0 (socket says 172.21.15.72:56148) 2024-04-26T23:45:05.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.254+0000 7f7d9aa00700 1 -- 172.21.15.72:0/4213205387 learned_addr learned my addr 172.21.15.72:0/4213205387 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:45:05.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.254+0000 7f7d9b400700 1 --2- 172.21.15.72:0/4213205387 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7d9c1094c0 0x7f7d9c1a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:05.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.254+0000 7f7d9a000700 1 --2- 172.21.15.72:0/4213205387 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d9c103e30 0x7f7d9c19c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:05.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.255+0000 7f7d9b400700 1 -- 172.21.15.72:0/4213205387 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d9c103e30 msgr2=0x7f7d9c19c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:05.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.255+0000 7f7d9b400700 1 --2- 172.21.15.72:0/4213205387 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d9c103e30 0x7f7d9c19c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:05.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.255+0000 7f7d9b400700 1 -- 172.21.15.72:0/4213205387 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d9c103470 msgr2=0x7f7d9c19bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:05.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.255+0000 7f7d9b400700 1 --2- 172.21.15.72:0/4213205387 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d9c103470 0x7f7d9c19bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:05.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.255+0000 7f7d9b400700 1 -- 172.21.15.72:0/4213205387 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d9c1a0d70 con 0x7f7d9c1094c0 2024-04-26T23:45:05.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.255+0000 7f7d9aa00700 1 --2- 172.21.15.72:0/4213205387 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d9c103470 0x7f7d9c19bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:45:05.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.255+0000 7f7d9b400700 1 --2- 172.21.15.72:0/4213205387 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7d9c1094c0 0x7f7d9c1a0630 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f7d9400efc0 tx=0x7f7d9400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:05.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.255+0000 7f7d93400700 1 -- 172.21.15.72:0/4213205387 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d940092e0 con 0x7f7d9c1094c0 2024-04-26T23:45:05.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.255+0000 7f7da1a00700 1 -- 172.21.15.72:0/4213205387 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d9c1a1000 con 0x7f7d9c1094c0 2024-04-26T23:45:05.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.255+0000 7f7d93400700 1 -- 172.21.15.72:0/4213205387 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d94004bb0 con 0x7f7d9c1094c0 2024-04-26T23:45:05.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.255+0000 7f7d93400700 1 -- 172.21.15.72:0/4213205387 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d94004020 con 0x7f7d9c1094c0 2024-04-26T23:45:05.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.256+0000 7f7da1a00700 1 -- 172.21.15.72:0/4213205387 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d9c1a88e0 con 0x7f7d9c1094c0 2024-04-26T23:45:05.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.257+0000 7f7da1a00700 1 -- 172.21.15.72:0/4213205387 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d9c06a110 con 0x7f7d9c1094c0 2024-04-26T23:45:05.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.259+0000 7f7d93400700 1 -- 172.21.15.72:0/4213205387 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7d9401b030 con 0x7f7d9c1094c0 2024-04-26T23:45:05.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.260+0000 7f7d93400700 1 --2- 172.21.15.72:0/4213205387 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f7d800a3cd0 0x7f7d800a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:05.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.260+0000 7f7d93400700 1 -- 172.21.15.72:0/4213205387 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f7d940d0410 con 0x7f7d9c1094c0 2024-04-26T23:45:05.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.260+0000 7f7d9aa00700 1 --2- 172.21.15.72:0/4213205387 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f7d800a3cd0 0x7f7d800a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:05.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.261+0000 7f7d9aa00700 1 --2- 172.21.15.72:0/4213205387 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f7d800a3cd0 0x7f7d800a6190 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c006010 tx=0x7f7d8c00b280 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:05.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.265+0000 7f7d93400700 1 -- 172.21.15.72:0/4213205387 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7d94022020 con 0x7f7d9c1094c0 2024-04-26T23:45:05.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.485+0000 7f7da1a00700 1 -- 172.21.15.72:0/4213205387 --> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7d9c110000 con 0x7f7d800a3cd0 2024-04-26T23:45:05.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.500+0000 7f7d93400700 1 -- 172.21.15.72:0/4213205387 <== mgr.14152 v2:172.21.15.72:6800/4281177709 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340662 (secure 0 0 0) 0x7f7d9c110000 con 0x7f7d800a3cd0 2024-04-26T23:45:05.508 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:45:05.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.507+0000 7f7da1a00700 1 -- 172.21.15.72:0/4213205387 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f7d800a3cd0 msgr2=0x7f7d800a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:05.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.507+0000 7f7da1a00700 1 --2- 172.21.15.72:0/4213205387 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f7d800a3cd0 0x7f7d800a6190 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c006010 tx=0x7f7d8c00b280 comp rx=0 tx=0).stop 2024-04-26T23:45:05.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.507+0000 7f7da1a00700 1 -- 172.21.15.72:0/4213205387 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7d9c1094c0 msgr2=0x7f7d9c1a0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:05.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.507+0000 7f7da1a00700 1 --2- 172.21.15.72:0/4213205387 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7d9c1094c0 0x7f7d9c1a0630 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f7d9400efc0 tx=0x7f7d9400c3a0 comp rx=0 tx=0).stop 2024-04-26T23:45:05.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.507+0000 7f7da1a00700 1 -- 172.21.15.72:0/4213205387 shutdown_connections 2024-04-26T23:45:05.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.507+0000 7f7da1a00700 1 --2- 172.21.15.72:0/4213205387 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f7d800a3cd0 0x7f7d800a6190 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:05.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.507+0000 7f7da1a00700 1 --2- 172.21.15.72:0/4213205387 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7d9c1094c0 0x7f7d9c1a0630 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:05.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.507+0000 7f7da1a00700 1 --2- 172.21.15.72:0/4213205387 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d9c103470 0x7f7d9c19bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:05.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.507+0000 7f7da1a00700 1 --2- 172.21.15.72:0/4213205387 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d9c103e30 0x7f7d9c19c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:05.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.507+0000 7f7da1a00700 1 -- 172.21.15.72:0/4213205387 >> 172.21.15.72:0/4213205387 conn(0x7f7d9c07a580 msgr2=0x7f7d9c10e8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:05.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.508+0000 7f7da1a00700 1 -- 172.21.15.72:0/4213205387 shutdown_connections 2024-04-26T23:45:05.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:05.508+0000 7f7da1a00700 1 -- 172.21.15.72:0/4213205387 wait complete. 2024-04-26T23:45:05.513 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-04-26T23:45:05.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:05 smithi072 ceph-mon[28184]: pgmap v386: 129 pgs: 1 peering, 128 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 127 B/s wr, 7 op/s 2024-04-26T23:45:05.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:05 smithi076 ceph-mon[32774]: pgmap v386: 129 pgs: 1 peering, 128 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 127 B/s wr, 7 op/s 2024-04-26T23:45:06.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:05 smithi136 ceph-mon[33050]: pgmap v386: 129 pgs: 1 peering, 128 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 127 B/s wr, 7 op/s 2024-04-26T23:45:06.353 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":386,"stamp":"2024-04-26T23:45:04.463313+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":649820,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":716,"num_read_kb":466,"num_write":347,"num_write_kb":3050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":1180848,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":280,"ondisk_log_size":280,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":357388,"kb_used_data":12332,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124584436,"statfs":{"total":1151940427776,"available":1151574462464,"internally_reserved":0,"allocated":12627968,"data_stored":8088044,"data_compressed":402163,"data_compressed_allocated":544768,"data_compressed_original":2588672,"omap_allocated":19139,"internal_metadata":353154365},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":59,"num_read_kb":29,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003959"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-26T23:44:59.538966+0000","last_change":"2024-04-26T23:43:03.301992+0000","last_active":"2024-04-26T23:44:59.538966+0000","last_peered":"2024-04-26T23:44:59.538966+0000","last_clean":"2024-04-26T23:44:59.538966+0000","last_became_active":"2024-04-26T23:43:03.301498+0000","last_became_peered":"2024-04-26T23:43:03.301498+0000","last_unstale":"2024-04-26T23:44:59.538966+0000","last_undegraded":"2024-04-26T23:44:59.538966+0000","last_fullsized":"2024-04-26T23:44:59.538966+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:40:53.567805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.548533+0000","last_change":"2024-04-26T23:43:03.284964+0000","last_active":"2024-04-26T23:45:00.548533+0000","last_peered":"2024-04-26T23:45:00.548533+0000","last_clean":"2024-04-26T23:45:00.548533+0000","last_became_active":"2024-04-26T23:43:03.284185+0000","last_became_peered":"2024-04-26T23:43:03.284185+0000","last_unstale":"2024-04-26T23:45:00.548533+0000","last_undegraded":"2024-04-26T23:45:00.548533+0000","last_fullsized":"2024-04-26T23:45:00.548533+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:55:35.325775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.547061+0000","last_change":"2024-04-26T23:43:03.300333+0000","last_active":"2024-04-26T23:45:00.547061+0000","last_peered":"2024-04-26T23:45:00.547061+0000","last_clean":"2024-04-26T23:45:00.547061+0000","last_became_active":"2024-04-26T23:43:03.295779+0000","last_became_peered":"2024-04-26T23:43:03.295779+0000","last_unstale":"2024-04-26T23:45:00.547061+0000","last_undegraded":"2024-04-26T23:45:00.547061+0000","last_fullsized":"2024-04-26T23:45:00.547061+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:45:06.598862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.541449+0000","last_change":"2024-04-26T23:43:03.290211+0000","last_active":"2024-04-26T23:45:00.541449+0000","last_peered":"2024-04-26T23:45:00.541449+0000","last_clean":"2024-04-26T23:45:00.541449+0000","last_became_active":"2024-04-26T23:43:03.289853+0000","last_became_peered":"2024-04-26T23:43:03.289853+0000","last_unstale":"2024-04-26T23:45:00.541449+0000","last_undegraded":"2024-04-26T23:45:00.541449+0000","last_fullsized":"2024-04-26T23:45:00.541449+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:14:45.032300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.543721+0000","last_change":"2024-04-26T23:43:03.309667+0000","last_active":"2024-04-26T23:45:00.543721+0000","last_peered":"2024-04-26T23:45:00.543721+0000","last_clean":"2024-04-26T23:45:00.543721+0000","last_became_active":"2024-04-26T23:43:03.309305+0000","last_became_peered":"2024-04-26T23:43:03.309305+0000","last_unstale":"2024-04-26T23:45:00.543721+0000","last_undegraded":"2024-04-26T23:45:00.543721+0000","last_fullsized":"2024-04-26T23:45:00.543721+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:31:56.739866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.548338+0000","last_change":"2024-04-26T23:43:03.292107+0000","last_active":"2024-04-26T23:45:00.548338+0000","last_peered":"2024-04-26T23:45:00.548338+0000","last_clean":"2024-04-26T23:45:00.548338+0000","last_became_active":"2024-04-26T23:43:03.291760+0000","last_became_peered":"2024-04-26T23:43:03.291760+0000","last_unstale":"2024-04-26T23:45:00.548338+0000","last_undegraded":"2024-04-26T23:45:00.548338+0000","last_fullsized":"2024-04-26T23:45:00.548338+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:31:54.654379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.571420+0000","last_change":"2024-04-26T23:43:03.294923+0000","last_active":"2024-04-26T23:43:59.571420+0000","last_peered":"2024-04-26T23:43:59.571420+0000","last_clean":"2024-04-26T23:43:59.571420+0000","last_became_active":"2024-04-26T23:43:03.294523+0000","last_became_peered":"2024-04-26T23:43:03.294523+0000","last_unstale":"2024-04-26T23:43:59.571420+0000","last_undegraded":"2024-04-26T23:43:59.571420+0000","last_fullsized":"2024-04-26T23:43:59.571420+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:36:19.419773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.542769+0000","last_change":"2024-04-26T23:43:03.301335+0000","last_active":"2024-04-26T23:45:00.542769+0000","last_peered":"2024-04-26T23:45:00.542769+0000","last_clean":"2024-04-26T23:45:00.542769+0000","last_became_active":"2024-04-26T23:43:03.300918+0000","last_became_peered":"2024-04-26T23:43:03.300918+0000","last_unstale":"2024-04-26T23:45:00.542769+0000","last_undegraded":"2024-04-26T23:45:00.542769+0000","last_fullsized":"2024-04-26T23:45:00.542769+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:15:05.719452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-26T23:44:59.539520+0000","last_change":"2024-04-26T23:43:59.574776+0000","last_active":"2024-04-26T23:44:59.539520+0000","last_peered":"2024-04-26T23:44:59.539520+0000","last_clean":"2024-04-26T23:44:59.539520+0000","last_became_active":"2024-04-26T23:43:59.574299+0000","last_became_peered":"2024-04-26T23:43:59.574299+0000","last_unstale":"2024-04-26T23:44:59.539520+0000","last_undegraded":"2024-04-26T23:44:59.539520+0000","last_fullsized":"2024-04-26T23:44:59.539520+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:14:16.089377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,11],"acting":[1,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.548713+0000","last_change":"2024-04-26T23:43:03.287643+0000","last_active":"2024-04-26T23:45:00.548713+0000","last_peered":"2024-04-26T23:45:00.548713+0000","last_clean":"2024-04-26T23:45:00.548713+0000","last_became_active":"2024-04-26T23:43:03.287324+0000","last_became_peered":"2024-04-26T23:43:03.287324+0000","last_unstale":"2024-04-26T23:45:00.548713+0000","last_undegraded":"2024-04-26T23:45:00.548713+0000","last_fullsized":"2024-04-26T23:45:00.548713+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:50:14.816548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.572333+0000","last_change":"2024-04-26T23:43:03.296421+0000","last_active":"2024-04-26T23:43:59.572333+0000","last_peered":"2024-04-26T23:43:59.572333+0000","last_clean":"2024-04-26T23:43:59.572333+0000","last_became_active":"2024-04-26T23:43:03.296060+0000","last_became_peered":"2024-04-26T23:43:03.296060+0000","last_unstale":"2024-04-26T23:43:59.572333+0000","last_undegraded":"2024-04-26T23:43:59.572333+0000","last_fullsized":"2024-04-26T23:43:59.572333+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:50:44.393446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.541254+0000","last_change":"2024-04-26T23:43:03.288562+0000","last_active":"2024-04-26T23:45:00.541254+0000","last_peered":"2024-04-26T23:45:00.541254+0000","last_clean":"2024-04-26T23:45:00.541254+0000","last_became_active":"2024-04-26T23:43:03.288219+0000","last_became_peered":"2024-04-26T23:43:03.288219+0000","last_unstale":"2024-04-26T23:45:00.541254+0000","last_undegraded":"2024-04-26T23:45:00.541254+0000","last_fullsized":"2024-04-26T23:45:00.541254+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:54:32.107146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.547419+0000","last_change":"2024-04-26T23:43:03.296840+0000","last_active":"2024-04-26T23:45:00.547419+0000","last_peered":"2024-04-26T23:45:00.547419+0000","last_clean":"2024-04-26T23:45:00.547419+0000","last_became_active":"2024-04-26T23:43:03.296501+0000","last_became_peered":"2024-04-26T23:43:03.296501+0000","last_unstale":"2024-04-26T23:45:00.547419+0000","last_undegraded":"2024-04-26T23:45:00.547419+0000","last_fullsized":"2024-04-26T23:45:00.547419+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:07:05.293388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.546627+0000","last_change":"2024-04-26T23:43:03.291739+0000","last_active":"2024-04-26T23:45:00.546627+0000","last_peered":"2024-04-26T23:45:00.546627+0000","last_clean":"2024-04-26T23:45:00.546627+0000","last_became_active":"2024-04-26T23:43:03.287839+0000","last_became_peered":"2024-04-26T23:43:03.287839+0000","last_unstale":"2024-04-26T23:45:00.546627+0000","last_undegraded":"2024-04-26T23:45:00.546627+0000","last_fullsized":"2024-04-26T23:45:00.546627+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:10:56.698324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.546051+0000","last_change":"2024-04-26T23:43:03.295767+0000","last_active":"2024-04-26T23:45:00.546051+0000","last_peered":"2024-04-26T23:45:00.546051+0000","last_clean":"2024-04-26T23:45:00.546051+0000","last_became_active":"2024-04-26T23:43:03.295339+0000","last_became_peered":"2024-04-26T23:43:03.295339+0000","last_unstale":"2024-04-26T23:45:00.546051+0000","last_undegraded":"2024-04-26T23:45:00.546051+0000","last_fullsized":"2024-04-26T23:45:00.546051+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:21:44.410953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-26T23:44:59.537359+0000","last_change":"2024-04-26T23:43:03.301687+0000","last_active":"2024-04-26T23:44:59.537359+0000","last_peered":"2024-04-26T23:44:59.537359+0000","last_clean":"2024-04-26T23:44:59.537359+0000","last_became_active":"2024-04-26T23:43:03.301355+0000","last_became_peered":"2024-04-26T23:43:03.301355+0000","last_unstale":"2024-04-26T23:44:59.537359+0000","last_undegraded":"2024-04-26T23:44:59.537359+0000","last_fullsized":"2024-04-26T23:44:59.537359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:00:57.991022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-26T23:44:59.538842+0000","last_change":"2024-04-26T23:43:03.302109+0000","last_active":"2024-04-26T23:44:59.538842+0000","last_peered":"2024-04-26T23:44:59.538842+0000","last_clean":"2024-04-26T23:44:59.538842+0000","last_became_active":"2024-04-26T23:43:03.301725+0000","last_became_peered":"2024-04-26T23:43:03.301725+0000","last_unstale":"2024-04-26T23:44:59.538842+0000","last_undegraded":"2024-04-26T23:44:59.538842+0000","last_fullsized":"2024-04-26T23:44:59.538842+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:36:44.630125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.557182+0000","last_change":"2024-04-26T23:43:03.298880+0000","last_active":"2024-04-26T23:45:00.557182+0000","last_peered":"2024-04-26T23:45:00.557182+0000","last_clean":"2024-04-26T23:45:00.557182+0000","last_became_active":"2024-04-26T23:43:03.298464+0000","last_became_peered":"2024-04-26T23:43:03.298464+0000","last_unstale":"2024-04-26T23:45:00.557182+0000","last_undegraded":"2024-04-26T23:45:00.557182+0000","last_fullsized":"2024-04-26T23:45:00.557182+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:42:54.745002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.567554+0000","last_change":"2024-04-26T23:43:03.309454+0000","last_active":"2024-04-26T23:43:59.567554+0000","last_peered":"2024-04-26T23:43:59.567554+0000","last_clean":"2024-04-26T23:43:59.567554+0000","last_became_active":"2024-04-26T23:43:03.309091+0000","last_became_peered":"2024-04-26T23:43:03.309091+0000","last_unstale":"2024-04-26T23:43:59.567554+0000","last_undegraded":"2024-04-26T23:43:59.567554+0000","last_fullsized":"2024-04-26T23:43:59.567554+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:46:12.333410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.566347+0000","last_change":"2024-04-26T23:43:07.541317+0000","last_active":"2024-04-26T23:43:59.566347+0000","last_peered":"2024-04-26T23:43:59.566347+0000","last_clean":"2024-04-26T23:43:59.566347+0000","last_became_active":"2024-04-26T23:43:07.540760+0000","last_became_peered":"2024-04-26T23:43:07.540760+0000","last_unstale":"2024-04-26T23:43:59.566347+0000","last_undegraded":"2024-04-26T23:43:59.566347+0000","last_fullsized":"2024-04-26T23:43:59.566347+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:28:55.545416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.557109+0000","last_change":"2024-04-26T23:43:02.450956+0000","last_active":"2024-04-26T23:45:00.557109+0000","last_peered":"2024-04-26T23:45:00.557109+0000","last_clean":"2024-04-26T23:45:00.557109+0000","last_became_active":"2024-04-26T23:43:02.446733+0000","last_became_peered":"2024-04-26T23:43:02.446733+0000","last_unstale":"2024-04-26T23:45:00.557109+0000","last_undegraded":"2024-04-26T23:45:00.557109+0000","last_fullsized":"2024-04-26T23:45:00.557109+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:58:47.613575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.551817+0000","last_change":"2024-04-26T23:43:03.311681+0000","last_active":"2024-04-26T23:45:00.551817+0000","last_peered":"2024-04-26T23:45:00.551817+0000","last_clean":"2024-04-26T23:45:00.551817+0000","last_became_active":"2024-04-26T23:43:03.311087+0000","last_became_peered":"2024-04-26T23:43:03.311087+0000","last_unstale":"2024-04-26T23:45:00.551817+0000","last_undegraded":"2024-04-26T23:45:00.551817+0000","last_fullsized":"2024-04-26T23:45:00.551817+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:13:00.744062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.540679+0000","last_change":"2024-04-26T23:43:07.543547+0000","last_active":"2024-04-26T23:45:00.540679+0000","last_peered":"2024-04-26T23:45:00.540679+0000","last_clean":"2024-04-26T23:45:00.540679+0000","last_became_active":"2024-04-26T23:43:07.543073+0000","last_became_peered":"2024-04-26T23:43:07.543073+0000","last_unstale":"2024-04-26T23:45:00.540679+0000","last_undegraded":"2024-04-26T23:45:00.540679+0000","last_fullsized":"2024-04-26T23:45:00.540679+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:30:05.418875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.552100+0000","last_change":"2024-04-26T23:43:02.460459+0000","last_active":"2024-04-26T23:45:00.552100+0000","last_peered":"2024-04-26T23:45:00.552100+0000","last_clean":"2024-04-26T23:45:00.552100+0000","last_became_active":"2024-04-26T23:43:02.459652+0000","last_became_peered":"2024-04-26T23:43:02.459652+0000","last_unstale":"2024-04-26T23:45:00.552100+0000","last_undegraded":"2024-04-26T23:45:00.552100+0000","last_fullsized":"2024-04-26T23:45:00.552100+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:27:19.166748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.549553+0000","last_change":"2024-04-26T23:43:03.300078+0000","last_active":"2024-04-26T23:45:00.549553+0000","last_peered":"2024-04-26T23:45:00.549553+0000","last_clean":"2024-04-26T23:45:00.549553+0000","last_became_active":"2024-04-26T23:43:03.296319+0000","last_became_peered":"2024-04-26T23:43:03.296319+0000","last_unstale":"2024-04-26T23:45:00.549553+0000","last_undegraded":"2024-04-26T23:45:00.549553+0000","last_fullsized":"2024-04-26T23:45:00.549553+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:33:51.859539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.540906+0000","last_change":"2024-04-26T23:43:07.543494+0000","last_active":"2024-04-26T23:45:00.540906+0000","last_peered":"2024-04-26T23:45:00.540906+0000","last_clean":"2024-04-26T23:45:00.540906+0000","last_became_active":"2024-04-26T23:43:07.542806+0000","last_became_peered":"2024-04-26T23:43:07.542806+0000","last_unstale":"2024-04-26T23:45:00.540906+0000","last_undegraded":"2024-04-26T23:45:00.540906+0000","last_fullsized":"2024-04-26T23:45:00.540906+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:54:40.522615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544910+0000","last_change":"2024-04-26T23:43:02.438943+0000","last_active":"2024-04-26T23:45:00.544910+0000","last_peered":"2024-04-26T23:45:00.544910+0000","last_clean":"2024-04-26T23:45:00.544910+0000","last_became_active":"2024-04-26T23:43:02.438604+0000","last_became_peered":"2024-04-26T23:43:02.438604+0000","last_unstale":"2024-04-26T23:45:00.544910+0000","last_undegraded":"2024-04-26T23:45:00.544910+0000","last_fullsized":"2024-04-26T23:45:00.544910+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:06:51.054342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-26T23:44:59.538833+0000","last_change":"2024-04-26T23:43:03.293856+0000","last_active":"2024-04-26T23:44:59.538833+0000","last_peered":"2024-04-26T23:44:59.538833+0000","last_clean":"2024-04-26T23:44:59.538833+0000","last_became_active":"2024-04-26T23:43:03.293189+0000","last_became_peered":"2024-04-26T23:43:03.293189+0000","last_unstale":"2024-04-26T23:44:59.538833+0000","last_undegraded":"2024-04-26T23:44:59.538833+0000","last_fullsized":"2024-04-26T23:44:59.538833+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:50:27.545567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.540743+0000","last_change":"2024-04-26T23:43:07.535396+0000","last_active":"2024-04-26T23:45:00.540743+0000","last_peered":"2024-04-26T23:45:00.540743+0000","last_clean":"2024-04-26T23:45:00.540743+0000","last_became_active":"2024-04-26T23:43:07.534914+0000","last_became_peered":"2024-04-26T23:43:07.534914+0000","last_unstale":"2024-04-26T23:45:00.540743+0000","last_undegraded":"2024-04-26T23:45:00.540743+0000","last_fullsized":"2024-04-26T23:45:00.540743+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:27:31.199079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544008+0000","last_change":"2024-04-26T23:43:02.442935+0000","last_active":"2024-04-26T23:45:00.544008+0000","last_peered":"2024-04-26T23:45:00.544008+0000","last_clean":"2024-04-26T23:45:00.544008+0000","last_became_active":"2024-04-26T23:43:02.442588+0000","last_became_peered":"2024-04-26T23:43:02.442588+0000","last_unstale":"2024-04-26T23:45:00.544008+0000","last_undegraded":"2024-04-26T23:45:00.544008+0000","last_fullsized":"2024-04-26T23:45:00.544008+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:05:26.442001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.538765+0000","last_change":"2024-04-26T23:43:03.289407+0000","last_active":"2024-04-26T23:45:00.538765+0000","last_peered":"2024-04-26T23:45:00.538765+0000","last_clean":"2024-04-26T23:45:00.538765+0000","last_became_active":"2024-04-26T23:43:03.287951+0000","last_became_peered":"2024-04-26T23:43:03.287951+0000","last_unstale":"2024-04-26T23:45:00.538765+0000","last_undegraded":"2024-04-26T23:45:00.538765+0000","last_fullsized":"2024-04-26T23:45:00.538765+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:24:42.687653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.567047+0000","last_change":"2024-04-26T23:43:07.527019+0000","last_active":"2024-04-26T23:43:59.567047+0000","last_peered":"2024-04-26T23:43:59.567047+0000","last_clean":"2024-04-26T23:43:59.567047+0000","last_became_active":"2024-04-26T23:43:07.526516+0000","last_became_peered":"2024-04-26T23:43:07.526516+0000","last_unstale":"2024-04-26T23:43:59.567047+0000","last_undegraded":"2024-04-26T23:43:59.567047+0000","last_fullsized":"2024-04-26T23:43:59.567047+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:44:41.267144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'6","reported_seq":134,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:02.343613+0000","last_change":"2024-04-26T23:43:02.445159+0000","last_active":"2024-04-26T23:45:02.343613+0000","last_peered":"2024-04-26T23:45:02.343613+0000","last_clean":"2024-04-26T23:45:02.343613+0000","last_became_active":"2024-04-26T23:43:02.444781+0000","last_became_peered":"2024-04-26T23:43:02.444781+0000","last_unstale":"2024-04-26T23:45:02.343613+0000","last_undegraded":"2024-04-26T23:45:02.343613+0000","last_fullsized":"2024-04-26T23:45:02.343613+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:45:16.509790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":97,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.543904+0000","last_change":"2024-04-26T23:43:03.292717+0000","last_active":"2024-04-26T23:45:00.543904+0000","last_peered":"2024-04-26T23:45:00.543904+0000","last_clean":"2024-04-26T23:45:00.543904+0000","last_became_active":"2024-04-26T23:43:03.292355+0000","last_became_peered":"2024-04-26T23:43:03.292355+0000","last_unstale":"2024-04-26T23:45:00.543904+0000","last_undegraded":"2024-04-26T23:45:00.543904+0000","last_fullsized":"2024-04-26T23:45:00.543904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:06:02.063165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.566558+0000","last_change":"2024-04-26T23:43:07.540758+0000","last_active":"2024-04-26T23:43:59.566558+0000","last_peered":"2024-04-26T23:43:59.566558+0000","last_clean":"2024-04-26T23:43:59.566558+0000","last_became_active":"2024-04-26T23:43:07.540305+0000","last_became_peered":"2024-04-26T23:43:07.540305+0000","last_unstale":"2024-04-26T23:43:59.566558+0000","last_undegraded":"2024-04-26T23:43:59.566558+0000","last_fullsized":"2024-04-26T23:43:59.566558+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:08:04.491544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.548197+0000","last_change":"2024-04-26T23:43:02.453650+0000","last_active":"2024-04-26T23:45:00.548197+0000","last_peered":"2024-04-26T23:45:00.548197+0000","last_clean":"2024-04-26T23:45:00.548197+0000","last_became_active":"2024-04-26T23:43:02.453319+0000","last_became_peered":"2024-04-26T23:43:02.453319+0000","last_unstale":"2024-04-26T23:45:00.548197+0000","last_undegraded":"2024-04-26T23:45:00.548197+0000","last_fullsized":"2024-04-26T23:45:00.548197+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:23:27.570603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.538674+0000","last_change":"2024-04-26T23:43:59.870417+0000","last_active":"2024-04-26T23:45:00.538674+0000","last_peered":"2024-04-26T23:45:00.538674+0000","last_clean":"2024-04-26T23:45:00.538674+0000","last_became_active":"2024-04-26T23:43:59.870053+0000","last_became_peered":"2024-04-26T23:43:59.870053+0000","last_unstale":"2024-04-26T23:45:00.538674+0000","last_undegraded":"2024-04-26T23:45:00.538674+0000","last_fullsized":"2024-04-26T23:45:00.538674+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:07:31.020611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-26T23:44:59.538672+0000","last_change":"2024-04-26T23:43:07.546026+0000","last_active":"2024-04-26T23:44:59.538672+0000","last_peered":"2024-04-26T23:44:59.538672+0000","last_clean":"2024-04-26T23:44:59.538672+0000","last_became_active":"2024-04-26T23:43:07.545668+0000","last_became_peered":"2024-04-26T23:43:07.545668+0000","last_unstale":"2024-04-26T23:44:59.538672+0000","last_undegraded":"2024-04-26T23:44:59.538672+0000","last_fullsized":"2024-04-26T23:44:59.538672+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:54:20.580267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.548684+0000","last_change":"2024-04-26T23:43:02.454937+0000","last_active":"2024-04-26T23:45:00.548684+0000","last_peered":"2024-04-26T23:45:00.548684+0000","last_clean":"2024-04-26T23:45:00.548684+0000","last_became_active":"2024-04-26T23:43:02.454585+0000","last_became_peered":"2024-04-26T23:43:02.454585+0000","last_unstale":"2024-04-26T23:45:00.548684+0000","last_undegraded":"2024-04-26T23:45:00.548684+0000","last_fullsized":"2024-04-26T23:45:00.548684+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:55:54.999411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.554062+0000","last_change":"2024-04-26T23:43:59.870821+0000","last_active":"2024-04-26T23:45:00.554062+0000","last_peered":"2024-04-26T23:45:00.554062+0000","last_clean":"2024-04-26T23:45:00.554062+0000","last_became_active":"2024-04-26T23:43:59.870240+0000","last_became_peered":"2024-04-26T23:43:59.870240+0000","last_unstale":"2024-04-26T23:45:00.554062+0000","last_undegraded":"2024-04-26T23:45:00.554062+0000","last_fullsized":"2024-04-26T23:45:00.554062+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:33:00.284232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,6],"acting":[9,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.548525+0000","last_change":"2024-04-26T23:43:07.535682+0000","last_active":"2024-04-26T23:45:00.548525+0000","last_peered":"2024-04-26T23:45:00.548525+0000","last_clean":"2024-04-26T23:45:00.548525+0000","last_became_active":"2024-04-26T23:43:07.535221+0000","last_became_peered":"2024-04-26T23:43:07.535221+0000","last_unstale":"2024-04-26T23:45:00.548525+0000","last_undegraded":"2024-04-26T23:45:00.548525+0000","last_fullsized":"2024-04-26T23:45:00.548525+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:33:54.105329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.538382+0000","last_change":"2024-04-26T23:43:02.443541+0000","last_active":"2024-04-26T23:45:00.538382+0000","last_peered":"2024-04-26T23:45:00.538382+0000","last_clean":"2024-04-26T23:45:00.538382+0000","last_became_active":"2024-04-26T23:43:02.442948+0000","last_became_peered":"2024-04-26T23:43:02.442948+0000","last_unstale":"2024-04-26T23:45:00.538382+0000","last_undegraded":"2024-04-26T23:45:00.538382+0000","last_fullsized":"2024-04-26T23:45:00.538382+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:58:44.595575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.555477+0000","last_change":"2024-04-26T23:43:03.311029+0000","last_active":"2024-04-26T23:45:00.555477+0000","last_peered":"2024-04-26T23:45:00.555477+0000","last_clean":"2024-04-26T23:45:00.555477+0000","last_became_active":"2024-04-26T23:43:03.310722+0000","last_became_peered":"2024-04-26T23:43:03.310722+0000","last_unstale":"2024-04-26T23:45:00.555477+0000","last_undegraded":"2024-04-26T23:45:00.555477+0000","last_fullsized":"2024-04-26T23:45:00.555477+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:07:54.449492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.543870+0000","last_change":"2024-04-26T23:43:07.537391+0000","last_active":"2024-04-26T23:45:00.543870+0000","last_peered":"2024-04-26T23:45:00.543870+0000","last_clean":"2024-04-26T23:45:00.543870+0000","last_became_active":"2024-04-26T23:43:07.536216+0000","last_became_peered":"2024-04-26T23:43:07.536216+0000","last_unstale":"2024-04-26T23:45:00.543870+0000","last_undegraded":"2024-04-26T23:45:00.543870+0000","last_fullsized":"2024-04-26T23:45:00.543870+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:45:55.326283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":127,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:02.342834+0000","last_change":"2024-04-26T23:43:02.451929+0000","last_active":"2024-04-26T23:45:02.342834+0000","last_peered":"2024-04-26T23:45:02.342834+0000","last_clean":"2024-04-26T23:45:02.342834+0000","last_became_active":"2024-04-26T23:43:02.451700+0000","last_became_peered":"2024-04-26T23:43:02.451700+0000","last_unstale":"2024-04-26T23:45:02.342834+0000","last_undegraded":"2024-04-26T23:45:02.342834+0000","last_fullsized":"2024-04-26T23:45:02.342834+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:21:31.279681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":94,"num_read_kb":94,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-26T23:44:59.539275+0000","last_change":"2024-04-26T23:43:03.295462+0000","last_active":"2024-04-26T23:44:59.539275+0000","last_peered":"2024-04-26T23:44:59.539275+0000","last_clean":"2024-04-26T23:44:59.539275+0000","last_became_active":"2024-04-26T23:43:03.295044+0000","last_became_peered":"2024-04-26T23:43:03.295044+0000","last_unstale":"2024-04-26T23:44:59.539275+0000","last_undegraded":"2024-04-26T23:44:59.539275+0000","last_fullsized":"2024-04-26T23:44:59.539275+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:56:19.445166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.554405+0000","last_change":"2024-04-26T23:43:07.539397+0000","last_active":"2024-04-26T23:45:00.554405+0000","last_peered":"2024-04-26T23:45:00.554405+0000","last_clean":"2024-04-26T23:45:00.554405+0000","last_became_active":"2024-04-26T23:43:07.533081+0000","last_became_peered":"2024-04-26T23:43:07.533081+0000","last_unstale":"2024-04-26T23:45:00.554405+0000","last_undegraded":"2024-04-26T23:45:00.554405+0000","last_fullsized":"2024-04-26T23:45:00.554405+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:24:39.693721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.571849+0000","last_change":"2024-04-26T23:43:02.439919+0000","last_active":"2024-04-26T23:43:59.571849+0000","last_peered":"2024-04-26T23:43:59.571849+0000","last_clean":"2024-04-26T23:43:59.571849+0000","last_became_active":"2024-04-26T23:43:02.439561+0000","last_became_peered":"2024-04-26T23:43:02.439561+0000","last_unstale":"2024-04-26T23:43:59.571849+0000","last_undegraded":"2024-04-26T23:43:59.571849+0000","last_fullsized":"2024-04-26T23:43:59.571849+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:30:22.905464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.549199+0000","last_change":"2024-04-26T23:43:03.294088+0000","last_active":"2024-04-26T23:45:00.549199+0000","last_peered":"2024-04-26T23:45:00.549199+0000","last_clean":"2024-04-26T23:45:00.549199+0000","last_became_active":"2024-04-26T23:43:03.293726+0000","last_became_peered":"2024-04-26T23:43:03.293726+0000","last_unstale":"2024-04-26T23:45:00.549199+0000","last_undegraded":"2024-04-26T23:45:00.549199+0000","last_fullsized":"2024-04-26T23:45:00.549199+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:19:42.575193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.541076+0000","last_change":"2024-04-26T23:43:07.543056+0000","last_active":"2024-04-26T23:45:00.541076+0000","last_peered":"2024-04-26T23:45:00.541076+0000","last_clean":"2024-04-26T23:45:00.541076+0000","last_became_active":"2024-04-26T23:43:07.542324+0000","last_became_peered":"2024-04-26T23:43:07.542324+0000","last_unstale":"2024-04-26T23:45:00.541076+0000","last_undegraded":"2024-04-26T23:45:00.541076+0000","last_fullsized":"2024-04-26T23:45:00.541076+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:23:46.415932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.538595+0000","last_change":"2024-04-26T23:43:02.454135+0000","last_active":"2024-04-26T23:45:00.538595+0000","last_peered":"2024-04-26T23:45:00.538595+0000","last_clean":"2024-04-26T23:45:00.538595+0000","last_became_active":"2024-04-26T23:43:02.453798+0000","last_became_peered":"2024-04-26T23:43:02.453798+0000","last_unstale":"2024-04-26T23:45:00.538595+0000","last_undegraded":"2024-04-26T23:45:00.538595+0000","last_fullsized":"2024-04-26T23:45:00.538595+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:16:25.286569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-26T23:44:59.539446+0000","last_change":"2024-04-26T23:43:03.293520+0000","last_active":"2024-04-26T23:44:59.539446+0000","last_peered":"2024-04-26T23:44:59.539446+0000","last_clean":"2024-04-26T23:44:59.539446+0000","last_became_active":"2024-04-26T23:43:03.292537+0000","last_became_peered":"2024-04-26T23:43:03.292537+0000","last_unstale":"2024-04-26T23:44:59.539446+0000","last_undegraded":"2024-04-26T23:44:59.539446+0000","last_fullsized":"2024-04-26T23:44:59.539446+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:19:28.808217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.555126+0000","last_change":"2024-04-26T23:43:07.528116+0000","last_active":"2024-04-26T23:45:00.555126+0000","last_peered":"2024-04-26T23:45:00.555126+0000","last_clean":"2024-04-26T23:45:00.555126+0000","last_became_active":"2024-04-26T23:43:07.527766+0000","last_became_peered":"2024-04-26T23:43:07.527766+0000","last_unstale":"2024-04-26T23:45:00.555126+0000","last_undegraded":"2024-04-26T23:45:00.555126+0000","last_fullsized":"2024-04-26T23:45:00.555126+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:39:23.949964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'33","reported_seq":258,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:03.356078+0000","last_change":"2024-04-26T23:43:02.437454+0000","last_active":"2024-04-26T23:45:03.356078+0000","last_peered":"2024-04-26T23:45:03.356078+0000","last_clean":"2024-04-26T23:45:03.356078+0000","last_became_active":"2024-04-26T23:43:02.437142+0000","last_became_peered":"2024-04-26T23:43:02.437142+0000","last_unstale":"2024-04-26T23:45:03.356078+0000","last_undegraded":"2024-04-26T23:45:03.356078+0000","last_fullsized":"2024-04-26T23:45:03.356078+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:06:37.751066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54826,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":195,"num_read_kb":150,"num_write":34,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544594+0000","last_change":"2024-04-26T23:43:03.287831+0000","last_active":"2024-04-26T23:45:00.544594+0000","last_peered":"2024-04-26T23:45:00.544594+0000","last_clean":"2024-04-26T23:45:00.544594+0000","last_became_active":"2024-04-26T23:43:03.287479+0000","last_became_peered":"2024-04-26T23:43:03.287479+0000","last_unstale":"2024-04-26T23:45:00.544594+0000","last_undegraded":"2024-04-26T23:45:00.544594+0000","last_fullsized":"2024-04-26T23:45:00.544594+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:39:41.726893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.539031+0000","last_change":"2024-04-26T23:43:07.543985+0000","last_active":"2024-04-26T23:45:00.539031+0000","last_peered":"2024-04-26T23:45:00.539031+0000","last_clean":"2024-04-26T23:45:00.539031+0000","last_became_active":"2024-04-26T23:43:07.543327+0000","last_became_peered":"2024-04-26T23:43:07.543327+0000","last_unstale":"2024-04-26T23:45:00.539031+0000","last_undegraded":"2024-04-26T23:45:00.539031+0000","last_fullsized":"2024-04-26T23:45:00.539031+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:24:10.577822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.566396+0000","last_change":"2024-04-26T23:43:02.447627+0000","last_active":"2024-04-26T23:43:59.566396+0000","last_peered":"2024-04-26T23:43:59.566396+0000","last_clean":"2024-04-26T23:43:59.566396+0000","last_became_active":"2024-04-26T23:43:02.447325+0000","last_became_peered":"2024-04-26T23:43:02.447325+0000","last_unstale":"2024-04-26T23:43:59.566396+0000","last_undegraded":"2024-04-26T23:43:59.566396+0000","last_fullsized":"2024-04-26T23:43:59.566396+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:19:03.757202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.547818+0000","last_change":"2024-04-26T23:43:03.313388+0000","last_active":"2024-04-26T23:45:00.547818+0000","last_peered":"2024-04-26T23:45:00.547818+0000","last_clean":"2024-04-26T23:45:00.547818+0000","last_became_active":"2024-04-26T23:43:03.312899+0000","last_became_peered":"2024-04-26T23:43:03.312899+0000","last_unstale":"2024-04-26T23:45:00.547818+0000","last_undegraded":"2024-04-26T23:45:00.547818+0000","last_fullsized":"2024-04-26T23:45:00.547818+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:58:37.126485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-26T23:44:59.538190+0000","last_change":"2024-04-26T23:43:07.536706+0000","last_active":"2024-04-26T23:44:59.538190+0000","last_peered":"2024-04-26T23:44:59.538190+0000","last_clean":"2024-04-26T23:44:59.538190+0000","last_became_active":"2024-04-26T23:43:07.536316+0000","last_became_peered":"2024-04-26T23:43:07.536316+0000","last_unstale":"2024-04-26T23:44:59.538190+0000","last_undegraded":"2024-04-26T23:44:59.538190+0000","last_fullsized":"2024-04-26T23:44:59.538190+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:36:46.397725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.538435+0000","last_change":"2024-04-26T23:43:02.448647+0000","last_active":"2024-04-26T23:45:00.538435+0000","last_peered":"2024-04-26T23:45:00.538435+0000","last_clean":"2024-04-26T23:45:00.538435+0000","last_became_active":"2024-04-26T23:43:02.448392+0000","last_became_peered":"2024-04-26T23:43:02.448392+0000","last_unstale":"2024-04-26T23:45:00.538435+0000","last_undegraded":"2024-04-26T23:45:00.538435+0000","last_fullsized":"2024-04-26T23:45:00.538435+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:19:01.367361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.571932+0000","last_change":"2024-04-26T23:43:03.302980+0000","last_active":"2024-04-26T23:43:59.571932+0000","last_peered":"2024-04-26T23:43:59.571932+0000","last_clean":"2024-04-26T23:43:59.571932+0000","last_became_active":"2024-04-26T23:43:03.302589+0000","last_became_peered":"2024-04-26T23:43:03.302589+0000","last_unstale":"2024-04-26T23:43:59.571932+0000","last_undegraded":"2024-04-26T23:43:59.571932+0000","last_fullsized":"2024-04-26T23:43:59.571932+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:57:18.641063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.556728+0000","last_change":"2024-04-26T23:43:07.547842+0000","last_active":"2024-04-26T23:45:00.556728+0000","last_peered":"2024-04-26T23:45:00.556728+0000","last_clean":"2024-04-26T23:45:00.556728+0000","last_became_active":"2024-04-26T23:43:07.541774+0000","last_became_peered":"2024-04-26T23:43:07.541774+0000","last_unstale":"2024-04-26T23:45:00.556728+0000","last_undegraded":"2024-04-26T23:45:00.556728+0000","last_fullsized":"2024-04-26T23:45:00.556728+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:25:49.999465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'4","reported_seq":132,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:02.396473+0000","last_change":"2024-04-26T23:43:02.437210+0000","last_active":"2024-04-26T23:45:02.396473+0000","last_peered":"2024-04-26T23:45:02.396473+0000","last_clean":"2024-04-26T23:45:02.396473+0000","last_became_active":"2024-04-26T23:43:02.436886+0000","last_became_peered":"2024-04-26T23:43:02.436886+0000","last_unstale":"2024-04-26T23:45:02.396473+0000","last_undegraded":"2024-04-26T23:45:02.396473+0000","last_fullsized":"2024-04-26T23:45:02.396473+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:07:03.663757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":98,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.868608+0000","last_change":"2024-04-26T23:43:59.868608+0000","last_active":"2024-04-26T23:43:59.868608+0000","last_peered":"2024-04-26T23:43:59.868608+0000","last_clean":"2024-04-26T23:43:59.868608+0000","last_became_active":"2024-04-26T23:43:59.868231+0000","last_became_peered":"2024-04-26T23:43:59.868231+0000","last_unstale":"2024-04-26T23:43:59.868608+0000","last_undegraded":"2024-04-26T23:43:59.868608+0000","last_fullsized":"2024-04-26T23:43:59.868608+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:28:42.932733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.559597+0000","last_change":"2024-04-26T23:45:00.559597+0000","last_active":"2024-04-26T23:45:00.559597+0000","last_peered":"2024-04-26T23:45:00.559597+0000","last_clean":"2024-04-26T23:45:00.559597+0000","last_became_active":"2024-04-26T23:45:00.558473+0000","last_became_peered":"2024-04-26T23:45:00.558473+0000","last_unstale":"2024-04-26T23:45:00.559597+0000","last_undegraded":"2024-04-26T23:45:00.559597+0000","last_fullsized":"2024-04-26T23:45:00.559597+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:07:29.856136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":70,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.543200+0000","last_change":"2024-04-26T23:43:59.589239+0000","last_active":"2024-04-26T23:45:00.543200+0000","last_peered":"2024-04-26T23:45:00.543200+0000","last_clean":"2024-04-26T23:45:00.543200+0000","last_became_active":"2024-04-26T23:43:59.576239+0000","last_became_peered":"2024-04-26T23:43:59.576239+0000","last_unstale":"2024-04-26T23:45:00.543200+0000","last_undegraded":"2024-04-26T23:45:00.543200+0000","last_fullsized":"2024-04-26T23:45:00.543200+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:28:53.326961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.538251+0000","last_change":"2024-04-26T23:43:03.290567+0000","last_active":"2024-04-26T23:45:00.538251+0000","last_peered":"2024-04-26T23:45:00.538251+0000","last_clean":"2024-04-26T23:45:00.538251+0000","last_became_active":"2024-04-26T23:43:03.287807+0000","last_became_peered":"2024-04-26T23:43:03.287807+0000","last_unstale":"2024-04-26T23:45:00.538251+0000","last_undegraded":"2024-04-26T23:45:00.538251+0000","last_fullsized":"2024-04-26T23:45:00.538251+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:25:24.393978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.538200+0000","last_change":"2024-04-26T23:43:07.535402+0000","last_active":"2024-04-26T23:45:00.538200+0000","last_peered":"2024-04-26T23:45:00.538200+0000","last_clean":"2024-04-26T23:45:00.538200+0000","last_became_active":"2024-04-26T23:43:07.534870+0000","last_became_peered":"2024-04-26T23:43:07.534870+0000","last_unstale":"2024-04-26T23:45:00.538200+0000","last_undegraded":"2024-04-26T23:45:00.538200+0000","last_fullsized":"2024-04-26T23:45:00.538200+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:26:37.703831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":2,"reported_epoch":92,"state":"peering","last_fresh":"2024-04-26T23:43:08.418042+0000","last_change":"2024-04-26T23:45:00.495897+0000","last_active":"2024-04-26T23:45:00.495897+0000","last_peered":"2024-04-26T23:43:08.418042+0000","last_clean":"2024-04-26T23:43:08.418042+0000","last_became_active":"2024-04-26T23:43:02.440294+0000","last_became_peered":"2024-04-26T23:43:02.440294+0000","last_unstale":"2024-04-26T23:43:08.418042+0000","last_undegraded":"2024-04-26T23:43:08.418042+0000","last_fullsized":"2024-04-26T23:43:08.418042+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-26T23:44:59.538568+0000","last_change":"2024-04-26T23:43:59.576561+0000","last_active":"2024-04-26T23:44:59.538568+0000","last_peered":"2024-04-26T23:44:59.538568+0000","last_clean":"2024-04-26T23:44:59.538568+0000","last_became_active":"2024-04-26T23:43:59.574898+0000","last_became_peered":"2024-04-26T23:43:59.574898+0000","last_unstale":"2024-04-26T23:44:59.538568+0000","last_undegraded":"2024-04-26T23:44:59.538568+0000","last_fullsized":"2024-04-26T23:44:59.538568+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:57:04.569103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.549787+0000","last_change":"2024-04-26T23:45:00.549787+0000","last_active":"2024-04-26T23:45:00.549787+0000","last_peered":"2024-04-26T23:45:00.549787+0000","last_clean":"2024-04-26T23:45:00.549787+0000","last_became_active":"2024-04-26T23:45:00.549316+0000","last_became_peered":"2024-04-26T23:45:00.549316+0000","last_unstale":"2024-04-26T23:45:00.549787+0000","last_undegraded":"2024-04-26T23:45:00.549787+0000","last_fullsized":"2024-04-26T23:45:00.549787+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:21:36.188388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.555492+0000","last_change":"2024-04-26T23:43:02.457971+0000","last_active":"2024-04-26T23:45:00.555492+0000","last_peered":"2024-04-26T23:45:00.555492+0000","last_clean":"2024-04-26T23:45:00.555492+0000","last_became_active":"2024-04-26T23:43:02.457627+0000","last_became_peered":"2024-04-26T23:43:02.457627+0000","last_unstale":"2024-04-26T23:45:00.555492+0000","last_undegraded":"2024-04-26T23:45:00.555492+0000","last_fullsized":"2024-04-26T23:45:00.555492+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:25:51.749774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.548355+0000","last_change":"2024-04-26T23:43:03.305569+0000","last_active":"2024-04-26T23:45:00.548355+0000","last_peered":"2024-04-26T23:45:00.548355+0000","last_clean":"2024-04-26T23:45:00.548355+0000","last_became_active":"2024-04-26T23:43:03.305211+0000","last_became_peered":"2024-04-26T23:43:03.305211+0000","last_unstale":"2024-04-26T23:45:00.548355+0000","last_undegraded":"2024-04-26T23:45:00.548355+0000","last_fullsized":"2024-04-26T23:45:00.548355+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:54:55.847680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.553421+0000","last_change":"2024-04-26T23:43:07.535408+0000","last_active":"2024-04-26T23:45:00.553421+0000","last_peered":"2024-04-26T23:45:00.553421+0000","last_clean":"2024-04-26T23:45:00.553421+0000","last_became_active":"2024-04-26T23:43:07.534957+0000","last_became_peered":"2024-04-26T23:43:07.534957+0000","last_unstale":"2024-04-26T23:45:00.553421+0000","last_undegraded":"2024-04-26T23:45:00.553421+0000","last_fullsized":"2024-04-26T23:45:00.553421+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:35:31.103532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.571993+0000","last_change":"2024-04-26T23:43:02.453636+0000","last_active":"2024-04-26T23:43:59.571993+0000","last_peered":"2024-04-26T23:43:59.571993+0000","last_clean":"2024-04-26T23:43:59.571993+0000","last_became_active":"2024-04-26T23:43:02.453310+0000","last_became_peered":"2024-04-26T23:43:02.453310+0000","last_unstale":"2024-04-26T23:43:59.571993+0000","last_undegraded":"2024-04-26T23:43:59.571993+0000","last_fullsized":"2024-04-26T23:43:59.571993+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:42:29.295303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.548481+0000","last_change":"2024-04-26T23:43:03.283184+0000","last_active":"2024-04-26T23:45:00.548481+0000","last_peered":"2024-04-26T23:45:00.548481+0000","last_clean":"2024-04-26T23:45:00.548481+0000","last_became_active":"2024-04-26T23:43:03.282780+0000","last_became_peered":"2024-04-26T23:43:03.282780+0000","last_unstale":"2024-04-26T23:45:00.548481+0000","last_undegraded":"2024-04-26T23:45:00.548481+0000","last_fullsized":"2024-04-26T23:45:00.548481+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:00:13.907101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.555563+0000","last_change":"2024-04-26T23:43:07.859107+0000","last_active":"2024-04-26T23:45:00.555563+0000","last_peered":"2024-04-26T23:45:00.555563+0000","last_clean":"2024-04-26T23:45:00.555563+0000","last_became_active":"2024-04-26T23:43:07.856492+0000","last_became_peered":"2024-04-26T23:43:07.856492+0000","last_unstale":"2024-04-26T23:45:00.555563+0000","last_undegraded":"2024-04-26T23:45:00.555563+0000","last_fullsized":"2024-04-26T23:45:00.555563+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:13:36.570398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.538547+0000","last_change":"2024-04-26T23:43:03.289078+0000","last_active":"2024-04-26T23:45:00.538547+0000","last_peered":"2024-04-26T23:45:00.538547+0000","last_clean":"2024-04-26T23:45:00.538547+0000","last_became_active":"2024-04-26T23:43:03.288729+0000","last_became_peered":"2024-04-26T23:43:03.288729+0000","last_unstale":"2024-04-26T23:45:00.538547+0000","last_undegraded":"2024-04-26T23:45:00.538547+0000","last_fullsized":"2024-04-26T23:45:00.538547+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:32:36.346472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":129,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:02.395686+0000","last_change":"2024-04-26T23:43:02.463355+0000","last_active":"2024-04-26T23:45:02.395686+0000","last_peered":"2024-04-26T23:45:02.395686+0000","last_clean":"2024-04-26T23:45:02.395686+0000","last_became_active":"2024-04-26T23:43:02.455832+0000","last_became_peered":"2024-04-26T23:43:02.455832+0000","last_unstale":"2024-04-26T23:45:02.395686+0000","last_undegraded":"2024-04-26T23:45:02.395686+0000","last_fullsized":"2024-04-26T23:45:02.395686+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:33:33.837999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":94,"num_read_kb":94,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.572048+0000","last_change":"2024-04-26T23:43:07.540777+0000","last_active":"2024-04-26T23:43:59.572048+0000","last_peered":"2024-04-26T23:43:59.572048+0000","last_clean":"2024-04-26T23:43:59.572048+0000","last_became_active":"2024-04-26T23:43:07.540389+0000","last_became_peered":"2024-04-26T23:43:07.540389+0000","last_unstale":"2024-04-26T23:43:59.572048+0000","last_undegraded":"2024-04-26T23:43:59.572048+0000","last_fullsized":"2024-04-26T23:43:59.572048+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:57:28.674051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544701+0000","last_change":"2024-04-26T23:43:03.308391+0000","last_active":"2024-04-26T23:45:00.544701+0000","last_peered":"2024-04-26T23:45:00.544701+0000","last_clean":"2024-04-26T23:45:00.544701+0000","last_became_active":"2024-04-26T23:43:03.306762+0000","last_became_peered":"2024-04-26T23:43:03.306762+0000","last_unstale":"2024-04-26T23:45:00.544701+0000","last_undegraded":"2024-04-26T23:45:00.544701+0000","last_fullsized":"2024-04-26T23:45:00.544701+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:34:01.083036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.540940+0000","last_change":"2024-04-26T23:43:02.448037+0000","last_active":"2024-04-26T23:45:00.540940+0000","last_peered":"2024-04-26T23:45:00.540940+0000","last_clean":"2024-04-26T23:45:00.540940+0000","last_became_active":"2024-04-26T23:43:02.447706+0000","last_became_peered":"2024-04-26T23:43:02.447706+0000","last_unstale":"2024-04-26T23:45:00.540940+0000","last_undegraded":"2024-04-26T23:45:00.540940+0000","last_fullsized":"2024-04-26T23:45:00.540940+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:43:56.295191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":348,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.571505+0000","last_change":"2024-04-26T23:41:42.266448+0000","last_active":"2024-04-26T23:43:59.571505+0000","last_peered":"2024-04-26T23:43:59.571505+0000","last_clean":"2024-04-26T23:43:59.571505+0000","last_became_active":"2024-04-26T23:41:42.265898+0000","last_became_peered":"2024-04-26T23:41:42.265898+0000","last_unstale":"2024-04-26T23:43:59.571505+0000","last_undegraded":"2024-04-26T23:43:59.571505+0000","last_fullsized":"2024-04-26T23:43:59.571505+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:37:44.460409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:37:44.460409+0000","last_clean_scrub_stamp":"2024-04-26T23:37:44.460409+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:40:48.177191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.557115+0000","last_change":"2024-04-26T23:45:00.557115+0000","last_active":"2024-04-26T23:45:00.557115+0000","last_peered":"2024-04-26T23:45:00.557115+0000","last_clean":"2024-04-26T23:45:00.557115+0000","last_became_active":"2024-04-26T23:45:00.556698+0000","last_became_peered":"2024-04-26T23:45:00.556698+0000","last_unstale":"2024-04-26T23:45:00.557115+0000","last_undegraded":"2024-04-26T23:45:00.557115+0000","last_fullsized":"2024-04-26T23:45:00.557115+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:23:11.253779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.548048+0000","last_change":"2024-04-26T23:43:03.301989+0000","last_active":"2024-04-26T23:45:00.548048+0000","last_peered":"2024-04-26T23:45:00.548048+0000","last_clean":"2024-04-26T23:45:00.548048+0000","last_became_active":"2024-04-26T23:43:03.301179+0000","last_became_peered":"2024-04-26T23:43:03.301179+0000","last_unstale":"2024-04-26T23:45:00.548048+0000","last_undegraded":"2024-04-26T23:45:00.548048+0000","last_fullsized":"2024-04-26T23:45:00.548048+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:29:33.012009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.571573+0000","last_change":"2024-04-26T23:43:02.439648+0000","last_active":"2024-04-26T23:43:59.571573+0000","last_peered":"2024-04-26T23:43:59.571573+0000","last_clean":"2024-04-26T23:43:59.571573+0000","last_became_active":"2024-04-26T23:43:02.439340+0000","last_became_peered":"2024-04-26T23:43:02.439340+0000","last_unstale":"2024-04-26T23:43:59.571573+0000","last_undegraded":"2024-04-26T23:43:59.571573+0000","last_fullsized":"2024-04-26T23:43:59.571573+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:10:01.096657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-26T23:44:59.538058+0000","last_change":"2024-04-26T23:43:07.534317+0000","last_active":"2024-04-26T23:44:59.538058+0000","last_peered":"2024-04-26T23:44:59.538058+0000","last_clean":"2024-04-26T23:44:59.538058+0000","last_became_active":"2024-04-26T23:43:07.533808+0000","last_became_peered":"2024-04-26T23:43:07.533808+0000","last_unstale":"2024-04-26T23:44:59.538058+0000","last_undegraded":"2024-04-26T23:44:59.538058+0000","last_fullsized":"2024-04-26T23:44:59.538058+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:54:29.952089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544753+0000","last_change":"2024-04-26T23:43:02.457501+0000","last_active":"2024-04-26T23:45:00.544753+0000","last_peered":"2024-04-26T23:45:00.544753+0000","last_clean":"2024-04-26T23:45:00.544753+0000","last_became_active":"2024-04-26T23:43:02.457024+0000","last_became_peered":"2024-04-26T23:43:02.457024+0000","last_unstale":"2024-04-26T23:45:00.544753+0000","last_undegraded":"2024-04-26T23:45:00.544753+0000","last_fullsized":"2024-04-26T23:45:00.544753+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:21:58.542186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.553577+0000","last_change":"2024-04-26T23:43:03.303434+0000","last_active":"2024-04-26T23:45:00.553577+0000","last_peered":"2024-04-26T23:45:00.553577+0000","last_clean":"2024-04-26T23:45:00.553577+0000","last_became_active":"2024-04-26T23:43:03.302455+0000","last_became_peered":"2024-04-26T23:43:03.302455+0000","last_unstale":"2024-04-26T23:45:00.553577+0000","last_undegraded":"2024-04-26T23:45:00.553577+0000","last_fullsized":"2024-04-26T23:45:00.553577+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:57:05.530635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.542401+0000","last_change":"2024-04-26T23:43:07.856932+0000","last_active":"2024-04-26T23:45:00.542401+0000","last_peered":"2024-04-26T23:45:00.542401+0000","last_clean":"2024-04-26T23:45:00.542401+0000","last_became_active":"2024-04-26T23:43:07.855699+0000","last_became_peered":"2024-04-26T23:43:07.855699+0000","last_unstale":"2024-04-26T23:45:00.542401+0000","last_undegraded":"2024-04-26T23:45:00.542401+0000","last_fullsized":"2024-04-26T23:45:00.542401+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:20:28.994301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.557150+0000","last_change":"2024-04-26T23:43:02.449852+0000","last_active":"2024-04-26T23:45:00.557150+0000","last_peered":"2024-04-26T23:45:00.557150+0000","last_clean":"2024-04-26T23:45:00.557150+0000","last_became_active":"2024-04-26T23:43:02.449422+0000","last_became_peered":"2024-04-26T23:43:02.449422+0000","last_unstale":"2024-04-26T23:45:00.557150+0000","last_undegraded":"2024-04-26T23:45:00.557150+0000","last_fullsized":"2024-04-26T23:45:00.557150+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:07:25.453872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.551659+0000","last_change":"2024-04-26T23:43:03.304733+0000","last_active":"2024-04-26T23:45:00.551659+0000","last_peered":"2024-04-26T23:45:00.551659+0000","last_clean":"2024-04-26T23:45:00.551659+0000","last_became_active":"2024-04-26T23:43:03.304383+0000","last_became_peered":"2024-04-26T23:43:03.304383+0000","last_unstale":"2024-04-26T23:45:00.551659+0000","last_undegraded":"2024-04-26T23:45:00.551659+0000","last_fullsized":"2024-04-26T23:45:00.551659+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:49:28.679255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.566122+0000","last_change":"2024-04-26T23:43:07.525310+0000","last_active":"2024-04-26T23:43:59.566122+0000","last_peered":"2024-04-26T23:43:59.566122+0000","last_clean":"2024-04-26T23:43:59.566122+0000","last_became_active":"2024-04-26T23:43:07.524502+0000","last_became_peered":"2024-04-26T23:43:07.524502+0000","last_unstale":"2024-04-26T23:43:59.566122+0000","last_undegraded":"2024-04-26T23:43:59.566122+0000","last_fullsized":"2024-04-26T23:43:59.566122+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:07:45.927837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.541373+0000","last_change":"2024-04-26T23:43:02.446238+0000","last_active":"2024-04-26T23:45:00.541373+0000","last_peered":"2024-04-26T23:45:00.541373+0000","last_clean":"2024-04-26T23:45:00.541373+0000","last_became_active":"2024-04-26T23:43:02.445959+0000","last_became_peered":"2024-04-26T23:43:02.445959+0000","last_unstale":"2024-04-26T23:45:00.541373+0000","last_undegraded":"2024-04-26T23:45:00.541373+0000","last_fullsized":"2024-04-26T23:45:00.541373+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:25:10.007481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.548843+0000","last_change":"2024-04-26T23:43:03.306035+0000","last_active":"2024-04-26T23:45:00.548843+0000","last_peered":"2024-04-26T23:45:00.548843+0000","last_clean":"2024-04-26T23:45:00.548843+0000","last_became_active":"2024-04-26T23:43:03.305604+0000","last_became_peered":"2024-04-26T23:43:03.305604+0000","last_unstale":"2024-04-26T23:45:00.548843+0000","last_undegraded":"2024-04-26T23:45:00.548843+0000","last_fullsized":"2024-04-26T23:45:00.548843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:02:38.567754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.557767+0000","last_change":"2024-04-26T23:45:00.557767+0000","last_active":"2024-04-26T23:45:00.557767+0000","last_peered":"2024-04-26T23:45:00.557767+0000","last_clean":"2024-04-26T23:45:00.557767+0000","last_became_active":"2024-04-26T23:45:00.557207+0000","last_became_peered":"2024-04-26T23:45:00.557207+0000","last_unstale":"2024-04-26T23:45:00.557767+0000","last_undegraded":"2024-04-26T23:45:00.557767+0000","last_fullsized":"2024-04-26T23:45:00.557767+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:18:06.396984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,10,9],"acting":[4,8,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-26T23:44:59.537488+0000","last_change":"2024-04-26T23:43:02.443464+0000","last_active":"2024-04-26T23:44:59.537488+0000","last_peered":"2024-04-26T23:44:59.537488+0000","last_clean":"2024-04-26T23:44:59.537488+0000","last_became_active":"2024-04-26T23:43:02.443170+0000","last_became_peered":"2024-04-26T23:43:02.443170+0000","last_unstale":"2024-04-26T23:44:59.537488+0000","last_undegraded":"2024-04-26T23:44:59.537488+0000","last_fullsized":"2024-04-26T23:44:59.537488+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:45:51.431082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544694+0000","last_change":"2024-04-26T23:43:03.282714+0000","last_active":"2024-04-26T23:45:00.544694+0000","last_peered":"2024-04-26T23:45:00.544694+0000","last_clean":"2024-04-26T23:45:00.544694+0000","last_became_active":"2024-04-26T23:43:03.282353+0000","last_became_peered":"2024-04-26T23:43:03.282353+0000","last_unstale":"2024-04-26T23:45:00.544694+0000","last_undegraded":"2024-04-26T23:45:00.544694+0000","last_fullsized":"2024-04-26T23:45:00.544694+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:18:24.082946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.556962+0000","last_change":"2024-04-26T23:43:07.543301+0000","last_active":"2024-04-26T23:45:00.556962+0000","last_peered":"2024-04-26T23:45:00.556962+0000","last_clean":"2024-04-26T23:45:00.556962+0000","last_became_active":"2024-04-26T23:43:07.540434+0000","last_became_peered":"2024-04-26T23:43:07.540434+0000","last_unstale":"2024-04-26T23:45:00.556962+0000","last_undegraded":"2024-04-26T23:45:00.556962+0000","last_fullsized":"2024-04-26T23:45:00.556962+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:15:18.542595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.543289+0000","last_change":"2024-04-26T23:43:02.451391+0000","last_active":"2024-04-26T23:45:00.543289+0000","last_peered":"2024-04-26T23:45:00.543289+0000","last_clean":"2024-04-26T23:45:00.543289+0000","last_became_active":"2024-04-26T23:43:02.450735+0000","last_became_peered":"2024-04-26T23:43:02.450735+0000","last_unstale":"2024-04-26T23:45:00.543289+0000","last_undegraded":"2024-04-26T23:45:00.543289+0000","last_fullsized":"2024-04-26T23:45:00.543289+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:57:48.658671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-26T23:44:59.537799+0000","last_change":"2024-04-26T23:43:03.301858+0000","last_active":"2024-04-26T23:44:59.537799+0000","last_peered":"2024-04-26T23:44:59.537799+0000","last_clean":"2024-04-26T23:44:59.537799+0000","last_became_active":"2024-04-26T23:43:03.301474+0000","last_became_peered":"2024-04-26T23:43:03.301474+0000","last_unstale":"2024-04-26T23:44:59.537799+0000","last_undegraded":"2024-04-26T23:44:59.537799+0000","last_fullsized":"2024-04-26T23:44:59.537799+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:24:56.386127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.541095+0000","last_change":"2024-04-26T23:43:07.544512+0000","last_active":"2024-04-26T23:45:00.541095+0000","last_peered":"2024-04-26T23:45:00.541095+0000","last_clean":"2024-04-26T23:45:00.541095+0000","last_became_active":"2024-04-26T23:43:07.544029+0000","last_became_peered":"2024-04-26T23:43:07.544029+0000","last_unstale":"2024-04-26T23:45:00.541095+0000","last_undegraded":"2024-04-26T23:45:00.541095+0000","last_fullsized":"2024-04-26T23:45:00.541095+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:38:08.149123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-26T23:44:59.538555+0000","last_change":"2024-04-26T23:43:59.579721+0000","last_active":"2024-04-26T23:44:59.538555+0000","last_peered":"2024-04-26T23:44:59.538555+0000","last_clean":"2024-04-26T23:44:59.538555+0000","last_became_active":"2024-04-26T23:43:59.576675+0000","last_became_peered":"2024-04-26T23:43:59.576675+0000","last_unstale":"2024-04-26T23:44:59.538555+0000","last_undegraded":"2024-04-26T23:44:59.538555+0000","last_fullsized":"2024-04-26T23:44:59.538555+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:59:30.124137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.549817+0000","last_change":"2024-04-26T23:43:03.307895+0000","last_active":"2024-04-26T23:45:00.549817+0000","last_peered":"2024-04-26T23:45:00.549817+0000","last_clean":"2024-04-26T23:45:00.549817+0000","last_became_active":"2024-04-26T23:43:03.307478+0000","last_became_peered":"2024-04-26T23:43:03.307478+0000","last_unstale":"2024-04-26T23:45:00.549817+0000","last_undegraded":"2024-04-26T23:45:00.549817+0000","last_fullsized":"2024-04-26T23:45:00.549817+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:57:14.356234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.541058+0000","last_change":"2024-04-26T23:43:07.855803+0000","last_active":"2024-04-26T23:45:00.541058+0000","last_peered":"2024-04-26T23:45:00.541058+0000","last_clean":"2024-04-26T23:45:00.541058+0000","last_became_active":"2024-04-26T23:43:07.855200+0000","last_became_peered":"2024-04-26T23:43:07.855200+0000","last_unstale":"2024-04-26T23:45:00.541058+0000","last_undegraded":"2024-04-26T23:45:00.541058+0000","last_fullsized":"2024-04-26T23:45:00.541058+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:02:46.223330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.543821+0000","last_change":"2024-04-26T23:43:02.437513+0000","last_active":"2024-04-26T23:45:00.543821+0000","last_peered":"2024-04-26T23:45:00.543821+0000","last_clean":"2024-04-26T23:45:00.543821+0000","last_became_active":"2024-04-26T23:43:02.437153+0000","last_became_peered":"2024-04-26T23:43:02.437153+0000","last_unstale":"2024-04-26T23:45:00.543821+0000","last_undegraded":"2024-04-26T23:45:00.543821+0000","last_fullsized":"2024-04-26T23:45:00.543821+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:11:45.477579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.572239+0000","last_change":"2024-04-26T23:43:03.297835+0000","last_active":"2024-04-26T23:43:59.572239+0000","last_peered":"2024-04-26T23:43:59.572239+0000","last_clean":"2024-04-26T23:43:59.572239+0000","last_became_active":"2024-04-26T23:43:03.297481+0000","last_became_peered":"2024-04-26T23:43:03.297481+0000","last_unstale":"2024-04-26T23:43:59.572239+0000","last_undegraded":"2024-04-26T23:43:59.572239+0000","last_fullsized":"2024-04-26T23:43:59.572239+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:52:54.757461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-26T23:44:59.537979+0000","last_change":"2024-04-26T23:43:07.536849+0000","last_active":"2024-04-26T23:44:59.537979+0000","last_peered":"2024-04-26T23:44:59.537979+0000","last_clean":"2024-04-26T23:44:59.537979+0000","last_became_active":"2024-04-26T23:43:07.536214+0000","last_became_peered":"2024-04-26T23:43:07.536214+0000","last_unstale":"2024-04-26T23:44:59.537979+0000","last_undegraded":"2024-04-26T23:44:59.537979+0000","last_fullsized":"2024-04-26T23:44:59.537979+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:33:16.782368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.566803+0000","last_change":"2024-04-26T23:43:02.464181+0000","last_active":"2024-04-26T23:43:59.566803+0000","last_peered":"2024-04-26T23:43:59.566803+0000","last_clean":"2024-04-26T23:43:59.566803+0000","last_became_active":"2024-04-26T23:43:02.463830+0000","last_became_peered":"2024-04-26T23:43:02.463830+0000","last_unstale":"2024-04-26T23:43:59.566803+0000","last_undegraded":"2024-04-26T23:43:59.566803+0000","last_fullsized":"2024-04-26T23:43:59.566803+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:14:52.831302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.572417+0000","last_change":"2024-04-26T23:43:03.298497+0000","last_active":"2024-04-26T23:43:59.572417+0000","last_peered":"2024-04-26T23:43:59.572417+0000","last_clean":"2024-04-26T23:43:59.572417+0000","last_became_active":"2024-04-26T23:43:03.298194+0000","last_became_peered":"2024-04-26T23:43:03.298194+0000","last_unstale":"2024-04-26T23:43:59.572417+0000","last_undegraded":"2024-04-26T23:43:59.572417+0000","last_fullsized":"2024-04-26T23:43:59.572417+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:39:19.007294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544441+0000","last_change":"2024-04-26T23:43:07.527962+0000","last_active":"2024-04-26T23:45:00.544441+0000","last_peered":"2024-04-26T23:45:00.544441+0000","last_clean":"2024-04-26T23:45:00.544441+0000","last_became_active":"2024-04-26T23:43:07.527386+0000","last_became_peered":"2024-04-26T23:43:07.527386+0000","last_unstale":"2024-04-26T23:45:00.544441+0000","last_undegraded":"2024-04-26T23:45:00.544441+0000","last_fullsized":"2024-04-26T23:45:00.544441+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:57:21.711670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.538734+0000","last_change":"2024-04-26T23:43:02.453988+0000","last_active":"2024-04-26T23:45:00.538734+0000","last_peered":"2024-04-26T23:45:00.538734+0000","last_clean":"2024-04-26T23:45:00.538734+0000","last_became_active":"2024-04-26T23:43:02.453654+0000","last_became_peered":"2024-04-26T23:43:02.453654+0000","last_unstale":"2024-04-26T23:45:00.538734+0000","last_undegraded":"2024-04-26T23:45:00.538734+0000","last_fullsized":"2024-04-26T23:45:00.538734+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:46:08.718705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.571441+0000","last_change":"2024-04-26T23:43:03.305861+0000","last_active":"2024-04-26T23:43:59.571441+0000","last_peered":"2024-04-26T23:43:59.571441+0000","last_clean":"2024-04-26T23:43:59.571441+0000","last_became_active":"2024-04-26T23:43:03.305301+0000","last_became_peered":"2024-04-26T23:43:03.305301+0000","last_unstale":"2024-04-26T23:43:59.571441+0000","last_undegraded":"2024-04-26T23:43:59.571441+0000","last_fullsized":"2024-04-26T23:43:59.571441+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:08:24.174443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.537998+0000","last_change":"2024-04-26T23:43:07.529914+0000","last_active":"2024-04-26T23:45:00.537998+0000","last_peered":"2024-04-26T23:45:00.537998+0000","last_clean":"2024-04-26T23:45:00.537998+0000","last_became_active":"2024-04-26T23:43:07.529509+0000","last_became_peered":"2024-04-26T23:43:07.529509+0000","last_unstale":"2024-04-26T23:45:00.537998+0000","last_undegraded":"2024-04-26T23:45:00.537998+0000","last_fullsized":"2024-04-26T23:45:00.537998+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:50:23.293672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.552220+0000","last_change":"2024-04-26T23:43:02.452096+0000","last_active":"2024-04-26T23:45:00.552220+0000","last_peered":"2024-04-26T23:45:00.552220+0000","last_clean":"2024-04-26T23:45:00.552220+0000","last_became_active":"2024-04-26T23:43:02.450039+0000","last_became_peered":"2024-04-26T23:43:02.450039+0000","last_unstale":"2024-04-26T23:45:00.552220+0000","last_undegraded":"2024-04-26T23:45:00.552220+0000","last_fullsized":"2024-04-26T23:45:00.552220+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:47:03.062575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.572594+0000","last_change":"2024-04-26T23:43:03.294790+0000","last_active":"2024-04-26T23:43:59.572594+0000","last_peered":"2024-04-26T23:43:59.572594+0000","last_clean":"2024-04-26T23:43:59.572594+0000","last_became_active":"2024-04-26T23:43:03.294460+0000","last_became_peered":"2024-04-26T23:43:03.294460+0000","last_unstale":"2024-04-26T23:43:59.572594+0000","last_undegraded":"2024-04-26T23:43:59.572594+0000","last_fullsized":"2024-04-26T23:43:59.572594+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:57:40.879315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-26T23:44:59.537779+0000","last_change":"2024-04-26T23:43:03.284344+0000","last_active":"2024-04-26T23:44:59.537779+0000","last_peered":"2024-04-26T23:44:59.537779+0000","last_clean":"2024-04-26T23:44:59.537779+0000","last_became_active":"2024-04-26T23:43:03.283954+0000","last_became_peered":"2024-04-26T23:43:03.283954+0000","last_unstale":"2024-04-26T23:44:59.537779+0000","last_undegraded":"2024-04-26T23:44:59.537779+0000","last_fullsized":"2024-04-26T23:44:59.537779+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:12:56.516067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.574959+0000","last_change":"2024-04-26T23:43:59.574959+0000","last_active":"2024-04-26T23:43:59.574959+0000","last_peered":"2024-04-26T23:43:59.574959+0000","last_clean":"2024-04-26T23:43:59.574959+0000","last_became_active":"2024-04-26T23:43:59.574588+0000","last_became_peered":"2024-04-26T23:43:59.574588+0000","last_unstale":"2024-04-26T23:43:59.574959+0000","last_undegraded":"2024-04-26T23:43:59.574959+0000","last_fullsized":"2024-04-26T23:43:59.574959+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:52:56.689439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.574946+0000","last_change":"2024-04-26T23:43:59.574946+0000","last_active":"2024-04-26T23:43:59.574946+0000","last_peered":"2024-04-26T23:43:59.574946+0000","last_clean":"2024-04-26T23:43:59.574946+0000","last_became_active":"2024-04-26T23:43:59.574238+0000","last_became_peered":"2024-04-26T23:43:59.574238+0000","last_unstale":"2024-04-26T23:43:59.574946+0000","last_undegraded":"2024-04-26T23:43:59.574946+0000","last_fullsized":"2024-04-26T23:43:59.574946+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:58:41.619868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,0],"acting":[2,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.545087+0000","last_change":"2024-04-26T23:43:03.304580+0000","last_active":"2024-04-26T23:45:00.545087+0000","last_peered":"2024-04-26T23:45:00.545087+0000","last_clean":"2024-04-26T23:45:00.545087+0000","last_became_active":"2024-04-26T23:43:03.303128+0000","last_became_peered":"2024-04-26T23:43:03.303128+0000","last_unstale":"2024-04-26T23:45:00.545087+0000","last_undegraded":"2024-04-26T23:45:00.545087+0000","last_fullsized":"2024-04-26T23:45:00.545087+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:11:36.868298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.553732+0000","last_change":"2024-04-26T23:43:03.306441+0000","last_active":"2024-04-26T23:45:00.553732+0000","last_peered":"2024-04-26T23:45:00.553732+0000","last_clean":"2024-04-26T23:45:00.553732+0000","last_became_active":"2024-04-26T23:43:03.305209+0000","last_became_peered":"2024-04-26T23:43:03.305209+0000","last_unstale":"2024-04-26T23:45:00.553732+0000","last_undegraded":"2024-04-26T23:45:00.553732+0000","last_fullsized":"2024-04-26T23:45:00.553732+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:41:39.757001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-26T23:44:59.538411+0000","last_change":"2024-04-26T23:43:03.284291+0000","last_active":"2024-04-26T23:44:59.538411+0000","last_peered":"2024-04-26T23:44:59.538411+0000","last_clean":"2024-04-26T23:44:59.538411+0000","last_became_active":"2024-04-26T23:43:03.284006+0000","last_became_peered":"2024-04-26T23:43:03.284006+0000","last_unstale":"2024-04-26T23:44:59.538411+0000","last_undegraded":"2024-04-26T23:44:59.538411+0000","last_fullsized":"2024-04-26T23:44:59.538411+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:24:01.386322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T23:43:59.567715+0000","last_change":"2024-04-26T23:43:03.286478+0000","last_active":"2024-04-26T23:43:59.567715+0000","last_peered":"2024-04-26T23:43:59.567715+0000","last_clean":"2024-04-26T23:43:59.567715+0000","last_became_active":"2024-04-26T23:43:03.286115+0000","last_became_peered":"2024-04-26T23:43:03.286115+0000","last_unstale":"2024-04-26T23:43:59.567715+0000","last_undegraded":"2024-04-26T23:43:59.567715+0000","last_fullsized":"2024-04-26T23:43:59.567715+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:34:57.534124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.552134+0000","last_change":"2024-04-26T23:43:59.872042+0000","last_active":"2024-04-26T23:45:00.552134+0000","last_peered":"2024-04-26T23:45:00.552134+0000","last_clean":"2024-04-26T23:45:00.552134+0000","last_became_active":"2024-04-26T23:43:59.870288+0000","last_became_peered":"2024-04-26T23:43:59.870288+0000","last_unstale":"2024-04-26T23:45:00.552134+0000","last_undegraded":"2024-04-26T23:45:00.552134+0000","last_fullsized":"2024-04-26T23:45:00.552134+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:16:38.518370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.538251+0000","last_change":"2024-04-26T23:43:03.300602+0000","last_active":"2024-04-26T23:45:00.538251+0000","last_peered":"2024-04-26T23:45:00.538251+0000","last_clean":"2024-04-26T23:45:00.538251+0000","last_became_active":"2024-04-26T23:43:03.300225+0000","last_became_peered":"2024-04-26T23:43:03.300225+0000","last_unstale":"2024-04-26T23:45:00.538251+0000","last_undegraded":"2024-04-26T23:45:00.538251+0000","last_fullsized":"2024-04-26T23:45:00.538251+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:52:36.638153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.543535+0000","last_change":"2024-04-26T23:43:03.305631+0000","last_active":"2024-04-26T23:45:00.543535+0000","last_peered":"2024-04-26T23:45:00.543535+0000","last_clean":"2024-04-26T23:45:00.543535+0000","last_became_active":"2024-04-26T23:43:03.300193+0000","last_became_peered":"2024-04-26T23:43:03.300193+0000","last_unstale":"2024-04-26T23:45:00.543535+0000","last_undegraded":"2024-04-26T23:45:00.543535+0000","last_fullsized":"2024-04-26T23:45:00.543535+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:27:08.355753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-26T23:44:59.538596+0000","last_change":"2024-04-26T23:43:03.309514+0000","last_active":"2024-04-26T23:44:59.538596+0000","last_peered":"2024-04-26T23:44:59.538596+0000","last_clean":"2024-04-26T23:44:59.538596+0000","last_became_active":"2024-04-26T23:43:03.309085+0000","last_became_peered":"2024-04-26T23:43:03.309085+0000","last_unstale":"2024-04-26T23:44:59.538596+0000","last_undegraded":"2024-04-26T23:44:59.538596+0000","last_fullsized":"2024-04-26T23:44:59.538596+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:08:45.649513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.547848+0000","last_change":"2024-04-26T23:43:03.303553+0000","last_active":"2024-04-26T23:45:00.547848+0000","last_peered":"2024-04-26T23:45:00.547848+0000","last_clean":"2024-04-26T23:45:00.547848+0000","last_became_active":"2024-04-26T23:43:03.302979+0000","last_became_peered":"2024-04-26T23:43:03.302979+0000","last_unstale":"2024-04-26T23:45:00.547848+0000","last_undegraded":"2024-04-26T23:45:00.547848+0000","last_fullsized":"2024-04-26T23:45:00.547848+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:44:25.809138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-04-26T23:44:59.538194+0000","last_change":"2024-04-26T23:43:03.288433+0000","last_active":"2024-04-26T23:44:59.538194+0000","last_peered":"2024-04-26T23:44:59.538194+0000","last_clean":"2024-04-26T23:44:59.538194+0000","last_became_active":"2024-04-26T23:43:03.288125+0000","last_became_peered":"2024-04-26T23:43:03.288125+0000","last_unstale":"2024-04-26T23:44:59.538194+0000","last_undegraded":"2024-04-26T23:44:59.538194+0000","last_fullsized":"2024-04-26T23:44:59.538194+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:53:21.673755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":282161,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":59218,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":590,"num_read_kb":357,"num_write":164,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":177632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141,"ondisk_log_size":141,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383719,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1007616,"data_stored":510413,"data_compressed":23547,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645356,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32316,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712836,"statfs":{"total":95995035648,"available":95961944064,"internally_reserved":0,"allocated":1159168,"data_stored":1162375,"data_compressed":62415,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808881,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1028096,"data_stored":514745,"data_compressed":23547,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070519,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1024000,"data_stored":510569,"data_compressed":23547,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332157,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32220,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712932,"statfs":{"total":95995035648,"available":95962042368,"internally_reserved":0,"allocated":1060864,"data_stored":1097287,"data_compressed":56460,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495683,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1015808,"data_stored":515302,"data_compressed":26528,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574943,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1064960,"data_stored":560561,"data_compressed":23547,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738468,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":516690,"data_compressed":26528,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411418,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":552608,"data_compressed":20573,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182483,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1028096,"data_stored":515496,"data_compressed":26528,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920847,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1036288,"data_stored":520455,"data_compressed":26528,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757322,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32248,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712904,"statfs":{"total":95995035648,"available":95962013696,"internally_reserved":0,"allocated":1093632,"data_stored":1111543,"data_compressed":62415,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34242,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34242,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34242,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35887,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35887,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35887,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35887,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35887,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":54826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":56455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":56795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T23:45:06.359 INFO:tasks.cephadm.ceph_manager.ceph:PG 2.f is not active+clean 2024-04-26T23:45:06.359 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '2.f', 'version': "89'2", 'reported_seq': 2, 'reported_epoch': 92, 'state': 'peering', 'last_fresh': '2024-04-26T23:43:08.418042+0000', 'last_change': '2024-04-26T23:45:00.495897+0000', 'last_active': '2024-04-26T23:45:00.495897+0000', 'last_peered': '2024-04-26T23:43:08.418042+0000', 'last_clean': '2024-04-26T23:43:08.418042+0000', 'last_became_active': '2024-04-26T23:43:02.440294+0000', 'last_became_peered': '2024-04-26T23:43:02.440294+0000', 'last_unstale': '2024-04-26T23:43:08.418042+0000', 'last_undegraded': '2024-04-26T23:43:08.418042+0000', 'last_fullsized': '2024-04-26T23:43:08.418042+0000', 'mapping_epoch': 92, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 82, 'last_epoch_clean': 83, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-26T23:43:01.410793+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-26T23:43:01.410793+0000', 'last_clean_scrub_stamp': '2024-04-26T23:43:01.410793+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 2, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 1, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 1, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [11, 5, 4], 'acting': [11, 5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 11, 'acting_primary': 11, 'purged_snaps': []} 2024-04-26T23:45:06.360 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph status --format=json 2024-04-26T23:45:06.833 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:45:06.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:06 smithi076 ceph-mon[32774]: from='client.25064 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T23:45:06.980 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:06 smithi072 ceph-mon[28184]: from='client.25064 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T23:45:07.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:06 smithi136 ceph-mon[33050]: from='client.25064 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T23:45:07.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:07 smithi072 ceph-mon[28184]: pgmap v387: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 127 B/s wr, 7 op/s; 2 B/s, 0 objects/s recovering 2024-04-26T23:45:07.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:07 smithi072 ceph-mon[28184]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-26T23:45:07.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:07 smithi072 ceph-mon[28184]: Cluster is now healthy 2024-04-26T23:45:07.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:07 smithi076 ceph-mon[32774]: pgmap v387: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 127 B/s wr, 7 op/s; 2 B/s, 0 objects/s recovering 2024-04-26T23:45:07.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:07 smithi076 ceph-mon[32774]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-26T23:45:07.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:07 smithi076 ceph-mon[32774]: Cluster is now healthy 2024-04-26T23:45:08.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:07 smithi136 ceph-mon[33050]: pgmap v387: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 127 B/s wr, 7 op/s; 2 B/s, 0 objects/s recovering 2024-04-26T23:45:08.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:07 smithi136 ceph-mon[33050]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-26T23:45:08.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:07 smithi136 ceph-mon[33050]: Cluster is now healthy 2024-04-26T23:45:08.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:08 smithi076 ceph-mon[32774]: pgmap v388: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 114 B/s wr, 3 op/s; 2 B/s, 0 objects/s recovering 2024-04-26T23:45:08.978 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:08 smithi072 ceph-mon[28184]: pgmap v388: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 114 B/s wr, 3 op/s; 2 B/s, 0 objects/s recovering 2024-04-26T23:45:09.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:08 smithi136 ceph-mon[33050]: pgmap v388: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 114 B/s wr, 3 op/s; 2 B/s, 0 objects/s recovering 2024-04-26T23:45:09.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.138+0000 7f27ae400700 1 -- 172.21.15.72:0/890699898 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27a8105640 msgr2=0x7f27a8105ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:09.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.138+0000 7f27ae400700 1 --2- 172.21.15.72:0/890699898 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27a8105640 0x7f27a8105ac0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2794009990 tx=0x7f2794009ca0 comp rx=0 tx=0).stop 2024-04-26T23:45:09.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.138+0000 7f27ae400700 1 -- 172.21.15.72:0/890699898 shutdown_connections 2024-04-26T23:45:09.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.138+0000 7f27ae400700 1 --2- 172.21.15.72:0/890699898 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f27a8106000 0x7f27a8112510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:09.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.138+0000 7f27ae400700 1 --2- 172.21.15.72:0/890699898 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f27a810b640 0x7f27a810ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:09.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.138+0000 7f27ae400700 1 --2- 172.21.15.72:0/890699898 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27a8105640 0x7f27a8105ac0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:09.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.138+0000 7f27ae400700 1 -- 172.21.15.72:0/890699898 >> 172.21.15.72:0/890699898 conn(0x7f27a8101130 msgr2=0x7f27a8103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:09.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.138+0000 7f27ae400700 1 -- 172.21.15.72:0/890699898 shutdown_connections 2024-04-26T23:45:09.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.139+0000 7f27ae400700 1 -- 172.21.15.72:0/890699898 wait complete. 2024-04-26T23:45:09.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.139+0000 7f27ae400700 1 Processor -- start 2024-04-26T23:45:09.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.140+0000 7f27ae400700 1 -- start start 2024-04-26T23:45:09.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.140+0000 7f27ae400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f27a8105640 0x7f27a81a1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:09.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.140+0000 7f27ae400700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f27a8106000 0x7f27a81a1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:09.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.140+0000 7f27ae400700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27a810b640 0x7f27a819bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:09.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.140+0000 7f27ae400700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f27a8115870 con 0x7f27a810b640 2024-04-26T23:45:09.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.140+0000 7f27ae400700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f27a81156f0 con 0x7f27a8105640 2024-04-26T23:45:09.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.140+0000 7f27ae400700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f27a81159f0 con 0x7f27a8106000 2024-04-26T23:45:09.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.140+0000 7f27a7400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f27a8105640 0x7f27a81a1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:09.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.140+0000 7f27a6a00700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f27a8106000 0x7f27a81a1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:09.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.140+0000 7f27a6a00700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f27a8106000 0x7f27a81a1fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.72:40826/0 (socket says 172.21.15.72:40826) 2024-04-26T23:45:09.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.140+0000 7f27a6a00700 1 -- 172.21.15.72:0/3927530106 learned_addr learned my addr 172.21.15.72:0/3927530106 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:45:09.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.140+0000 7f27a7400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f27a8105640 0x7f27a81a1a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:32916/0 (socket says 172.21.15.72:32916) 2024-04-26T23:45:09.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.141+0000 7f27a6a00700 1 -- 172.21.15.72:0/3927530106 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f27a8105640 msgr2=0x7f27a81a1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:09.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.141+0000 7f27a6a00700 1 --2- 172.21.15.72:0/3927530106 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f27a8105640 0x7f27a81a1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:09.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.141+0000 7f27a6a00700 1 -- 172.21.15.72:0/3927530106 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27a810b640 msgr2=0x7f27a819bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:09.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.141+0000 7f27a7e00700 1 --2- 172.21.15.72:0/3927530106 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27a810b640 0x7f27a819bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:09.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.141+0000 7f27a6a00700 1 --2- 172.21.15.72:0/3927530106 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27a810b640 0x7f27a819bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:09.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.141+0000 7f27a6a00700 1 -- 172.21.15.72:0/3927530106 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27a819c410 con 0x7f27a8106000 2024-04-26T23:45:09.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.141+0000 7f27a7400700 1 --2- 172.21.15.72:0/3927530106 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f27a8105640 0x7f27a81a1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:45:09.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.141+0000 7f27a7e00700 1 --2- 172.21.15.72:0/3927530106 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27a810b640 0x7f27a819bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T23:45:09.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.141+0000 7f27a6a00700 1 --2- 172.21.15.72:0/3927530106 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f27a8106000 0x7f27a81a1fa0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2794005910 tx=0x7f279400b670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:09.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.142+0000 7f279fe00700 1 -- 172.21.15.72:0/3927530106 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2794027070 con 0x7f27a8106000 2024-04-26T23:45:09.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.142+0000 7f279fe00700 1 -- 172.21.15.72:0/3927530106 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2794022470 con 0x7f27a8106000 2024-04-26T23:45:09.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.142+0000 7f27ae400700 1 -- 172.21.15.72:0/3927530106 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27a819c6a0 con 0x7f27a8106000 2024-04-26T23:45:09.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.142+0000 7f279fe00700 1 -- 172.21.15.72:0/3927530106 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2794017650 con 0x7f27a8106000 2024-04-26T23:45:09.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.142+0000 7f27ae400700 1 -- 172.21.15.72:0/3927530106 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27a81a8810 con 0x7f27a8106000 2024-04-26T23:45:09.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.143+0000 7f27ae400700 1 -- 172.21.15.72:0/3927530106 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27a8106d80 con 0x7f27a8106000 2024-04-26T23:45:09.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.145+0000 7f279fe00700 1 -- 172.21.15.72:0/3927530106 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f279402a070 con 0x7f27a8106000 2024-04-26T23:45:09.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.146+0000 7f279fe00700 1 --2- 172.21.15.72:0/3927530106 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f278c0a3cd0 0x7f278c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:09.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.147+0000 7f279fe00700 1 -- 172.21.15.72:0/3927530106 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f27940d1720 con 0x7f27a8106000 2024-04-26T23:45:09.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.147+0000 7f27a7400700 1 --2- 172.21.15.72:0/3927530106 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f278c0a3cd0 0x7f278c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:09.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.147+0000 7f27a7400700 1 --2- 172.21.15.72:0/3927530106 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f278c0a3cd0 0x7f278c0a6190 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f2798009fd0 tx=0x7f2798009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:09.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.151+0000 7f279fe00700 1 -- 172.21.15.72:0/3927530106 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f2794013070 con 0x7f27a8106000 2024-04-26T23:45:09.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.470+0000 7f27ae400700 1 -- 172.21.15.72:0/3927530106 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f27a802e490 con 0x7f27a8106000 2024-04-26T23:45:09.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.472+0000 7f279fe00700 1 -- 172.21.15.72:0/3927530106 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1657 (secure 0 0 0) 0x7f279409dbe0 con 0x7f27a8106000 2024-04-26T23:45:09.476 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:45:09.476 INFO:teuthology.orchestra.run.smithi072.stdout:{"fsid":"3c8453ae-0425-11ef-bc93-c7b262605968","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":539,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":93,"num_osds":12,"num_up_osds":12,"osd_up_since":1714174924,"num_in_osds":12,"osd_in_since":1714174904,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":129}],"num_pgs":129,"num_pools":4,"num_objects":64,"data_bytes":649820,"bytes_used":366080000,"bytes_avail":1151574347776,"bytes_total":1151940427776,"recovering_objects_per_sec":0,"recovering_bytes_per_sec":2,"recovering_keys_per_sec":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"read_bytes_sec":1605,"write_bytes_sec":114,"read_op_per_sec":3,"write_op_per_sec":0},"fsmap":{"epoch":23,"id":1,"up":3,"in":3,"max":3,"by_rank":[{"filesystem_id":1,"rank":2,"name":"c","status":"up:standby-replay","gid":24514},{"filesystem_id":1,"rank":1,"name":"h","status":"up:standby-replay","gid":24533},{"filesystem_id":1,"rank":0,"name":"i","status":"up:standby-replay","gid":24535},{"filesystem_id":1,"rank":2,"name":"g","status":"up:active","gid":24553},{"filesystem_id":1,"rank":1,"name":"a","status":"up:active","gid":24571},{"filesystem_id":1,"rank":0,"name":"b","status":"up:active","gid":24590}],"up:standby":6},"mgrmap":{"available":true,"num_standbys":2,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.72:8443/"}},"servicemap":{"epoch":27,"modified":"2024-04-26T23:44:36.449778+0000","services":{}},"progress_events":{}} 2024-04-26T23:45:09.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.475+0000 7f27ae400700 1 -- 172.21.15.72:0/3927530106 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f278c0a3cd0 msgr2=0x7f278c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:09.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.475+0000 7f27ae400700 1 --2- 172.21.15.72:0/3927530106 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f278c0a3cd0 0x7f278c0a6190 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f2798009fd0 tx=0x7f2798009260 comp rx=0 tx=0).stop 2024-04-26T23:45:09.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.475+0000 7f27ae400700 1 -- 172.21.15.72:0/3927530106 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f27a8106000 msgr2=0x7f27a81a1fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:09.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.475+0000 7f27ae400700 1 --2- 172.21.15.72:0/3927530106 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f27a8106000 0x7f27a81a1fa0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2794005910 tx=0x7f279400b670 comp rx=0 tx=0).stop 2024-04-26T23:45:09.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.475+0000 7f27ae400700 1 -- 172.21.15.72:0/3927530106 shutdown_connections 2024-04-26T23:45:09.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.475+0000 7f27ae400700 1 --2- 172.21.15.72:0/3927530106 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f278c0a3cd0 0x7f278c0a6190 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:09.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.475+0000 7f27ae400700 1 --2- 172.21.15.72:0/3927530106 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27a810b640 0x7f27a819bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:09.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.475+0000 7f27ae400700 1 --2- 172.21.15.72:0/3927530106 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f27a8105640 0x7f27a81a1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:09.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.475+0000 7f27ae400700 1 --2- 172.21.15.72:0/3927530106 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f27a8106000 0x7f27a81a1fa0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:09.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.475+0000 7f27ae400700 1 -- 172.21.15.72:0/3927530106 >> 172.21.15.72:0/3927530106 conn(0x7f27a8101130 msgr2=0x7f27a8101760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:09.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.476+0000 7f27ae400700 1 -- 172.21.15.72:0/3927530106 shutdown_connections 2024-04-26T23:45:09.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:09.476+0000 7f27ae400700 1 -- 172.21.15.72:0/3927530106 wait complete. 2024-04-26T23:45:09.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:09 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/3927530106' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-26T23:45:09.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:09 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/3927530106' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-26T23:45:10.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:09 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/3927530106' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-26T23:45:10.264 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-04-26T23:45:10.264 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T23:45:10.704 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:45:10.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:10 smithi076 ceph-mon[32774]: pgmap v389: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 102 B/s wr, 6 op/s; 2 B/s, 0 objects/s recovering 2024-04-26T23:45:11.004 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:10 smithi072 ceph-mon[28184]: pgmap v389: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 102 B/s wr, 6 op/s; 2 B/s, 0 objects/s recovering 2024-04-26T23:45:11.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:10 smithi136 ceph-mon[33050]: pgmap v389: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 102 B/s wr, 6 op/s; 2 B/s, 0 objects/s recovering 2024-04-26T23:45:12.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:12.918+0000 7f951fe00700 1 -- 172.21.15.72:0/368909547 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f951806c360 msgr2=0x7f95181102d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:12.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:12.918+0000 7f951fe00700 1 --2- 172.21.15.72:0/368909547 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f951806c360 0x7f95181102d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f951000b0a0 tx=0x7f951000b3b0 comp rx=0 tx=0).stop 2024-04-26T23:45:12.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:12.919+0000 7f951fe00700 1 -- 172.21.15.72:0/368909547 shutdown_connections 2024-04-26T23:45:12.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:12.919+0000 7f951fe00700 1 --2- 172.21.15.72:0/368909547 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f951806c360 0x7f95181102d0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:12.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:12.919+0000 7f951fe00700 1 --2- 172.21.15.72:0/368909547 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9518109490 0x7f9518109870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:12.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:12.919+0000 7f951fe00700 1 --2- 172.21.15.72:0/368909547 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f951806b9c0 0x7f951806be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:12.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:12.919+0000 7f951fe00700 1 -- 172.21.15.72:0/368909547 >> 172.21.15.72:0/368909547 conn(0x7f9518079290 msgr2=0x7f95180796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:12.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:12.919+0000 7f951fe00700 1 -- 172.21.15.72:0/368909547 shutdown_connections 2024-04-26T23:45:12.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:12.919+0000 7f951fe00700 1 -- 172.21.15.72:0/368909547 wait complete. 2024-04-26T23:45:12.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:12.920+0000 7f951fe00700 1 Processor -- start 2024-04-26T23:45:12.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:12.920+0000 7f951fe00700 1 -- start start 2024-04-26T23:45:12.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:12.920+0000 7f951fe00700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f951806b9c0 0x7f95181a18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:12.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:12.920+0000 7f951fe00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f951806c360 0x7f95181a1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:12.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:12.921+0000 7f951fe00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9518109490 0x7f951819ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:12.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:12.921+0000 7f951fe00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f9518113590 con 0x7f951806c360 2024-04-26T23:45:12.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:12.921+0000 7f951fe00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f9518113410 con 0x7f9518109490 2024-04-26T23:45:12.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:12.921+0000 7f951fe00700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f9518113710 con 0x7f951806b9c0 2024-04-26T23:45:12.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:12.921+0000 7f951ca00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f951806c360 0x7f95181a1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:12.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:12.921+0000 7f951ca00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f951806c360 0x7f95181a1e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:50118/0 (socket says 172.21.15.72:50118) 2024-04-26T23:45:12.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:12.921+0000 7f951ca00700 1 -- 172.21.15.72:0/1613669628 learned_addr learned my addr 172.21.15.72:0/1613669628 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:45:12.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:12.921+0000 7f951d400700 1 --2- 172.21.15.72:0/1613669628 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f951806b9c0 0x7f95181a18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:12.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:12.921+0000 7f951de00700 1 --2- 172.21.15.72:0/1613669628 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9518109490 0x7f951819ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:12.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:12.921+0000 7f951de00700 1 -- 172.21.15.72:0/1613669628 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f951806b9c0 msgr2=0x7f95181a18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:12.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:12.921+0000 7f951de00700 1 --2- 172.21.15.72:0/1613669628 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f951806b9c0 0x7f95181a18c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:12.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:12.921+0000 7f951de00700 1 -- 172.21.15.72:0/1613669628 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f951806c360 msgr2=0x7f95181a1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:12.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:12.921+0000 7f951de00700 1 --2- 172.21.15.72:0/1613669628 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f951806c360 0x7f95181a1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:12.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:12.922+0000 7f951de00700 1 -- 172.21.15.72:0/1613669628 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f951819c1e0 con 0x7f9518109490 2024-04-26T23:45:12.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:12.922+0000 7f951ca00700 1 --2- 172.21.15.72:0/1613669628 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f951806c360 0x7f95181a1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:45:12.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:12.922+0000 7f951de00700 1 --2- 172.21.15.72:0/1613669628 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9518109490 0x7f951819ba80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f951000b070 tx=0x7f9510004d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:12.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:12.922+0000 7f9516000700 1 -- 172.21.15.72:0/1613669628 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9510021070 con 0x7f9518109490 2024-04-26T23:45:12.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:12.922+0000 7f951fe00700 1 -- 172.21.15.72:0/1613669628 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f951819c470 con 0x7f9518109490 2024-04-26T23:45:12.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:12.922+0000 7f9516000700 1 -- 172.21.15.72:0/1613669628 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9510007e30 con 0x7f9518109490 2024-04-26T23:45:12.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:12.922+0000 7f9516000700 1 -- 172.21.15.72:0/1613669628 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95100045d0 con 0x7f9518109490 2024-04-26T23:45:12.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:12.922+0000 7f951fe00700 1 -- 172.21.15.72:0/1613669628 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95181a8790 con 0x7f9518109490 2024-04-26T23:45:12.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:12.924+0000 7f951fe00700 1 -- 172.21.15.72:0/1613669628 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f951806a110 con 0x7f9518109490 2024-04-26T23:45:12.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:12.925+0000 7f9516000700 1 -- 172.21.15.72:0/1613669628 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f9510003c00 con 0x7f9518109490 2024-04-26T23:45:12.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:12.927+0000 7f9516000700 1 --2- 172.21.15.72:0/1613669628 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f95000a3cd0 0x7f95000a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:12.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:12.927+0000 7f9516000700 1 -- 172.21.15.72:0/1613669628 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f951000f020 con 0x7f9518109490 2024-04-26T23:45:12.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:12.927+0000 7f951d400700 1 --2- 172.21.15.72:0/1613669628 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f95000a3cd0 0x7f95000a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:12.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:12.928+0000 7f951d400700 1 --2- 172.21.15.72:0/1613669628 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f95000a3cd0 0x7f95000a6190 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f951819cbc0 tx=0x7f950c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:12.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:12.931+0000 7f9516000700 1 -- 172.21.15.72:0/1613669628 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f951002a020 con 0x7f9518109490 2024-04-26T23:45:13.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:13.151+0000 7f951fe00700 1 -- 172.21.15.72:0/1613669628 --> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9518063860 con 0x7f95000a3cd0 2024-04-26T23:45:13.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:13.180+0000 7f9516000700 1 -- 172.21.15.72:0/1613669628 <== mgr.14152 v2:172.21.15.72:6800/4281177709 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340653 (secure 0 0 0) 0x7f9518063860 con 0x7f95000a3cd0 2024-04-26T23:45:13.186 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:45:13.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:13.188+0000 7f951fe00700 1 -- 172.21.15.72:0/1613669628 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f95000a3cd0 msgr2=0x7f95000a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:13.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:13.189+0000 7f951fe00700 1 --2- 172.21.15.72:0/1613669628 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f95000a3cd0 0x7f95000a6190 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f951819cbc0 tx=0x7f950c009260 comp rx=0 tx=0).stop 2024-04-26T23:45:13.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:13.189+0000 7f951fe00700 1 -- 172.21.15.72:0/1613669628 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9518109490 msgr2=0x7f951819ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:13.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:13.189+0000 7f951fe00700 1 --2- 172.21.15.72:0/1613669628 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9518109490 0x7f951819ba80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f951000b070 tx=0x7f9510004d00 comp rx=0 tx=0).stop 2024-04-26T23:45:13.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:13.189+0000 7f951fe00700 1 -- 172.21.15.72:0/1613669628 shutdown_connections 2024-04-26T23:45:13.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:13.189+0000 7f951fe00700 1 --2- 172.21.15.72:0/1613669628 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f95000a3cd0 0x7f95000a6190 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:13.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:13.189+0000 7f951fe00700 1 --2- 172.21.15.72:0/1613669628 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9518109490 0x7f951819ba80 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:13.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:13.189+0000 7f951fe00700 1 --2- 172.21.15.72:0/1613669628 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f951806b9c0 0x7f95181a18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:13.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:13.189+0000 7f951fe00700 1 --2- 172.21.15.72:0/1613669628 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f951806c360 0x7f95181a1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:13.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:13.189+0000 7f951fe00700 1 -- 172.21.15.72:0/1613669628 >> 172.21.15.72:0/1613669628 conn(0x7f9518079290 msgr2=0x7f951810e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:13.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:13.189+0000 7f951fe00700 1 -- 172.21.15.72:0/1613669628 shutdown_connections 2024-04-26T23:45:13.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:13.190+0000 7f951fe00700 1 -- 172.21.15.72:0/1613669628 wait complete. 2024-04-26T23:45:13.204 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-04-26T23:45:13.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:13 smithi072 ceph-mon[28184]: pgmap v390: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s; 1 B/s, 0 objects/s recovering 2024-04-26T23:45:13.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:13 smithi136 ceph-mon[33050]: pgmap v390: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s; 1 B/s, 0 objects/s recovering 2024-04-26T23:45:13.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:13 smithi076 ceph-mon[32774]: pgmap v390: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s; 1 B/s, 0 objects/s recovering 2024-04-26T23:45:14.088 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":390,"stamp":"2024-04-26T23:45:12.467235+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":649820,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":746,"num_read_kb":481,"num_write":347,"num_write_kb":3050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":1180870,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":280,"ondisk_log_size":280,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":357500,"kb_used_data":12428,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124584324,"statfs":{"total":1151940427776,"available":1151574347776,"internally_reserved":0,"allocated":12726272,"data_stored":8164070,"data_compressed":425997,"data_compressed_allocated":577536,"data_compressed_original":2654208,"omap_allocated":19139,"internal_metadata":353154365},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":-2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":60,"num_read_kb":30,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.930550"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.542078+0000","last_change":"2024-04-26T23:43:03.301992+0000","last_active":"2024-04-26T23:45:00.542078+0000","last_peered":"2024-04-26T23:45:00.542078+0000","last_clean":"2024-04-26T23:45:00.542078+0000","last_became_active":"2024-04-26T23:43:03.301498+0000","last_became_peered":"2024-04-26T23:43:03.301498+0000","last_unstale":"2024-04-26T23:45:00.542078+0000","last_undegraded":"2024-04-26T23:45:00.542078+0000","last_fullsized":"2024-04-26T23:45:00.542078+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:40:53.567805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.548674+0000","last_change":"2024-04-26T23:43:03.284964+0000","last_active":"2024-04-26T23:45:00.548674+0000","last_peered":"2024-04-26T23:45:00.548674+0000","last_clean":"2024-04-26T23:45:00.548674+0000","last_became_active":"2024-04-26T23:43:03.284185+0000","last_became_peered":"2024-04-26T23:43:03.284185+0000","last_unstale":"2024-04-26T23:45:00.548674+0000","last_undegraded":"2024-04-26T23:45:00.548674+0000","last_fullsized":"2024-04-26T23:45:00.548674+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:55:35.325775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.547719+0000","last_change":"2024-04-26T23:43:03.300333+0000","last_active":"2024-04-26T23:45:00.547719+0000","last_peered":"2024-04-26T23:45:00.547719+0000","last_clean":"2024-04-26T23:45:00.547719+0000","last_became_active":"2024-04-26T23:43:03.295779+0000","last_became_peered":"2024-04-26T23:43:03.295779+0000","last_unstale":"2024-04-26T23:45:00.547719+0000","last_undegraded":"2024-04-26T23:45:00.547719+0000","last_fullsized":"2024-04-26T23:45:00.547719+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:45:06.598862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.541449+0000","last_change":"2024-04-26T23:43:03.290211+0000","last_active":"2024-04-26T23:45:00.541449+0000","last_peered":"2024-04-26T23:45:00.541449+0000","last_clean":"2024-04-26T23:45:00.541449+0000","last_became_active":"2024-04-26T23:43:03.289853+0000","last_became_peered":"2024-04-26T23:43:03.289853+0000","last_unstale":"2024-04-26T23:45:00.541449+0000","last_undegraded":"2024-04-26T23:45:00.541449+0000","last_fullsized":"2024-04-26T23:45:00.541449+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:14:45.032300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.543721+0000","last_change":"2024-04-26T23:43:03.309667+0000","last_active":"2024-04-26T23:45:00.543721+0000","last_peered":"2024-04-26T23:45:00.543721+0000","last_clean":"2024-04-26T23:45:00.543721+0000","last_became_active":"2024-04-26T23:43:03.309305+0000","last_became_peered":"2024-04-26T23:43:03.309305+0000","last_unstale":"2024-04-26T23:45:00.543721+0000","last_undegraded":"2024-04-26T23:45:00.543721+0000","last_fullsized":"2024-04-26T23:45:00.543721+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:31:56.739866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.548338+0000","last_change":"2024-04-26T23:43:03.292107+0000","last_active":"2024-04-26T23:45:00.548338+0000","last_peered":"2024-04-26T23:45:00.548338+0000","last_clean":"2024-04-26T23:45:00.548338+0000","last_became_active":"2024-04-26T23:43:03.291760+0000","last_became_peered":"2024-04-26T23:43:03.291760+0000","last_unstale":"2024-04-26T23:45:00.548338+0000","last_undegraded":"2024-04-26T23:45:00.548338+0000","last_fullsized":"2024-04-26T23:45:00.548338+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:31:54.654379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544201+0000","last_change":"2024-04-26T23:43:03.294923+0000","last_active":"2024-04-26T23:45:00.544201+0000","last_peered":"2024-04-26T23:45:00.544201+0000","last_clean":"2024-04-26T23:45:00.544201+0000","last_became_active":"2024-04-26T23:43:03.294523+0000","last_became_peered":"2024-04-26T23:43:03.294523+0000","last_unstale":"2024-04-26T23:45:00.544201+0000","last_undegraded":"2024-04-26T23:45:00.544201+0000","last_fullsized":"2024-04-26T23:45:00.544201+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:36:19.419773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.542769+0000","last_change":"2024-04-26T23:43:03.301335+0000","last_active":"2024-04-26T23:45:00.542769+0000","last_peered":"2024-04-26T23:45:00.542769+0000","last_clean":"2024-04-26T23:45:00.542769+0000","last_became_active":"2024-04-26T23:43:03.300918+0000","last_became_peered":"2024-04-26T23:43:03.300918+0000","last_unstale":"2024-04-26T23:45:00.542769+0000","last_undegraded":"2024-04-26T23:45:00.542769+0000","last_fullsized":"2024-04-26T23:45:00.542769+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:15:05.719452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.543973+0000","last_change":"2024-04-26T23:43:59.574776+0000","last_active":"2024-04-26T23:45:00.543973+0000","last_peered":"2024-04-26T23:45:00.543973+0000","last_clean":"2024-04-26T23:45:00.543973+0000","last_became_active":"2024-04-26T23:43:59.574299+0000","last_became_peered":"2024-04-26T23:43:59.574299+0000","last_unstale":"2024-04-26T23:45:00.543973+0000","last_undegraded":"2024-04-26T23:45:00.543973+0000","last_fullsized":"2024-04-26T23:45:00.543973+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:14:16.089377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,11],"acting":[1,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.548713+0000","last_change":"2024-04-26T23:43:03.287643+0000","last_active":"2024-04-26T23:45:00.548713+0000","last_peered":"2024-04-26T23:45:00.548713+0000","last_clean":"2024-04-26T23:45:00.548713+0000","last_became_active":"2024-04-26T23:43:03.287324+0000","last_became_peered":"2024-04-26T23:43:03.287324+0000","last_unstale":"2024-04-26T23:45:00.548713+0000","last_undegraded":"2024-04-26T23:45:00.548713+0000","last_fullsized":"2024-04-26T23:45:00.548713+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:50:14.816548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544950+0000","last_change":"2024-04-26T23:43:03.296421+0000","last_active":"2024-04-26T23:45:00.544950+0000","last_peered":"2024-04-26T23:45:00.544950+0000","last_clean":"2024-04-26T23:45:00.544950+0000","last_became_active":"2024-04-26T23:43:03.296060+0000","last_became_peered":"2024-04-26T23:43:03.296060+0000","last_unstale":"2024-04-26T23:45:00.544950+0000","last_undegraded":"2024-04-26T23:45:00.544950+0000","last_fullsized":"2024-04-26T23:45:00.544950+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:50:44.393446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.541254+0000","last_change":"2024-04-26T23:43:03.288562+0000","last_active":"2024-04-26T23:45:00.541254+0000","last_peered":"2024-04-26T23:45:00.541254+0000","last_clean":"2024-04-26T23:45:00.541254+0000","last_became_active":"2024-04-26T23:43:03.288219+0000","last_became_peered":"2024-04-26T23:43:03.288219+0000","last_unstale":"2024-04-26T23:45:00.541254+0000","last_undegraded":"2024-04-26T23:45:00.541254+0000","last_fullsized":"2024-04-26T23:45:00.541254+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:54:32.107146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.547419+0000","last_change":"2024-04-26T23:43:03.296840+0000","last_active":"2024-04-26T23:45:00.547419+0000","last_peered":"2024-04-26T23:45:00.547419+0000","last_clean":"2024-04-26T23:45:00.547419+0000","last_became_active":"2024-04-26T23:43:03.296501+0000","last_became_peered":"2024-04-26T23:43:03.296501+0000","last_unstale":"2024-04-26T23:45:00.547419+0000","last_undegraded":"2024-04-26T23:45:00.547419+0000","last_fullsized":"2024-04-26T23:45:00.547419+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:07:05.293388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.546627+0000","last_change":"2024-04-26T23:43:03.291739+0000","last_active":"2024-04-26T23:45:00.546627+0000","last_peered":"2024-04-26T23:45:00.546627+0000","last_clean":"2024-04-26T23:45:00.546627+0000","last_became_active":"2024-04-26T23:43:03.287839+0000","last_became_peered":"2024-04-26T23:43:03.287839+0000","last_unstale":"2024-04-26T23:45:00.546627+0000","last_undegraded":"2024-04-26T23:45:00.546627+0000","last_fullsized":"2024-04-26T23:45:00.546627+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:10:56.698324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.546051+0000","last_change":"2024-04-26T23:43:03.295767+0000","last_active":"2024-04-26T23:45:00.546051+0000","last_peered":"2024-04-26T23:45:00.546051+0000","last_clean":"2024-04-26T23:45:00.546051+0000","last_became_active":"2024-04-26T23:43:03.295339+0000","last_became_peered":"2024-04-26T23:43:03.295339+0000","last_unstale":"2024-04-26T23:45:00.546051+0000","last_undegraded":"2024-04-26T23:45:00.546051+0000","last_fullsized":"2024-04-26T23:45:00.546051+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:21:44.410953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.542951+0000","last_change":"2024-04-26T23:43:03.301687+0000","last_active":"2024-04-26T23:45:00.542951+0000","last_peered":"2024-04-26T23:45:00.542951+0000","last_clean":"2024-04-26T23:45:00.542951+0000","last_became_active":"2024-04-26T23:43:03.301355+0000","last_became_peered":"2024-04-26T23:43:03.301355+0000","last_unstale":"2024-04-26T23:45:00.542951+0000","last_undegraded":"2024-04-26T23:45:00.542951+0000","last_fullsized":"2024-04-26T23:45:00.542951+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:00:57.991022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.541916+0000","last_change":"2024-04-26T23:43:03.302109+0000","last_active":"2024-04-26T23:45:00.541916+0000","last_peered":"2024-04-26T23:45:00.541916+0000","last_clean":"2024-04-26T23:45:00.541916+0000","last_became_active":"2024-04-26T23:43:03.301725+0000","last_became_peered":"2024-04-26T23:43:03.301725+0000","last_unstale":"2024-04-26T23:45:00.541916+0000","last_undegraded":"2024-04-26T23:45:00.541916+0000","last_fullsized":"2024-04-26T23:45:00.541916+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:36:44.630125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.557182+0000","last_change":"2024-04-26T23:43:03.298880+0000","last_active":"2024-04-26T23:45:00.557182+0000","last_peered":"2024-04-26T23:45:00.557182+0000","last_clean":"2024-04-26T23:45:00.557182+0000","last_became_active":"2024-04-26T23:43:03.298464+0000","last_became_peered":"2024-04-26T23:43:03.298464+0000","last_unstale":"2024-04-26T23:45:00.557182+0000","last_undegraded":"2024-04-26T23:45:00.557182+0000","last_fullsized":"2024-04-26T23:45:00.557182+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:42:54.745002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544317+0000","last_change":"2024-04-26T23:43:03.309454+0000","last_active":"2024-04-26T23:45:00.544317+0000","last_peered":"2024-04-26T23:45:00.544317+0000","last_clean":"2024-04-26T23:45:00.544317+0000","last_became_active":"2024-04-26T23:43:03.309091+0000","last_became_peered":"2024-04-26T23:43:03.309091+0000","last_unstale":"2024-04-26T23:45:00.544317+0000","last_undegraded":"2024-04-26T23:45:00.544317+0000","last_fullsized":"2024-04-26T23:45:00.544317+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:46:12.333410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.543829+0000","last_change":"2024-04-26T23:43:07.541317+0000","last_active":"2024-04-26T23:45:00.543829+0000","last_peered":"2024-04-26T23:45:00.543829+0000","last_clean":"2024-04-26T23:45:00.543829+0000","last_became_active":"2024-04-26T23:43:07.540760+0000","last_became_peered":"2024-04-26T23:43:07.540760+0000","last_unstale":"2024-04-26T23:45:00.543829+0000","last_undegraded":"2024-04-26T23:45:00.543829+0000","last_fullsized":"2024-04-26T23:45:00.543829+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:28:55.545416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.557174+0000","last_change":"2024-04-26T23:43:02.450956+0000","last_active":"2024-04-26T23:45:00.557174+0000","last_peered":"2024-04-26T23:45:00.557174+0000","last_clean":"2024-04-26T23:45:00.557174+0000","last_became_active":"2024-04-26T23:43:02.446733+0000","last_became_peered":"2024-04-26T23:43:02.446733+0000","last_unstale":"2024-04-26T23:45:00.557174+0000","last_undegraded":"2024-04-26T23:45:00.557174+0000","last_fullsized":"2024-04-26T23:45:00.557174+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:58:47.613575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.551893+0000","last_change":"2024-04-26T23:43:03.311681+0000","last_active":"2024-04-26T23:45:00.551893+0000","last_peered":"2024-04-26T23:45:00.551893+0000","last_clean":"2024-04-26T23:45:00.551893+0000","last_became_active":"2024-04-26T23:43:03.311087+0000","last_became_peered":"2024-04-26T23:43:03.311087+0000","last_unstale":"2024-04-26T23:45:00.551893+0000","last_undegraded":"2024-04-26T23:45:00.551893+0000","last_fullsized":"2024-04-26T23:45:00.551893+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:13:00.744062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.540679+0000","last_change":"2024-04-26T23:43:07.543547+0000","last_active":"2024-04-26T23:45:00.540679+0000","last_peered":"2024-04-26T23:45:00.540679+0000","last_clean":"2024-04-26T23:45:00.540679+0000","last_became_active":"2024-04-26T23:43:07.543073+0000","last_became_peered":"2024-04-26T23:43:07.543073+0000","last_unstale":"2024-04-26T23:45:00.540679+0000","last_undegraded":"2024-04-26T23:45:00.540679+0000","last_fullsized":"2024-04-26T23:45:00.540679+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:30:05.418875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.552100+0000","last_change":"2024-04-26T23:43:02.460459+0000","last_active":"2024-04-26T23:45:00.552100+0000","last_peered":"2024-04-26T23:45:00.552100+0000","last_clean":"2024-04-26T23:45:00.552100+0000","last_became_active":"2024-04-26T23:43:02.459652+0000","last_became_peered":"2024-04-26T23:43:02.459652+0000","last_unstale":"2024-04-26T23:45:00.552100+0000","last_undegraded":"2024-04-26T23:45:00.552100+0000","last_fullsized":"2024-04-26T23:45:00.552100+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:27:19.166748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.549681+0000","last_change":"2024-04-26T23:43:03.300078+0000","last_active":"2024-04-26T23:45:00.549681+0000","last_peered":"2024-04-26T23:45:00.549681+0000","last_clean":"2024-04-26T23:45:00.549681+0000","last_became_active":"2024-04-26T23:43:03.296319+0000","last_became_peered":"2024-04-26T23:43:03.296319+0000","last_unstale":"2024-04-26T23:45:00.549681+0000","last_undegraded":"2024-04-26T23:45:00.549681+0000","last_fullsized":"2024-04-26T23:45:00.549681+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:33:51.859539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.540906+0000","last_change":"2024-04-26T23:43:07.543494+0000","last_active":"2024-04-26T23:45:00.540906+0000","last_peered":"2024-04-26T23:45:00.540906+0000","last_clean":"2024-04-26T23:45:00.540906+0000","last_became_active":"2024-04-26T23:43:07.542806+0000","last_became_peered":"2024-04-26T23:43:07.542806+0000","last_unstale":"2024-04-26T23:45:00.540906+0000","last_undegraded":"2024-04-26T23:45:00.540906+0000","last_fullsized":"2024-04-26T23:45:00.540906+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:54:40.522615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544910+0000","last_change":"2024-04-26T23:43:02.438943+0000","last_active":"2024-04-26T23:45:00.544910+0000","last_peered":"2024-04-26T23:45:00.544910+0000","last_clean":"2024-04-26T23:45:00.544910+0000","last_became_active":"2024-04-26T23:43:02.438604+0000","last_became_peered":"2024-04-26T23:43:02.438604+0000","last_unstale":"2024-04-26T23:45:00.544910+0000","last_undegraded":"2024-04-26T23:45:00.544910+0000","last_fullsized":"2024-04-26T23:45:00.544910+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:06:51.054342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.543352+0000","last_change":"2024-04-26T23:43:03.293856+0000","last_active":"2024-04-26T23:45:00.543352+0000","last_peered":"2024-04-26T23:45:00.543352+0000","last_clean":"2024-04-26T23:45:00.543352+0000","last_became_active":"2024-04-26T23:43:03.293189+0000","last_became_peered":"2024-04-26T23:43:03.293189+0000","last_unstale":"2024-04-26T23:45:00.543352+0000","last_undegraded":"2024-04-26T23:45:00.543352+0000","last_fullsized":"2024-04-26T23:45:00.543352+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:50:27.545567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.540743+0000","last_change":"2024-04-26T23:43:07.535396+0000","last_active":"2024-04-26T23:45:00.540743+0000","last_peered":"2024-04-26T23:45:00.540743+0000","last_clean":"2024-04-26T23:45:00.540743+0000","last_became_active":"2024-04-26T23:43:07.534914+0000","last_became_peered":"2024-04-26T23:43:07.534914+0000","last_unstale":"2024-04-26T23:45:00.540743+0000","last_undegraded":"2024-04-26T23:45:00.540743+0000","last_fullsized":"2024-04-26T23:45:00.540743+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:27:31.199079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544008+0000","last_change":"2024-04-26T23:43:02.442935+0000","last_active":"2024-04-26T23:45:00.544008+0000","last_peered":"2024-04-26T23:45:00.544008+0000","last_clean":"2024-04-26T23:45:00.544008+0000","last_became_active":"2024-04-26T23:43:02.442588+0000","last_became_peered":"2024-04-26T23:43:02.442588+0000","last_unstale":"2024-04-26T23:45:00.544008+0000","last_undegraded":"2024-04-26T23:45:00.544008+0000","last_fullsized":"2024-04-26T23:45:00.544008+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:05:26.442001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.538765+0000","last_change":"2024-04-26T23:43:03.289407+0000","last_active":"2024-04-26T23:45:00.538765+0000","last_peered":"2024-04-26T23:45:00.538765+0000","last_clean":"2024-04-26T23:45:00.538765+0000","last_became_active":"2024-04-26T23:43:03.287951+0000","last_became_peered":"2024-04-26T23:43:03.287951+0000","last_unstale":"2024-04-26T23:45:00.538765+0000","last_undegraded":"2024-04-26T23:45:00.538765+0000","last_fullsized":"2024-04-26T23:45:00.538765+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:24:42.687653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544134+0000","last_change":"2024-04-26T23:43:07.527019+0000","last_active":"2024-04-26T23:45:00.544134+0000","last_peered":"2024-04-26T23:45:00.544134+0000","last_clean":"2024-04-26T23:45:00.544134+0000","last_became_active":"2024-04-26T23:43:07.526516+0000","last_became_peered":"2024-04-26T23:43:07.526516+0000","last_unstale":"2024-04-26T23:45:00.544134+0000","last_undegraded":"2024-04-26T23:45:00.544134+0000","last_fullsized":"2024-04-26T23:45:00.544134+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:44:41.267144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'6","reported_seq":139,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:07.351748+0000","last_change":"2024-04-26T23:43:02.445159+0000","last_active":"2024-04-26T23:45:07.351748+0000","last_peered":"2024-04-26T23:45:07.351748+0000","last_clean":"2024-04-26T23:45:07.351748+0000","last_became_active":"2024-04-26T23:43:02.444781+0000","last_became_peered":"2024-04-26T23:43:02.444781+0000","last_unstale":"2024-04-26T23:45:07.351748+0000","last_undegraded":"2024-04-26T23:45:07.351748+0000","last_fullsized":"2024-04-26T23:45:07.351748+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:45:16.509790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":102,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.543904+0000","last_change":"2024-04-26T23:43:03.292717+0000","last_active":"2024-04-26T23:45:00.543904+0000","last_peered":"2024-04-26T23:45:00.543904+0000","last_clean":"2024-04-26T23:45:00.543904+0000","last_became_active":"2024-04-26T23:43:03.292355+0000","last_became_peered":"2024-04-26T23:43:03.292355+0000","last_unstale":"2024-04-26T23:45:00.543904+0000","last_undegraded":"2024-04-26T23:45:00.543904+0000","last_fullsized":"2024-04-26T23:45:00.543904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:06:02.063165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544064+0000","last_change":"2024-04-26T23:43:07.540758+0000","last_active":"2024-04-26T23:45:00.544064+0000","last_peered":"2024-04-26T23:45:00.544064+0000","last_clean":"2024-04-26T23:45:00.544064+0000","last_became_active":"2024-04-26T23:43:07.540305+0000","last_became_peered":"2024-04-26T23:43:07.540305+0000","last_unstale":"2024-04-26T23:45:00.544064+0000","last_undegraded":"2024-04-26T23:45:00.544064+0000","last_fullsized":"2024-04-26T23:45:00.544064+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:08:04.491544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.548197+0000","last_change":"2024-04-26T23:43:02.453650+0000","last_active":"2024-04-26T23:45:00.548197+0000","last_peered":"2024-04-26T23:45:00.548197+0000","last_clean":"2024-04-26T23:45:00.548197+0000","last_became_active":"2024-04-26T23:43:02.453319+0000","last_became_peered":"2024-04-26T23:43:02.453319+0000","last_unstale":"2024-04-26T23:45:00.548197+0000","last_undegraded":"2024-04-26T23:45:00.548197+0000","last_fullsized":"2024-04-26T23:45:00.548197+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:23:27.570603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.538674+0000","last_change":"2024-04-26T23:43:59.870417+0000","last_active":"2024-04-26T23:45:00.538674+0000","last_peered":"2024-04-26T23:45:00.538674+0000","last_clean":"2024-04-26T23:45:00.538674+0000","last_became_active":"2024-04-26T23:43:59.870053+0000","last_became_peered":"2024-04-26T23:43:59.870053+0000","last_unstale":"2024-04-26T23:45:00.538674+0000","last_undegraded":"2024-04-26T23:45:00.538674+0000","last_fullsized":"2024-04-26T23:45:00.538674+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:07:31.020611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.541897+0000","last_change":"2024-04-26T23:43:07.546026+0000","last_active":"2024-04-26T23:45:00.541897+0000","last_peered":"2024-04-26T23:45:00.541897+0000","last_clean":"2024-04-26T23:45:00.541897+0000","last_became_active":"2024-04-26T23:43:07.545668+0000","last_became_peered":"2024-04-26T23:43:07.545668+0000","last_unstale":"2024-04-26T23:45:00.541897+0000","last_undegraded":"2024-04-26T23:45:00.541897+0000","last_fullsized":"2024-04-26T23:45:00.541897+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:54:20.580267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.548684+0000","last_change":"2024-04-26T23:43:02.454937+0000","last_active":"2024-04-26T23:45:00.548684+0000","last_peered":"2024-04-26T23:45:00.548684+0000","last_clean":"2024-04-26T23:45:00.548684+0000","last_became_active":"2024-04-26T23:43:02.454585+0000","last_became_peered":"2024-04-26T23:43:02.454585+0000","last_unstale":"2024-04-26T23:45:00.548684+0000","last_undegraded":"2024-04-26T23:45:00.548684+0000","last_fullsized":"2024-04-26T23:45:00.548684+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:55:54.999411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.554062+0000","last_change":"2024-04-26T23:43:59.870821+0000","last_active":"2024-04-26T23:45:00.554062+0000","last_peered":"2024-04-26T23:45:00.554062+0000","last_clean":"2024-04-26T23:45:00.554062+0000","last_became_active":"2024-04-26T23:43:59.870240+0000","last_became_peered":"2024-04-26T23:43:59.870240+0000","last_unstale":"2024-04-26T23:45:00.554062+0000","last_undegraded":"2024-04-26T23:45:00.554062+0000","last_fullsized":"2024-04-26T23:45:00.554062+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:33:00.284232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,6],"acting":[9,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.548674+0000","last_change":"2024-04-26T23:43:07.535682+0000","last_active":"2024-04-26T23:45:00.548674+0000","last_peered":"2024-04-26T23:45:00.548674+0000","last_clean":"2024-04-26T23:45:00.548674+0000","last_became_active":"2024-04-26T23:43:07.535221+0000","last_became_peered":"2024-04-26T23:43:07.535221+0000","last_unstale":"2024-04-26T23:45:00.548674+0000","last_undegraded":"2024-04-26T23:45:00.548674+0000","last_fullsized":"2024-04-26T23:45:00.548674+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:33:54.105329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.538382+0000","last_change":"2024-04-26T23:43:02.443541+0000","last_active":"2024-04-26T23:45:00.538382+0000","last_peered":"2024-04-26T23:45:00.538382+0000","last_clean":"2024-04-26T23:45:00.538382+0000","last_became_active":"2024-04-26T23:43:02.442948+0000","last_became_peered":"2024-04-26T23:43:02.442948+0000","last_unstale":"2024-04-26T23:45:00.538382+0000","last_undegraded":"2024-04-26T23:45:00.538382+0000","last_fullsized":"2024-04-26T23:45:00.538382+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:58:44.595575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.555477+0000","last_change":"2024-04-26T23:43:03.311029+0000","last_active":"2024-04-26T23:45:00.555477+0000","last_peered":"2024-04-26T23:45:00.555477+0000","last_clean":"2024-04-26T23:45:00.555477+0000","last_became_active":"2024-04-26T23:43:03.310722+0000","last_became_peered":"2024-04-26T23:43:03.310722+0000","last_unstale":"2024-04-26T23:45:00.555477+0000","last_undegraded":"2024-04-26T23:45:00.555477+0000","last_fullsized":"2024-04-26T23:45:00.555477+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:07:54.449492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.543870+0000","last_change":"2024-04-26T23:43:07.537391+0000","last_active":"2024-04-26T23:45:00.543870+0000","last_peered":"2024-04-26T23:45:00.543870+0000","last_clean":"2024-04-26T23:45:00.543870+0000","last_became_active":"2024-04-26T23:43:07.536216+0000","last_became_peered":"2024-04-26T23:43:07.536216+0000","last_unstale":"2024-04-26T23:45:00.543870+0000","last_undegraded":"2024-04-26T23:45:00.543870+0000","last_fullsized":"2024-04-26T23:45:00.543870+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:45:55.326283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":132,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:07.350998+0000","last_change":"2024-04-26T23:43:02.451929+0000","last_active":"2024-04-26T23:45:07.350998+0000","last_peered":"2024-04-26T23:45:07.350998+0000","last_clean":"2024-04-26T23:45:07.350998+0000","last_became_active":"2024-04-26T23:43:02.451700+0000","last_became_peered":"2024-04-26T23:43:02.451700+0000","last_unstale":"2024-04-26T23:45:07.350998+0000","last_undegraded":"2024-04-26T23:45:07.350998+0000","last_fullsized":"2024-04-26T23:45:07.350998+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:21:31.279681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":99,"num_read_kb":99,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.543772+0000","last_change":"2024-04-26T23:43:03.295462+0000","last_active":"2024-04-26T23:45:00.543772+0000","last_peered":"2024-04-26T23:45:00.543772+0000","last_clean":"2024-04-26T23:45:00.543772+0000","last_became_active":"2024-04-26T23:43:03.295044+0000","last_became_peered":"2024-04-26T23:43:03.295044+0000","last_unstale":"2024-04-26T23:45:00.543772+0000","last_undegraded":"2024-04-26T23:45:00.543772+0000","last_fullsized":"2024-04-26T23:45:00.543772+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:56:19.445166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.554405+0000","last_change":"2024-04-26T23:43:07.539397+0000","last_active":"2024-04-26T23:45:00.554405+0000","last_peered":"2024-04-26T23:45:00.554405+0000","last_clean":"2024-04-26T23:45:00.554405+0000","last_became_active":"2024-04-26T23:43:07.533081+0000","last_became_peered":"2024-04-26T23:43:07.533081+0000","last_unstale":"2024-04-26T23:45:00.554405+0000","last_undegraded":"2024-04-26T23:45:00.554405+0000","last_fullsized":"2024-04-26T23:45:00.554405+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:24:39.693721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544804+0000","last_change":"2024-04-26T23:43:02.439919+0000","last_active":"2024-04-26T23:45:00.544804+0000","last_peered":"2024-04-26T23:45:00.544804+0000","last_clean":"2024-04-26T23:45:00.544804+0000","last_became_active":"2024-04-26T23:43:02.439561+0000","last_became_peered":"2024-04-26T23:43:02.439561+0000","last_unstale":"2024-04-26T23:45:00.544804+0000","last_undegraded":"2024-04-26T23:45:00.544804+0000","last_fullsized":"2024-04-26T23:45:00.544804+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:30:22.905464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.549199+0000","last_change":"2024-04-26T23:43:03.294088+0000","last_active":"2024-04-26T23:45:00.549199+0000","last_peered":"2024-04-26T23:45:00.549199+0000","last_clean":"2024-04-26T23:45:00.549199+0000","last_became_active":"2024-04-26T23:43:03.293726+0000","last_became_peered":"2024-04-26T23:43:03.293726+0000","last_unstale":"2024-04-26T23:45:00.549199+0000","last_undegraded":"2024-04-26T23:45:00.549199+0000","last_fullsized":"2024-04-26T23:45:00.549199+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:19:42.575193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.541076+0000","last_change":"2024-04-26T23:43:07.543056+0000","last_active":"2024-04-26T23:45:00.541076+0000","last_peered":"2024-04-26T23:45:00.541076+0000","last_clean":"2024-04-26T23:45:00.541076+0000","last_became_active":"2024-04-26T23:43:07.542324+0000","last_became_peered":"2024-04-26T23:43:07.542324+0000","last_unstale":"2024-04-26T23:45:00.541076+0000","last_undegraded":"2024-04-26T23:45:00.541076+0000","last_fullsized":"2024-04-26T23:45:00.541076+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:23:46.415932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.538595+0000","last_change":"2024-04-26T23:43:02.454135+0000","last_active":"2024-04-26T23:45:00.538595+0000","last_peered":"2024-04-26T23:45:00.538595+0000","last_clean":"2024-04-26T23:45:00.538595+0000","last_became_active":"2024-04-26T23:43:02.453798+0000","last_became_peered":"2024-04-26T23:43:02.453798+0000","last_unstale":"2024-04-26T23:45:00.538595+0000","last_undegraded":"2024-04-26T23:45:00.538595+0000","last_fullsized":"2024-04-26T23:45:00.538595+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:16:25.286569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.543474+0000","last_change":"2024-04-26T23:43:03.293520+0000","last_active":"2024-04-26T23:45:00.543474+0000","last_peered":"2024-04-26T23:45:00.543474+0000","last_clean":"2024-04-26T23:45:00.543474+0000","last_became_active":"2024-04-26T23:43:03.292537+0000","last_became_peered":"2024-04-26T23:43:03.292537+0000","last_unstale":"2024-04-26T23:45:00.543474+0000","last_undegraded":"2024-04-26T23:45:00.543474+0000","last_fullsized":"2024-04-26T23:45:00.543474+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:19:28.808217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.555126+0000","last_change":"2024-04-26T23:43:07.528116+0000","last_active":"2024-04-26T23:45:00.555126+0000","last_peered":"2024-04-26T23:45:00.555126+0000","last_clean":"2024-04-26T23:45:00.555126+0000","last_became_active":"2024-04-26T23:43:07.527766+0000","last_became_peered":"2024-04-26T23:43:07.527766+0000","last_unstale":"2024-04-26T23:45:00.555126+0000","last_undegraded":"2024-04-26T23:45:00.555126+0000","last_fullsized":"2024-04-26T23:45:00.555126+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:39:23.949964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'33","reported_seq":268,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:08.364482+0000","last_change":"2024-04-26T23:43:02.437454+0000","last_active":"2024-04-26T23:45:08.364482+0000","last_peered":"2024-04-26T23:45:08.364482+0000","last_clean":"2024-04-26T23:45:08.364482+0000","last_became_active":"2024-04-26T23:43:02.437142+0000","last_became_peered":"2024-04-26T23:43:02.437142+0000","last_unstale":"2024-04-26T23:45:08.364482+0000","last_undegraded":"2024-04-26T23:45:08.364482+0000","last_fullsized":"2024-04-26T23:45:08.364482+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:06:37.751066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54826,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":205,"num_read_kb":155,"num_write":34,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544594+0000","last_change":"2024-04-26T23:43:03.287831+0000","last_active":"2024-04-26T23:45:00.544594+0000","last_peered":"2024-04-26T23:45:00.544594+0000","last_clean":"2024-04-26T23:45:00.544594+0000","last_became_active":"2024-04-26T23:43:03.287479+0000","last_became_peered":"2024-04-26T23:43:03.287479+0000","last_unstale":"2024-04-26T23:45:00.544594+0000","last_undegraded":"2024-04-26T23:45:00.544594+0000","last_fullsized":"2024-04-26T23:45:00.544594+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:39:41.726893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.539031+0000","last_change":"2024-04-26T23:43:07.543985+0000","last_active":"2024-04-26T23:45:00.539031+0000","last_peered":"2024-04-26T23:45:00.539031+0000","last_clean":"2024-04-26T23:45:00.539031+0000","last_became_active":"2024-04-26T23:43:07.543327+0000","last_became_peered":"2024-04-26T23:43:07.543327+0000","last_unstale":"2024-04-26T23:45:00.539031+0000","last_undegraded":"2024-04-26T23:45:00.539031+0000","last_fullsized":"2024-04-26T23:45:00.539031+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:24:10.577822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.543860+0000","last_change":"2024-04-26T23:43:02.447627+0000","last_active":"2024-04-26T23:45:00.543860+0000","last_peered":"2024-04-26T23:45:00.543860+0000","last_clean":"2024-04-26T23:45:00.543860+0000","last_became_active":"2024-04-26T23:43:02.447325+0000","last_became_peered":"2024-04-26T23:43:02.447325+0000","last_unstale":"2024-04-26T23:45:00.543860+0000","last_undegraded":"2024-04-26T23:45:00.543860+0000","last_fullsized":"2024-04-26T23:45:00.543860+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:19:03.757202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.547818+0000","last_change":"2024-04-26T23:43:03.313388+0000","last_active":"2024-04-26T23:45:00.547818+0000","last_peered":"2024-04-26T23:45:00.547818+0000","last_clean":"2024-04-26T23:45:00.547818+0000","last_became_active":"2024-04-26T23:43:03.312899+0000","last_became_peered":"2024-04-26T23:43:03.312899+0000","last_unstale":"2024-04-26T23:45:00.547818+0000","last_undegraded":"2024-04-26T23:45:00.547818+0000","last_fullsized":"2024-04-26T23:45:00.547818+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:58:37.126485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.541551+0000","last_change":"2024-04-26T23:43:07.536706+0000","last_active":"2024-04-26T23:45:00.541551+0000","last_peered":"2024-04-26T23:45:00.541551+0000","last_clean":"2024-04-26T23:45:00.541551+0000","last_became_active":"2024-04-26T23:43:07.536316+0000","last_became_peered":"2024-04-26T23:43:07.536316+0000","last_unstale":"2024-04-26T23:45:00.541551+0000","last_undegraded":"2024-04-26T23:45:00.541551+0000","last_fullsized":"2024-04-26T23:45:00.541551+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:36:46.397725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.538435+0000","last_change":"2024-04-26T23:43:02.448647+0000","last_active":"2024-04-26T23:45:00.538435+0000","last_peered":"2024-04-26T23:45:00.538435+0000","last_clean":"2024-04-26T23:45:00.538435+0000","last_became_active":"2024-04-26T23:43:02.448392+0000","last_became_peered":"2024-04-26T23:43:02.448392+0000","last_unstale":"2024-04-26T23:45:00.538435+0000","last_undegraded":"2024-04-26T23:45:00.538435+0000","last_fullsized":"2024-04-26T23:45:00.538435+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:19:01.367361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544605+0000","last_change":"2024-04-26T23:43:03.302980+0000","last_active":"2024-04-26T23:45:00.544605+0000","last_peered":"2024-04-26T23:45:00.544605+0000","last_clean":"2024-04-26T23:45:00.544605+0000","last_became_active":"2024-04-26T23:43:03.302589+0000","last_became_peered":"2024-04-26T23:43:03.302589+0000","last_unstale":"2024-04-26T23:45:00.544605+0000","last_undegraded":"2024-04-26T23:45:00.544605+0000","last_fullsized":"2024-04-26T23:45:00.544605+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:57:18.641063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.556728+0000","last_change":"2024-04-26T23:43:07.547842+0000","last_active":"2024-04-26T23:45:00.556728+0000","last_peered":"2024-04-26T23:45:00.556728+0000","last_clean":"2024-04-26T23:45:00.556728+0000","last_became_active":"2024-04-26T23:43:07.541774+0000","last_became_peered":"2024-04-26T23:43:07.541774+0000","last_unstale":"2024-04-26T23:45:00.556728+0000","last_undegraded":"2024-04-26T23:45:00.556728+0000","last_fullsized":"2024-04-26T23:45:00.556728+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:25:49.999465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'4","reported_seq":137,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:07.407194+0000","last_change":"2024-04-26T23:43:02.437210+0000","last_active":"2024-04-26T23:45:07.407194+0000","last_peered":"2024-04-26T23:45:07.407194+0000","last_clean":"2024-04-26T23:45:07.407194+0000","last_became_active":"2024-04-26T23:43:02.436886+0000","last_became_peered":"2024-04-26T23:43:02.436886+0000","last_unstale":"2024-04-26T23:45:07.407194+0000","last_undegraded":"2024-04-26T23:45:07.407194+0000","last_fullsized":"2024-04-26T23:45:07.407194+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:07:03.663757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":103,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544319+0000","last_change":"2024-04-26T23:43:59.868608+0000","last_active":"2024-04-26T23:45:00.544319+0000","last_peered":"2024-04-26T23:45:00.544319+0000","last_clean":"2024-04-26T23:45:00.544319+0000","last_became_active":"2024-04-26T23:43:59.868231+0000","last_became_peered":"2024-04-26T23:43:59.868231+0000","last_unstale":"2024-04-26T23:45:00.544319+0000","last_undegraded":"2024-04-26T23:45:00.544319+0000","last_fullsized":"2024-04-26T23:45:00.544319+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:28:42.932733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.559597+0000","last_change":"2024-04-26T23:45:00.559597+0000","last_active":"2024-04-26T23:45:00.559597+0000","last_peered":"2024-04-26T23:45:00.559597+0000","last_clean":"2024-04-26T23:45:00.559597+0000","last_became_active":"2024-04-26T23:45:00.558473+0000","last_became_peered":"2024-04-26T23:45:00.558473+0000","last_unstale":"2024-04-26T23:45:00.559597+0000","last_undegraded":"2024-04-26T23:45:00.559597+0000","last_fullsized":"2024-04-26T23:45:00.559597+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:07:29.856136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":70,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.543200+0000","last_change":"2024-04-26T23:43:59.589239+0000","last_active":"2024-04-26T23:45:00.543200+0000","last_peered":"2024-04-26T23:45:00.543200+0000","last_clean":"2024-04-26T23:45:00.543200+0000","last_became_active":"2024-04-26T23:43:59.576239+0000","last_became_peered":"2024-04-26T23:43:59.576239+0000","last_unstale":"2024-04-26T23:45:00.543200+0000","last_undegraded":"2024-04-26T23:45:00.543200+0000","last_fullsized":"2024-04-26T23:45:00.543200+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:28:53.326961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.538251+0000","last_change":"2024-04-26T23:43:03.290567+0000","last_active":"2024-04-26T23:45:00.538251+0000","last_peered":"2024-04-26T23:45:00.538251+0000","last_clean":"2024-04-26T23:45:00.538251+0000","last_became_active":"2024-04-26T23:43:03.287807+0000","last_became_peered":"2024-04-26T23:43:03.287807+0000","last_unstale":"2024-04-26T23:45:00.538251+0000","last_undegraded":"2024-04-26T23:45:00.538251+0000","last_fullsized":"2024-04-26T23:45:00.538251+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:25:24.393978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.538200+0000","last_change":"2024-04-26T23:43:07.535402+0000","last_active":"2024-04-26T23:45:00.538200+0000","last_peered":"2024-04-26T23:45:00.538200+0000","last_clean":"2024-04-26T23:45:00.538200+0000","last_became_active":"2024-04-26T23:43:07.534870+0000","last_became_peered":"2024-04-26T23:43:07.534870+0000","last_unstale":"2024-04-26T23:45:00.538200+0000","last_undegraded":"2024-04-26T23:45:00.538200+0000","last_fullsized":"2024-04-26T23:45:00.538200+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:26:37.703831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.549038+0000","last_change":"2024-04-26T23:45:00.549038+0000","last_active":"2024-04-26T23:45:00.549038+0000","last_peered":"2024-04-26T23:45:00.549038+0000","last_clean":"2024-04-26T23:45:00.549038+0000","last_became_active":"2024-04-26T23:45:00.543141+0000","last_became_peered":"2024-04-26T23:45:00.543141+0000","last_unstale":"2024-04-26T23:45:00.549038+0000","last_undegraded":"2024-04-26T23:45:00.549038+0000","last_fullsized":"2024-04-26T23:45:00.549038+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:03:27.079264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.541735+0000","last_change":"2024-04-26T23:43:59.576561+0000","last_active":"2024-04-26T23:45:00.541735+0000","last_peered":"2024-04-26T23:45:00.541735+0000","last_clean":"2024-04-26T23:45:00.541735+0000","last_became_active":"2024-04-26T23:43:59.574898+0000","last_became_peered":"2024-04-26T23:43:59.574898+0000","last_unstale":"2024-04-26T23:45:00.541735+0000","last_undegraded":"2024-04-26T23:45:00.541735+0000","last_fullsized":"2024-04-26T23:45:00.541735+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:57:04.569103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.549787+0000","last_change":"2024-04-26T23:45:00.549787+0000","last_active":"2024-04-26T23:45:00.549787+0000","last_peered":"2024-04-26T23:45:00.549787+0000","last_clean":"2024-04-26T23:45:00.549787+0000","last_became_active":"2024-04-26T23:45:00.549316+0000","last_became_peered":"2024-04-26T23:45:00.549316+0000","last_unstale":"2024-04-26T23:45:00.549787+0000","last_undegraded":"2024-04-26T23:45:00.549787+0000","last_fullsized":"2024-04-26T23:45:00.549787+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:21:36.188388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.555587+0000","last_change":"2024-04-26T23:43:02.457971+0000","last_active":"2024-04-26T23:45:00.555587+0000","last_peered":"2024-04-26T23:45:00.555587+0000","last_clean":"2024-04-26T23:45:00.555587+0000","last_became_active":"2024-04-26T23:43:02.457627+0000","last_became_peered":"2024-04-26T23:43:02.457627+0000","last_unstale":"2024-04-26T23:45:00.555587+0000","last_undegraded":"2024-04-26T23:45:00.555587+0000","last_fullsized":"2024-04-26T23:45:00.555587+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:25:51.749774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.548424+0000","last_change":"2024-04-26T23:43:03.305569+0000","last_active":"2024-04-26T23:45:00.548424+0000","last_peered":"2024-04-26T23:45:00.548424+0000","last_clean":"2024-04-26T23:45:00.548424+0000","last_became_active":"2024-04-26T23:43:03.305211+0000","last_became_peered":"2024-04-26T23:43:03.305211+0000","last_unstale":"2024-04-26T23:45:00.548424+0000","last_undegraded":"2024-04-26T23:45:00.548424+0000","last_fullsized":"2024-04-26T23:45:00.548424+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:54:55.847680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.553421+0000","last_change":"2024-04-26T23:43:07.535408+0000","last_active":"2024-04-26T23:45:00.553421+0000","last_peered":"2024-04-26T23:45:00.553421+0000","last_clean":"2024-04-26T23:45:00.553421+0000","last_became_active":"2024-04-26T23:43:07.534957+0000","last_became_peered":"2024-04-26T23:43:07.534957+0000","last_unstale":"2024-04-26T23:45:00.553421+0000","last_undegraded":"2024-04-26T23:45:00.553421+0000","last_fullsized":"2024-04-26T23:45:00.553421+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:35:31.103532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544403+0000","last_change":"2024-04-26T23:43:02.453636+0000","last_active":"2024-04-26T23:45:00.544403+0000","last_peered":"2024-04-26T23:45:00.544403+0000","last_clean":"2024-04-26T23:45:00.544403+0000","last_became_active":"2024-04-26T23:43:02.453310+0000","last_became_peered":"2024-04-26T23:43:02.453310+0000","last_unstale":"2024-04-26T23:45:00.544403+0000","last_undegraded":"2024-04-26T23:45:00.544403+0000","last_fullsized":"2024-04-26T23:45:00.544403+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:42:29.295303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.548563+0000","last_change":"2024-04-26T23:43:03.283184+0000","last_active":"2024-04-26T23:45:00.548563+0000","last_peered":"2024-04-26T23:45:00.548563+0000","last_clean":"2024-04-26T23:45:00.548563+0000","last_became_active":"2024-04-26T23:43:03.282780+0000","last_became_peered":"2024-04-26T23:43:03.282780+0000","last_unstale":"2024-04-26T23:45:00.548563+0000","last_undegraded":"2024-04-26T23:45:00.548563+0000","last_fullsized":"2024-04-26T23:45:00.548563+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:00:13.907101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.555563+0000","last_change":"2024-04-26T23:43:07.859107+0000","last_active":"2024-04-26T23:45:00.555563+0000","last_peered":"2024-04-26T23:45:00.555563+0000","last_clean":"2024-04-26T23:45:00.555563+0000","last_became_active":"2024-04-26T23:43:07.856492+0000","last_became_peered":"2024-04-26T23:43:07.856492+0000","last_unstale":"2024-04-26T23:45:00.555563+0000","last_undegraded":"2024-04-26T23:45:00.555563+0000","last_fullsized":"2024-04-26T23:45:00.555563+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:13:36.570398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.538547+0000","last_change":"2024-04-26T23:43:03.289078+0000","last_active":"2024-04-26T23:45:00.538547+0000","last_peered":"2024-04-26T23:45:00.538547+0000","last_clean":"2024-04-26T23:45:00.538547+0000","last_became_active":"2024-04-26T23:43:03.288729+0000","last_became_peered":"2024-04-26T23:43:03.288729+0000","last_unstale":"2024-04-26T23:45:00.538547+0000","last_undegraded":"2024-04-26T23:45:00.538547+0000","last_fullsized":"2024-04-26T23:45:00.538547+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:32:36.346472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":135,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:08.408711+0000","last_change":"2024-04-26T23:43:02.463355+0000","last_active":"2024-04-26T23:45:08.408711+0000","last_peered":"2024-04-26T23:45:08.408711+0000","last_clean":"2024-04-26T23:45:08.408711+0000","last_became_active":"2024-04-26T23:43:02.455832+0000","last_became_peered":"2024-04-26T23:43:02.455832+0000","last_unstale":"2024-04-26T23:45:08.408711+0000","last_undegraded":"2024-04-26T23:45:08.408711+0000","last_fullsized":"2024-04-26T23:45:08.408711+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:33:33.837999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":99,"num_read_kb":99,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544756+0000","last_change":"2024-04-26T23:43:07.540777+0000","last_active":"2024-04-26T23:45:00.544756+0000","last_peered":"2024-04-26T23:45:00.544756+0000","last_clean":"2024-04-26T23:45:00.544756+0000","last_became_active":"2024-04-26T23:43:07.540389+0000","last_became_peered":"2024-04-26T23:43:07.540389+0000","last_unstale":"2024-04-26T23:45:00.544756+0000","last_undegraded":"2024-04-26T23:45:00.544756+0000","last_fullsized":"2024-04-26T23:45:00.544756+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:57:28.674051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544824+0000","last_change":"2024-04-26T23:43:03.308391+0000","last_active":"2024-04-26T23:45:00.544824+0000","last_peered":"2024-04-26T23:45:00.544824+0000","last_clean":"2024-04-26T23:45:00.544824+0000","last_became_active":"2024-04-26T23:43:03.306762+0000","last_became_peered":"2024-04-26T23:43:03.306762+0000","last_unstale":"2024-04-26T23:45:00.544824+0000","last_undegraded":"2024-04-26T23:45:00.544824+0000","last_fullsized":"2024-04-26T23:45:00.544824+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:34:01.083036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.541054+0000","last_change":"2024-04-26T23:43:02.448037+0000","last_active":"2024-04-26T23:45:00.541054+0000","last_peered":"2024-04-26T23:45:00.541054+0000","last_clean":"2024-04-26T23:45:00.541054+0000","last_became_active":"2024-04-26T23:43:02.447706+0000","last_became_peered":"2024-04-26T23:43:02.447706+0000","last_unstale":"2024-04-26T23:45:00.541054+0000","last_undegraded":"2024-04-26T23:45:00.541054+0000","last_fullsized":"2024-04-26T23:45:00.541054+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:43:56.295191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":352,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544151+0000","last_change":"2024-04-26T23:41:42.266448+0000","last_active":"2024-04-26T23:45:00.544151+0000","last_peered":"2024-04-26T23:45:00.544151+0000","last_clean":"2024-04-26T23:45:00.544151+0000","last_became_active":"2024-04-26T23:41:42.265898+0000","last_became_peered":"2024-04-26T23:41:42.265898+0000","last_unstale":"2024-04-26T23:45:00.544151+0000","last_undegraded":"2024-04-26T23:45:00.544151+0000","last_fullsized":"2024-04-26T23:45:00.544151+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:37:44.460409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:37:44.460409+0000","last_clean_scrub_stamp":"2024-04-26T23:37:44.460409+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:40:48.177191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.557115+0000","last_change":"2024-04-26T23:45:00.557115+0000","last_active":"2024-04-26T23:45:00.557115+0000","last_peered":"2024-04-26T23:45:00.557115+0000","last_clean":"2024-04-26T23:45:00.557115+0000","last_became_active":"2024-04-26T23:45:00.556698+0000","last_became_peered":"2024-04-26T23:45:00.556698+0000","last_unstale":"2024-04-26T23:45:00.557115+0000","last_undegraded":"2024-04-26T23:45:00.557115+0000","last_fullsized":"2024-04-26T23:45:00.557115+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:23:11.253779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.548195+0000","last_change":"2024-04-26T23:43:03.301989+0000","last_active":"2024-04-26T23:45:00.548195+0000","last_peered":"2024-04-26T23:45:00.548195+0000","last_clean":"2024-04-26T23:45:00.548195+0000","last_became_active":"2024-04-26T23:43:03.301179+0000","last_became_peered":"2024-04-26T23:43:03.301179+0000","last_unstale":"2024-04-26T23:45:00.548195+0000","last_undegraded":"2024-04-26T23:45:00.548195+0000","last_fullsized":"2024-04-26T23:45:00.548195+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:29:33.012009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544372+0000","last_change":"2024-04-26T23:43:02.439648+0000","last_active":"2024-04-26T23:45:00.544372+0000","last_peered":"2024-04-26T23:45:00.544372+0000","last_clean":"2024-04-26T23:45:00.544372+0000","last_became_active":"2024-04-26T23:43:02.439340+0000","last_became_peered":"2024-04-26T23:43:02.439340+0000","last_unstale":"2024-04-26T23:45:00.544372+0000","last_undegraded":"2024-04-26T23:45:00.544372+0000","last_fullsized":"2024-04-26T23:45:00.544372+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:10:01.096657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.541291+0000","last_change":"2024-04-26T23:43:07.534317+0000","last_active":"2024-04-26T23:45:00.541291+0000","last_peered":"2024-04-26T23:45:00.541291+0000","last_clean":"2024-04-26T23:45:00.541291+0000","last_became_active":"2024-04-26T23:43:07.533808+0000","last_became_peered":"2024-04-26T23:43:07.533808+0000","last_unstale":"2024-04-26T23:45:00.541291+0000","last_undegraded":"2024-04-26T23:45:00.541291+0000","last_fullsized":"2024-04-26T23:45:00.541291+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:54:29.952089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.545460+0000","last_change":"2024-04-26T23:43:02.457501+0000","last_active":"2024-04-26T23:45:00.545460+0000","last_peered":"2024-04-26T23:45:00.545460+0000","last_clean":"2024-04-26T23:45:00.545460+0000","last_became_active":"2024-04-26T23:43:02.457024+0000","last_became_peered":"2024-04-26T23:43:02.457024+0000","last_unstale":"2024-04-26T23:45:00.545460+0000","last_undegraded":"2024-04-26T23:45:00.545460+0000","last_fullsized":"2024-04-26T23:45:00.545460+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:21:58.542186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.553642+0000","last_change":"2024-04-26T23:43:03.303434+0000","last_active":"2024-04-26T23:45:00.553642+0000","last_peered":"2024-04-26T23:45:00.553642+0000","last_clean":"2024-04-26T23:45:00.553642+0000","last_became_active":"2024-04-26T23:43:03.302455+0000","last_became_peered":"2024-04-26T23:43:03.302455+0000","last_unstale":"2024-04-26T23:45:00.553642+0000","last_undegraded":"2024-04-26T23:45:00.553642+0000","last_fullsized":"2024-04-26T23:45:00.553642+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:57:05.530635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.542546+0000","last_change":"2024-04-26T23:43:07.856932+0000","last_active":"2024-04-26T23:45:00.542546+0000","last_peered":"2024-04-26T23:45:00.542546+0000","last_clean":"2024-04-26T23:45:00.542546+0000","last_became_active":"2024-04-26T23:43:07.855699+0000","last_became_peered":"2024-04-26T23:43:07.855699+0000","last_unstale":"2024-04-26T23:45:00.542546+0000","last_undegraded":"2024-04-26T23:45:00.542546+0000","last_fullsized":"2024-04-26T23:45:00.542546+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:20:28.994301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.557150+0000","last_change":"2024-04-26T23:43:02.449852+0000","last_active":"2024-04-26T23:45:00.557150+0000","last_peered":"2024-04-26T23:45:00.557150+0000","last_clean":"2024-04-26T23:45:00.557150+0000","last_became_active":"2024-04-26T23:43:02.449422+0000","last_became_peered":"2024-04-26T23:43:02.449422+0000","last_unstale":"2024-04-26T23:45:00.557150+0000","last_undegraded":"2024-04-26T23:45:00.557150+0000","last_fullsized":"2024-04-26T23:45:00.557150+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:07:25.453872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.551659+0000","last_change":"2024-04-26T23:43:03.304733+0000","last_active":"2024-04-26T23:45:00.551659+0000","last_peered":"2024-04-26T23:45:00.551659+0000","last_clean":"2024-04-26T23:45:00.551659+0000","last_became_active":"2024-04-26T23:43:03.304383+0000","last_became_peered":"2024-04-26T23:43:03.304383+0000","last_unstale":"2024-04-26T23:45:00.551659+0000","last_undegraded":"2024-04-26T23:45:00.551659+0000","last_fullsized":"2024-04-26T23:45:00.551659+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:49:28.679255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.543627+0000","last_change":"2024-04-26T23:43:07.525310+0000","last_active":"2024-04-26T23:45:00.543627+0000","last_peered":"2024-04-26T23:45:00.543627+0000","last_clean":"2024-04-26T23:45:00.543627+0000","last_became_active":"2024-04-26T23:43:07.524502+0000","last_became_peered":"2024-04-26T23:43:07.524502+0000","last_unstale":"2024-04-26T23:45:00.543627+0000","last_undegraded":"2024-04-26T23:45:00.543627+0000","last_fullsized":"2024-04-26T23:45:00.543627+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:07:45.927837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.541373+0000","last_change":"2024-04-26T23:43:02.446238+0000","last_active":"2024-04-26T23:45:00.541373+0000","last_peered":"2024-04-26T23:45:00.541373+0000","last_clean":"2024-04-26T23:45:00.541373+0000","last_became_active":"2024-04-26T23:43:02.445959+0000","last_became_peered":"2024-04-26T23:43:02.445959+0000","last_unstale":"2024-04-26T23:45:00.541373+0000","last_undegraded":"2024-04-26T23:45:00.541373+0000","last_fullsized":"2024-04-26T23:45:00.541373+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:25:10.007481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.548843+0000","last_change":"2024-04-26T23:43:03.306035+0000","last_active":"2024-04-26T23:45:00.548843+0000","last_peered":"2024-04-26T23:45:00.548843+0000","last_clean":"2024-04-26T23:45:00.548843+0000","last_became_active":"2024-04-26T23:43:03.305604+0000","last_became_peered":"2024-04-26T23:43:03.305604+0000","last_unstale":"2024-04-26T23:45:00.548843+0000","last_undegraded":"2024-04-26T23:45:00.548843+0000","last_fullsized":"2024-04-26T23:45:00.548843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:02:38.567754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.557767+0000","last_change":"2024-04-26T23:45:00.557767+0000","last_active":"2024-04-26T23:45:00.557767+0000","last_peered":"2024-04-26T23:45:00.557767+0000","last_clean":"2024-04-26T23:45:00.557767+0000","last_became_active":"2024-04-26T23:45:00.557207+0000","last_became_peered":"2024-04-26T23:45:00.557207+0000","last_unstale":"2024-04-26T23:45:00.557767+0000","last_undegraded":"2024-04-26T23:45:00.557767+0000","last_fullsized":"2024-04-26T23:45:00.557767+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:18:06.396984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,10,9],"acting":[4,8,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.542963+0000","last_change":"2024-04-26T23:43:02.443464+0000","last_active":"2024-04-26T23:45:00.542963+0000","last_peered":"2024-04-26T23:45:00.542963+0000","last_clean":"2024-04-26T23:45:00.542963+0000","last_became_active":"2024-04-26T23:43:02.443170+0000","last_became_peered":"2024-04-26T23:43:02.443170+0000","last_unstale":"2024-04-26T23:45:00.542963+0000","last_undegraded":"2024-04-26T23:45:00.542963+0000","last_fullsized":"2024-04-26T23:45:00.542963+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:45:51.431082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544694+0000","last_change":"2024-04-26T23:43:03.282714+0000","last_active":"2024-04-26T23:45:00.544694+0000","last_peered":"2024-04-26T23:45:00.544694+0000","last_clean":"2024-04-26T23:45:00.544694+0000","last_became_active":"2024-04-26T23:43:03.282353+0000","last_became_peered":"2024-04-26T23:43:03.282353+0000","last_unstale":"2024-04-26T23:45:00.544694+0000","last_undegraded":"2024-04-26T23:45:00.544694+0000","last_fullsized":"2024-04-26T23:45:00.544694+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:18:24.082946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.556962+0000","last_change":"2024-04-26T23:43:07.543301+0000","last_active":"2024-04-26T23:45:00.556962+0000","last_peered":"2024-04-26T23:45:00.556962+0000","last_clean":"2024-04-26T23:45:00.556962+0000","last_became_active":"2024-04-26T23:43:07.540434+0000","last_became_peered":"2024-04-26T23:43:07.540434+0000","last_unstale":"2024-04-26T23:45:00.556962+0000","last_undegraded":"2024-04-26T23:45:00.556962+0000","last_fullsized":"2024-04-26T23:45:00.556962+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:15:18.542595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.543478+0000","last_change":"2024-04-26T23:43:02.451391+0000","last_active":"2024-04-26T23:45:00.543478+0000","last_peered":"2024-04-26T23:45:00.543478+0000","last_clean":"2024-04-26T23:45:00.543478+0000","last_became_active":"2024-04-26T23:43:02.450735+0000","last_became_peered":"2024-04-26T23:43:02.450735+0000","last_unstale":"2024-04-26T23:45:00.543478+0000","last_undegraded":"2024-04-26T23:45:00.543478+0000","last_fullsized":"2024-04-26T23:45:00.543478+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:57:48.658671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.541149+0000","last_change":"2024-04-26T23:43:03.301858+0000","last_active":"2024-04-26T23:45:00.541149+0000","last_peered":"2024-04-26T23:45:00.541149+0000","last_clean":"2024-04-26T23:45:00.541149+0000","last_became_active":"2024-04-26T23:43:03.301474+0000","last_became_peered":"2024-04-26T23:43:03.301474+0000","last_unstale":"2024-04-26T23:45:00.541149+0000","last_undegraded":"2024-04-26T23:45:00.541149+0000","last_fullsized":"2024-04-26T23:45:00.541149+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:24:56.386127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.541095+0000","last_change":"2024-04-26T23:43:07.544512+0000","last_active":"2024-04-26T23:45:00.541095+0000","last_peered":"2024-04-26T23:45:00.541095+0000","last_clean":"2024-04-26T23:45:00.541095+0000","last_became_active":"2024-04-26T23:43:07.544029+0000","last_became_peered":"2024-04-26T23:43:07.544029+0000","last_unstale":"2024-04-26T23:45:00.541095+0000","last_undegraded":"2024-04-26T23:45:00.541095+0000","last_fullsized":"2024-04-26T23:45:00.541095+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:38:08.149123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":19,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.541148+0000","last_change":"2024-04-26T23:43:59.579721+0000","last_active":"2024-04-26T23:45:00.541148+0000","last_peered":"2024-04-26T23:45:00.541148+0000","last_clean":"2024-04-26T23:45:00.541148+0000","last_became_active":"2024-04-26T23:43:59.576675+0000","last_became_peered":"2024-04-26T23:43:59.576675+0000","last_unstale":"2024-04-26T23:45:00.541148+0000","last_undegraded":"2024-04-26T23:45:00.541148+0000","last_fullsized":"2024-04-26T23:45:00.541148+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:59:30.124137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.549817+0000","last_change":"2024-04-26T23:43:03.307895+0000","last_active":"2024-04-26T23:45:00.549817+0000","last_peered":"2024-04-26T23:45:00.549817+0000","last_clean":"2024-04-26T23:45:00.549817+0000","last_became_active":"2024-04-26T23:43:03.307478+0000","last_became_peered":"2024-04-26T23:43:03.307478+0000","last_unstale":"2024-04-26T23:45:00.549817+0000","last_undegraded":"2024-04-26T23:45:00.549817+0000","last_fullsized":"2024-04-26T23:45:00.549817+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:57:14.356234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.541058+0000","last_change":"2024-04-26T23:43:07.855803+0000","last_active":"2024-04-26T23:45:00.541058+0000","last_peered":"2024-04-26T23:45:00.541058+0000","last_clean":"2024-04-26T23:45:00.541058+0000","last_became_active":"2024-04-26T23:43:07.855200+0000","last_became_peered":"2024-04-26T23:43:07.855200+0000","last_unstale":"2024-04-26T23:45:00.541058+0000","last_undegraded":"2024-04-26T23:45:00.541058+0000","last_fullsized":"2024-04-26T23:45:00.541058+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:02:46.223330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.543821+0000","last_change":"2024-04-26T23:43:02.437513+0000","last_active":"2024-04-26T23:45:00.543821+0000","last_peered":"2024-04-26T23:45:00.543821+0000","last_clean":"2024-04-26T23:45:00.543821+0000","last_became_active":"2024-04-26T23:43:02.437153+0000","last_became_peered":"2024-04-26T23:43:02.437153+0000","last_unstale":"2024-04-26T23:45:00.543821+0000","last_undegraded":"2024-04-26T23:45:00.543821+0000","last_fullsized":"2024-04-26T23:45:00.543821+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:11:45.477579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544475+0000","last_change":"2024-04-26T23:43:03.297835+0000","last_active":"2024-04-26T23:45:00.544475+0000","last_peered":"2024-04-26T23:45:00.544475+0000","last_clean":"2024-04-26T23:45:00.544475+0000","last_became_active":"2024-04-26T23:43:03.297481+0000","last_became_peered":"2024-04-26T23:43:03.297481+0000","last_unstale":"2024-04-26T23:45:00.544475+0000","last_undegraded":"2024-04-26T23:45:00.544475+0000","last_fullsized":"2024-04-26T23:45:00.544475+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:52:54.757461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.543077+0000","last_change":"2024-04-26T23:43:07.536849+0000","last_active":"2024-04-26T23:45:00.543077+0000","last_peered":"2024-04-26T23:45:00.543077+0000","last_clean":"2024-04-26T23:45:00.543077+0000","last_became_active":"2024-04-26T23:43:07.536214+0000","last_became_peered":"2024-04-26T23:43:07.536214+0000","last_unstale":"2024-04-26T23:45:00.543077+0000","last_undegraded":"2024-04-26T23:45:00.543077+0000","last_fullsized":"2024-04-26T23:45:00.543077+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:33:16.782368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544263+0000","last_change":"2024-04-26T23:43:02.464181+0000","last_active":"2024-04-26T23:45:00.544263+0000","last_peered":"2024-04-26T23:45:00.544263+0000","last_clean":"2024-04-26T23:45:00.544263+0000","last_became_active":"2024-04-26T23:43:02.463830+0000","last_became_peered":"2024-04-26T23:43:02.463830+0000","last_unstale":"2024-04-26T23:45:00.544263+0000","last_undegraded":"2024-04-26T23:45:00.544263+0000","last_fullsized":"2024-04-26T23:45:00.544263+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:14:52.831302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544954+0000","last_change":"2024-04-26T23:43:03.298497+0000","last_active":"2024-04-26T23:45:00.544954+0000","last_peered":"2024-04-26T23:45:00.544954+0000","last_clean":"2024-04-26T23:45:00.544954+0000","last_became_active":"2024-04-26T23:43:03.298194+0000","last_became_peered":"2024-04-26T23:43:03.298194+0000","last_unstale":"2024-04-26T23:45:00.544954+0000","last_undegraded":"2024-04-26T23:45:00.544954+0000","last_fullsized":"2024-04-26T23:45:00.544954+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:39:19.007294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544441+0000","last_change":"2024-04-26T23:43:07.527962+0000","last_active":"2024-04-26T23:45:00.544441+0000","last_peered":"2024-04-26T23:45:00.544441+0000","last_clean":"2024-04-26T23:45:00.544441+0000","last_became_active":"2024-04-26T23:43:07.527386+0000","last_became_peered":"2024-04-26T23:43:07.527386+0000","last_unstale":"2024-04-26T23:45:00.544441+0000","last_undegraded":"2024-04-26T23:45:00.544441+0000","last_fullsized":"2024-04-26T23:45:00.544441+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:57:21.711670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.538734+0000","last_change":"2024-04-26T23:43:02.453988+0000","last_active":"2024-04-26T23:45:00.538734+0000","last_peered":"2024-04-26T23:45:00.538734+0000","last_clean":"2024-04-26T23:45:00.538734+0000","last_became_active":"2024-04-26T23:43:02.453654+0000","last_became_peered":"2024-04-26T23:43:02.453654+0000","last_unstale":"2024-04-26T23:45:00.538734+0000","last_undegraded":"2024-04-26T23:45:00.538734+0000","last_fullsized":"2024-04-26T23:45:00.538734+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:46:08.718705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544559+0000","last_change":"2024-04-26T23:43:03.305861+0000","last_active":"2024-04-26T23:45:00.544559+0000","last_peered":"2024-04-26T23:45:00.544559+0000","last_clean":"2024-04-26T23:45:00.544559+0000","last_became_active":"2024-04-26T23:43:03.305301+0000","last_became_peered":"2024-04-26T23:43:03.305301+0000","last_unstale":"2024-04-26T23:45:00.544559+0000","last_undegraded":"2024-04-26T23:45:00.544559+0000","last_fullsized":"2024-04-26T23:45:00.544559+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:08:24.174443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.537998+0000","last_change":"2024-04-26T23:43:07.529914+0000","last_active":"2024-04-26T23:45:00.537998+0000","last_peered":"2024-04-26T23:45:00.537998+0000","last_clean":"2024-04-26T23:45:00.537998+0000","last_became_active":"2024-04-26T23:43:07.529509+0000","last_became_peered":"2024-04-26T23:43:07.529509+0000","last_unstale":"2024-04-26T23:45:00.537998+0000","last_undegraded":"2024-04-26T23:45:00.537998+0000","last_fullsized":"2024-04-26T23:45:00.537998+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:50:23.293672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.552220+0000","last_change":"2024-04-26T23:43:02.452096+0000","last_active":"2024-04-26T23:45:00.552220+0000","last_peered":"2024-04-26T23:45:00.552220+0000","last_clean":"2024-04-26T23:45:00.552220+0000","last_became_active":"2024-04-26T23:43:02.450039+0000","last_became_peered":"2024-04-26T23:43:02.450039+0000","last_unstale":"2024-04-26T23:45:00.552220+0000","last_undegraded":"2024-04-26T23:45:00.552220+0000","last_fullsized":"2024-04-26T23:45:00.552220+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:47:03.062575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.545023+0000","last_change":"2024-04-26T23:43:03.294790+0000","last_active":"2024-04-26T23:45:00.545023+0000","last_peered":"2024-04-26T23:45:00.545023+0000","last_clean":"2024-04-26T23:45:00.545023+0000","last_became_active":"2024-04-26T23:43:03.294460+0000","last_became_peered":"2024-04-26T23:43:03.294460+0000","last_unstale":"2024-04-26T23:45:00.545023+0000","last_undegraded":"2024-04-26T23:45:00.545023+0000","last_fullsized":"2024-04-26T23:45:00.545023+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:57:40.879315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.541289+0000","last_change":"2024-04-26T23:43:03.284344+0000","last_active":"2024-04-26T23:45:00.541289+0000","last_peered":"2024-04-26T23:45:00.541289+0000","last_clean":"2024-04-26T23:45:00.541289+0000","last_became_active":"2024-04-26T23:43:03.283954+0000","last_became_peered":"2024-04-26T23:43:03.283954+0000","last_unstale":"2024-04-26T23:45:00.541289+0000","last_undegraded":"2024-04-26T23:45:00.541289+0000","last_fullsized":"2024-04-26T23:45:00.541289+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:12:56.516067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544580+0000","last_change":"2024-04-26T23:43:59.574959+0000","last_active":"2024-04-26T23:45:00.544580+0000","last_peered":"2024-04-26T23:45:00.544580+0000","last_clean":"2024-04-26T23:45:00.544580+0000","last_became_active":"2024-04-26T23:43:59.574588+0000","last_became_peered":"2024-04-26T23:43:59.574588+0000","last_unstale":"2024-04-26T23:45:00.544580+0000","last_undegraded":"2024-04-26T23:45:00.544580+0000","last_fullsized":"2024-04-26T23:45:00.544580+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:52:56.689439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.543637+0000","last_change":"2024-04-26T23:43:59.574946+0000","last_active":"2024-04-26T23:45:00.543637+0000","last_peered":"2024-04-26T23:45:00.543637+0000","last_clean":"2024-04-26T23:45:00.543637+0000","last_became_active":"2024-04-26T23:43:59.574238+0000","last_became_peered":"2024-04-26T23:43:59.574238+0000","last_unstale":"2024-04-26T23:45:00.543637+0000","last_undegraded":"2024-04-26T23:45:00.543637+0000","last_fullsized":"2024-04-26T23:45:00.543637+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:58:41.619868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,0],"acting":[2,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.545984+0000","last_change":"2024-04-26T23:43:03.304580+0000","last_active":"2024-04-26T23:45:00.545984+0000","last_peered":"2024-04-26T23:45:00.545984+0000","last_clean":"2024-04-26T23:45:00.545984+0000","last_became_active":"2024-04-26T23:43:03.303128+0000","last_became_peered":"2024-04-26T23:43:03.303128+0000","last_unstale":"2024-04-26T23:45:00.545984+0000","last_undegraded":"2024-04-26T23:45:00.545984+0000","last_fullsized":"2024-04-26T23:45:00.545984+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:11:36.868298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.553732+0000","last_change":"2024-04-26T23:43:03.306441+0000","last_active":"2024-04-26T23:45:00.553732+0000","last_peered":"2024-04-26T23:45:00.553732+0000","last_clean":"2024-04-26T23:45:00.553732+0000","last_became_active":"2024-04-26T23:43:03.305209+0000","last_became_peered":"2024-04-26T23:43:03.305209+0000","last_unstale":"2024-04-26T23:45:00.553732+0000","last_undegraded":"2024-04-26T23:45:00.553732+0000","last_fullsized":"2024-04-26T23:45:00.553732+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:41:39.757001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.541740+0000","last_change":"2024-04-26T23:43:03.284291+0000","last_active":"2024-04-26T23:45:00.541740+0000","last_peered":"2024-04-26T23:45:00.541740+0000","last_clean":"2024-04-26T23:45:00.541740+0000","last_became_active":"2024-04-26T23:43:03.284006+0000","last_became_peered":"2024-04-26T23:43:03.284006+0000","last_unstale":"2024-04-26T23:45:00.541740+0000","last_undegraded":"2024-04-26T23:45:00.541740+0000","last_fullsized":"2024-04-26T23:45:00.541740+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:24:01.386322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544280+0000","last_change":"2024-04-26T23:43:03.286478+0000","last_active":"2024-04-26T23:45:00.544280+0000","last_peered":"2024-04-26T23:45:00.544280+0000","last_clean":"2024-04-26T23:45:00.544280+0000","last_became_active":"2024-04-26T23:43:03.286115+0000","last_became_peered":"2024-04-26T23:43:03.286115+0000","last_unstale":"2024-04-26T23:45:00.544280+0000","last_undegraded":"2024-04-26T23:45:00.544280+0000","last_fullsized":"2024-04-26T23:45:00.544280+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:34:57.534124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.552134+0000","last_change":"2024-04-26T23:43:59.872042+0000","last_active":"2024-04-26T23:45:00.552134+0000","last_peered":"2024-04-26T23:45:00.552134+0000","last_clean":"2024-04-26T23:45:00.552134+0000","last_became_active":"2024-04-26T23:43:59.870288+0000","last_became_peered":"2024-04-26T23:43:59.870288+0000","last_unstale":"2024-04-26T23:45:00.552134+0000","last_undegraded":"2024-04-26T23:45:00.552134+0000","last_fullsized":"2024-04-26T23:45:00.552134+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:16:38.518370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.538251+0000","last_change":"2024-04-26T23:43:03.300602+0000","last_active":"2024-04-26T23:45:00.538251+0000","last_peered":"2024-04-26T23:45:00.538251+0000","last_clean":"2024-04-26T23:45:00.538251+0000","last_became_active":"2024-04-26T23:43:03.300225+0000","last_became_peered":"2024-04-26T23:43:03.300225+0000","last_unstale":"2024-04-26T23:45:00.538251+0000","last_undegraded":"2024-04-26T23:45:00.538251+0000","last_fullsized":"2024-04-26T23:45:00.538251+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:52:36.638153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.543535+0000","last_change":"2024-04-26T23:43:03.305631+0000","last_active":"2024-04-26T23:45:00.543535+0000","last_peered":"2024-04-26T23:45:00.543535+0000","last_clean":"2024-04-26T23:45:00.543535+0000","last_became_active":"2024-04-26T23:43:03.300193+0000","last_became_peered":"2024-04-26T23:43:03.300193+0000","last_unstale":"2024-04-26T23:45:00.543535+0000","last_undegraded":"2024-04-26T23:45:00.543535+0000","last_fullsized":"2024-04-26T23:45:00.543535+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:27:08.355753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.543271+0000","last_change":"2024-04-26T23:43:03.309514+0000","last_active":"2024-04-26T23:45:00.543271+0000","last_peered":"2024-04-26T23:45:00.543271+0000","last_clean":"2024-04-26T23:45:00.543271+0000","last_became_active":"2024-04-26T23:43:03.309085+0000","last_became_peered":"2024-04-26T23:43:03.309085+0000","last_unstale":"2024-04-26T23:45:00.543271+0000","last_undegraded":"2024-04-26T23:45:00.543271+0000","last_fullsized":"2024-04-26T23:45:00.543271+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:08:45.649513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.547848+0000","last_change":"2024-04-26T23:43:03.303553+0000","last_active":"2024-04-26T23:45:00.547848+0000","last_peered":"2024-04-26T23:45:00.547848+0000","last_clean":"2024-04-26T23:45:00.547848+0000","last_became_active":"2024-04-26T23:43:03.302979+0000","last_became_peered":"2024-04-26T23:43:03.302979+0000","last_unstale":"2024-04-26T23:45:00.547848+0000","last_undegraded":"2024-04-26T23:45:00.547848+0000","last_fullsized":"2024-04-26T23:45:00.547848+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:44:25.809138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.541360+0000","last_change":"2024-04-26T23:43:03.288433+0000","last_active":"2024-04-26T23:45:00.541360+0000","last_peered":"2024-04-26T23:45:00.541360+0000","last_clean":"2024-04-26T23:45:00.541360+0000","last_became_active":"2024-04-26T23:43:03.288125+0000","last_became_peered":"2024-04-26T23:43:03.288125+0000","last_unstale":"2024-04-26T23:45:00.541360+0000","last_undegraded":"2024-04-26T23:45:00.541360+0000","last_fullsized":"2024-04-26T23:45:00.541360+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:53:21.673755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":282161,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":59218,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":620,"num_read_kb":372,"num_write":164,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":134,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":177654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141,"ondisk_log_size":141,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383721,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1024000,"data_stored":519912,"data_compressed":26528,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645357,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32316,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712836,"statfs":{"total":95995035648,"available":95961944064,"internally_reserved":0,"allocated":1159168,"data_stored":1162375,"data_compressed":62415,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808883,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1040384,"data_stored":524222,"data_compressed":26528,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070521,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1032192,"data_stored":520024,"data_compressed":26528,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332159,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1085440,"data_stored":1116346,"data_compressed":62415,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495684,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1015808,"data_stored":515302,"data_compressed":26528,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574945,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1077248,"data_stored":570038,"data_compressed":26528,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738469,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":516690,"data_compressed":26528,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411420,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1093632,"data_stored":571667,"data_compressed":26528,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182485,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1028096,"data_stored":515496,"data_compressed":26528,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920848,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1036288,"data_stored":520455,"data_compressed":26528,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757323,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32248,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712904,"statfs":{"total":95995035648,"available":95962013696,"internally_reserved":0,"allocated":1093632,"data_stored":1111543,"data_compressed":62415,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34242,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34242,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34242,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35887,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35887,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35887,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35887,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35887,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":54826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":56455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":56795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T23:45:14.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:14 smithi072 ceph-mon[28184]: from='client.25076 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T23:45:14.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:14 smithi136 ceph-mon[33050]: from='client.25076 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T23:45:14.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:14 smithi076 ceph-mon[32774]: from='client.25076 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T23:45:15.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:15 smithi072 ceph-mon[28184]: pgmap v391: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 1 B/s, 0 objects/s recovering 2024-04-26T23:45:15.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:15 smithi136 ceph-mon[33050]: pgmap v391: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 1 B/s, 0 objects/s recovering 2024-04-26T23:45:15.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:15 smithi076 ceph-mon[32774]: pgmap v391: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 1 B/s, 0 objects/s recovering 2024-04-26T23:45:17.094 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T23:45:17.539 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:45:17.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:17 smithi136 ceph-mon[33050]: pgmap v392: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 1 B/s, 0 objects/s recovering 2024-04-26T23:45:17.879 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:17 smithi072 ceph-mon[28184]: pgmap v392: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 1 B/s, 0 objects/s recovering 2024-04-26T23:45:17.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:17 smithi076 ceph-mon[32774]: pgmap v392: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 1 B/s, 0 objects/s recovering 2024-04-26T23:45:19.703 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:19 smithi072 ceph-mon[28184]: pgmap v393: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:45:19.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:19.863+0000 7f910c800700 1 -- 172.21.15.72:0/2724658314 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9104105660 msgr2=0x7f9104105ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:19.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:19.863+0000 7f910c800700 1 --2- 172.21.15.72:0/2724658314 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9104105660 0x7f9104105ae0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f90f40098f0 tx=0x7f90f4009c00 comp rx=0 tx=0).stop 2024-04-26T23:45:19.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:19.863+0000 7f910c800700 1 -- 172.21.15.72:0/2724658314 shutdown_connections 2024-04-26T23:45:19.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:19.863+0000 7f910c800700 1 --2- 172.21.15.72:0/2724658314 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9104106020 0x7f9104112530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:19.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:19.863+0000 7f910c800700 1 --2- 172.21.15.72:0/2724658314 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f910410b660 0x7f910410ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:19.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:19.863+0000 7f910c800700 1 --2- 172.21.15.72:0/2724658314 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9104105660 0x7f9104105ae0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:19.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:19.864+0000 7f910c800700 1 -- 172.21.15.72:0/2724658314 >> 172.21.15.72:0/2724658314 conn(0x7f9104101150 msgr2=0x7f9104103570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:19.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:19.864+0000 7f910c800700 1 -- 172.21.15.72:0/2724658314 shutdown_connections 2024-04-26T23:45:19.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:19.864+0000 7f910c800700 1 -- 172.21.15.72:0/2724658314 wait complete. 2024-04-26T23:45:19.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:19.865+0000 7f910c800700 1 Processor -- start 2024-04-26T23:45:19.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:19.865+0000 7f910c800700 1 -- start start 2024-04-26T23:45:19.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:19.865+0000 7f910c800700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9104105660 0x7f910419be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:19.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:19.865+0000 7f910c800700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9104106020 0x7f910419c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:19.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:19.865+0000 7f910c800700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f910410b660 0x7f91041a06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:19.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:19.865+0000 7f910c800700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f91041158f0 con 0x7f9104106020 2024-04-26T23:45:19.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:19.865+0000 7f910c800700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f9104115770 con 0x7f910410b660 2024-04-26T23:45:19.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:19.865+0000 7f910c800700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f9104115a70 con 0x7f9104105660 2024-04-26T23:45:19.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:19.865+0000 7f9109e00700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9104105660 0x7f910419be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:19.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:19.865+0000 7f9109400700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9104106020 0x7f910419c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:19.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:19.865+0000 7f9109e00700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9104105660 0x7f910419be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.72:48300/0 (socket says 172.21.15.72:48300) 2024-04-26T23:45:19.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:19.865+0000 7f9109e00700 1 -- 172.21.15.72:0/1525004092 learned_addr learned my addr 172.21.15.72:0/1525004092 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:45:19.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:19.866+0000 7f910a800700 1 --2- 172.21.15.72:0/1525004092 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f910410b660 0x7f91041a06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:19.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:19.866+0000 7f9109e00700 1 -- 172.21.15.72:0/1525004092 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f910410b660 msgr2=0x7f91041a06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:19.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:19.866+0000 7f9109e00700 1 --2- 172.21.15.72:0/1525004092 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f910410b660 0x7f91041a06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:19.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:19.866+0000 7f9109e00700 1 -- 172.21.15.72:0/1525004092 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9104106020 msgr2=0x7f910419c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:19.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:19.866+0000 7f9109e00700 1 --2- 172.21.15.72:0/1525004092 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9104106020 0x7f910419c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:19.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:19.866+0000 7f9109e00700 1 -- 172.21.15.72:0/1525004092 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91041a0e10 con 0x7f9104105660 2024-04-26T23:45:19.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:19.866+0000 7f9109400700 1 --2- 172.21.15.72:0/1525004092 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9104106020 0x7f910419c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:45:19.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:19.866+0000 7f9109e00700 1 --2- 172.21.15.72:0/1525004092 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9104105660 0x7f910419be00 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f90f800e9a0 tx=0x7f90f800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:19.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:19.867+0000 7f90fea00700 1 -- 172.21.15.72:0/1525004092 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90f800ca10 con 0x7f9104105660 2024-04-26T23:45:19.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:19.867+0000 7f910c800700 1 -- 172.21.15.72:0/1525004092 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91041a1100 con 0x7f9104105660 2024-04-26T23:45:19.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:19.867+0000 7f90fea00700 1 -- 172.21.15.72:0/1525004092 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90f800cbb0 con 0x7f9104105660 2024-04-26T23:45:19.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:19.867+0000 7f90fea00700 1 -- 172.21.15.72:0/1525004092 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90f8018680 con 0x7f9104105660 2024-04-26T23:45:19.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:19.867+0000 7f910c800700 1 -- 172.21.15.72:0/1525004092 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91041a8980 con 0x7f9104105660 2024-04-26T23:45:19.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:19.868+0000 7f910c800700 1 -- 172.21.15.72:0/1525004092 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f910406a110 con 0x7f9104105660 2024-04-26T23:45:19.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:19.870+0000 7f90fea00700 1 -- 172.21.15.72:0/1525004092 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f90f8010430 con 0x7f9104105660 2024-04-26T23:45:19.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:19.872+0000 7f90fea00700 1 --2- 172.21.15.72:0/1525004092 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f90ec0a3cd0 0x7f90ec0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:19.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:19.872+0000 7f90fea00700 1 -- 172.21.15.72:0/1525004092 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f90f80d0a60 con 0x7f9104105660 2024-04-26T23:45:19.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:19.872+0000 7f9109400700 1 --2- 172.21.15.72:0/1525004092 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f90ec0a3cd0 0x7f90ec0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:19.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:19.873+0000 7f9109400700 1 --2- 172.21.15.72:0/1525004092 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f90ec0a3cd0 0x7f90ec0a6190 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f90f40098c0 tx=0x7f90f4019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:19.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:19.876+0000 7f90fea00700 1 -- 172.21.15.72:0/1525004092 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f90f809cef0 con 0x7f9104105660 2024-04-26T23:45:19.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:19 smithi076 ceph-mon[32774]: pgmap v393: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:45:20.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:19 smithi136 ceph-mon[33050]: pgmap v393: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:45:20.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:20.099+0000 7f910c800700 1 -- 172.21.15.72:0/1525004092 --> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9104063860 con 0x7f90ec0a3cd0 2024-04-26T23:45:20.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:20.129+0000 7f90fea00700 1 -- 172.21.15.72:0/1525004092 <== mgr.14152 v2:172.21.15.72:6800/4281177709 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340655 (secure 0 0 0) 0x7f9104063860 con 0x7f90ec0a3cd0 2024-04-26T23:45:20.139 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:45:20.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:20.136+0000 7f910c800700 1 -- 172.21.15.72:0/1525004092 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f90ec0a3cd0 msgr2=0x7f90ec0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:20.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:20.136+0000 7f910c800700 1 --2- 172.21.15.72:0/1525004092 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f90ec0a3cd0 0x7f90ec0a6190 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f90f40098c0 tx=0x7f90f4019040 comp rx=0 tx=0).stop 2024-04-26T23:45:20.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:20.136+0000 7f910c800700 1 -- 172.21.15.72:0/1525004092 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9104105660 msgr2=0x7f910419be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:20.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:20.136+0000 7f910c800700 1 --2- 172.21.15.72:0/1525004092 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9104105660 0x7f910419be00 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f90f800e9a0 tx=0x7f90f800ecb0 comp rx=0 tx=0).stop 2024-04-26T23:45:20.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:20.136+0000 7f910c800700 1 -- 172.21.15.72:0/1525004092 shutdown_connections 2024-04-26T23:45:20.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:20.136+0000 7f910c800700 1 --2- 172.21.15.72:0/1525004092 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f90ec0a3cd0 0x7f90ec0a6190 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:20.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:20.136+0000 7f910c800700 1 --2- 172.21.15.72:0/1525004092 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f910410b660 0x7f91041a06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:20.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:20.136+0000 7f910c800700 1 --2- 172.21.15.72:0/1525004092 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9104105660 0x7f910419be00 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:20.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:20.136+0000 7f910c800700 1 --2- 172.21.15.72:0/1525004092 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9104106020 0x7f910419c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:20.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:20.136+0000 7f910c800700 1 -- 172.21.15.72:0/1525004092 >> 172.21.15.72:0/1525004092 conn(0x7f9104101150 msgr2=0x7f9104101910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:20.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:20.137+0000 7f910c800700 1 -- 172.21.15.72:0/1525004092 shutdown_connections 2024-04-26T23:45:20.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:20.137+0000 7f910c800700 1 -- 172.21.15.72:0/1525004092 wait complete. 2024-04-26T23:45:20.143 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-04-26T23:45:21.009 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":393,"stamp":"2024-04-26T23:45:18.470338+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":649820,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":776,"num_read_kb":496,"num_write":347,"num_write_kb":3050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":1180870,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":280,"ondisk_log_size":280,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":357500,"kb_used_data":12428,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124584324,"statfs":{"total":1151940427776,"available":1151574347776,"internally_reserved":0,"allocated":12726272,"data_stored":8164070,"data_compressed":425997,"data_compressed_allocated":577536,"data_compressed_original":2654208,"omap_allocated":19139,"internal_metadata":353154365},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":60,"num_read_kb":30,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006001"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.542078+0000","last_change":"2024-04-26T23:43:03.301992+0000","last_active":"2024-04-26T23:45:00.542078+0000","last_peered":"2024-04-26T23:45:00.542078+0000","last_clean":"2024-04-26T23:45:00.542078+0000","last_became_active":"2024-04-26T23:43:03.301498+0000","last_became_peered":"2024-04-26T23:43:03.301498+0000","last_unstale":"2024-04-26T23:45:00.542078+0000","last_undegraded":"2024-04-26T23:45:00.542078+0000","last_fullsized":"2024-04-26T23:45:00.542078+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:40:53.567805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.548674+0000","last_change":"2024-04-26T23:43:03.284964+0000","last_active":"2024-04-26T23:45:00.548674+0000","last_peered":"2024-04-26T23:45:00.548674+0000","last_clean":"2024-04-26T23:45:00.548674+0000","last_became_active":"2024-04-26T23:43:03.284185+0000","last_became_peered":"2024-04-26T23:43:03.284185+0000","last_unstale":"2024-04-26T23:45:00.548674+0000","last_undegraded":"2024-04-26T23:45:00.548674+0000","last_fullsized":"2024-04-26T23:45:00.548674+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:55:35.325775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.547719+0000","last_change":"2024-04-26T23:43:03.300333+0000","last_active":"2024-04-26T23:45:00.547719+0000","last_peered":"2024-04-26T23:45:00.547719+0000","last_clean":"2024-04-26T23:45:00.547719+0000","last_became_active":"2024-04-26T23:43:03.295779+0000","last_became_peered":"2024-04-26T23:43:03.295779+0000","last_unstale":"2024-04-26T23:45:00.547719+0000","last_undegraded":"2024-04-26T23:45:00.547719+0000","last_fullsized":"2024-04-26T23:45:00.547719+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:45:06.598862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.541449+0000","last_change":"2024-04-26T23:43:03.290211+0000","last_active":"2024-04-26T23:45:00.541449+0000","last_peered":"2024-04-26T23:45:00.541449+0000","last_clean":"2024-04-26T23:45:00.541449+0000","last_became_active":"2024-04-26T23:43:03.289853+0000","last_became_peered":"2024-04-26T23:43:03.289853+0000","last_unstale":"2024-04-26T23:45:00.541449+0000","last_undegraded":"2024-04-26T23:45:00.541449+0000","last_fullsized":"2024-04-26T23:45:00.541449+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:14:45.032300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.543721+0000","last_change":"2024-04-26T23:43:03.309667+0000","last_active":"2024-04-26T23:45:00.543721+0000","last_peered":"2024-04-26T23:45:00.543721+0000","last_clean":"2024-04-26T23:45:00.543721+0000","last_became_active":"2024-04-26T23:43:03.309305+0000","last_became_peered":"2024-04-26T23:43:03.309305+0000","last_unstale":"2024-04-26T23:45:00.543721+0000","last_undegraded":"2024-04-26T23:45:00.543721+0000","last_fullsized":"2024-04-26T23:45:00.543721+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:31:56.739866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.548338+0000","last_change":"2024-04-26T23:43:03.292107+0000","last_active":"2024-04-26T23:45:00.548338+0000","last_peered":"2024-04-26T23:45:00.548338+0000","last_clean":"2024-04-26T23:45:00.548338+0000","last_became_active":"2024-04-26T23:43:03.291760+0000","last_became_peered":"2024-04-26T23:43:03.291760+0000","last_unstale":"2024-04-26T23:45:00.548338+0000","last_undegraded":"2024-04-26T23:45:00.548338+0000","last_fullsized":"2024-04-26T23:45:00.548338+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:31:54.654379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544201+0000","last_change":"2024-04-26T23:43:03.294923+0000","last_active":"2024-04-26T23:45:00.544201+0000","last_peered":"2024-04-26T23:45:00.544201+0000","last_clean":"2024-04-26T23:45:00.544201+0000","last_became_active":"2024-04-26T23:43:03.294523+0000","last_became_peered":"2024-04-26T23:43:03.294523+0000","last_unstale":"2024-04-26T23:45:00.544201+0000","last_undegraded":"2024-04-26T23:45:00.544201+0000","last_fullsized":"2024-04-26T23:45:00.544201+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:36:19.419773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.542769+0000","last_change":"2024-04-26T23:43:03.301335+0000","last_active":"2024-04-26T23:45:00.542769+0000","last_peered":"2024-04-26T23:45:00.542769+0000","last_clean":"2024-04-26T23:45:00.542769+0000","last_became_active":"2024-04-26T23:43:03.300918+0000","last_became_peered":"2024-04-26T23:43:03.300918+0000","last_unstale":"2024-04-26T23:45:00.542769+0000","last_undegraded":"2024-04-26T23:45:00.542769+0000","last_fullsized":"2024-04-26T23:45:00.542769+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:15:05.719452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.543973+0000","last_change":"2024-04-26T23:43:59.574776+0000","last_active":"2024-04-26T23:45:00.543973+0000","last_peered":"2024-04-26T23:45:00.543973+0000","last_clean":"2024-04-26T23:45:00.543973+0000","last_became_active":"2024-04-26T23:43:59.574299+0000","last_became_peered":"2024-04-26T23:43:59.574299+0000","last_unstale":"2024-04-26T23:45:00.543973+0000","last_undegraded":"2024-04-26T23:45:00.543973+0000","last_fullsized":"2024-04-26T23:45:00.543973+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:14:16.089377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,11],"acting":[1,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.548713+0000","last_change":"2024-04-26T23:43:03.287643+0000","last_active":"2024-04-26T23:45:00.548713+0000","last_peered":"2024-04-26T23:45:00.548713+0000","last_clean":"2024-04-26T23:45:00.548713+0000","last_became_active":"2024-04-26T23:43:03.287324+0000","last_became_peered":"2024-04-26T23:43:03.287324+0000","last_unstale":"2024-04-26T23:45:00.548713+0000","last_undegraded":"2024-04-26T23:45:00.548713+0000","last_fullsized":"2024-04-26T23:45:00.548713+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:50:14.816548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544950+0000","last_change":"2024-04-26T23:43:03.296421+0000","last_active":"2024-04-26T23:45:00.544950+0000","last_peered":"2024-04-26T23:45:00.544950+0000","last_clean":"2024-04-26T23:45:00.544950+0000","last_became_active":"2024-04-26T23:43:03.296060+0000","last_became_peered":"2024-04-26T23:43:03.296060+0000","last_unstale":"2024-04-26T23:45:00.544950+0000","last_undegraded":"2024-04-26T23:45:00.544950+0000","last_fullsized":"2024-04-26T23:45:00.544950+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:50:44.393446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.541254+0000","last_change":"2024-04-26T23:43:03.288562+0000","last_active":"2024-04-26T23:45:00.541254+0000","last_peered":"2024-04-26T23:45:00.541254+0000","last_clean":"2024-04-26T23:45:00.541254+0000","last_became_active":"2024-04-26T23:43:03.288219+0000","last_became_peered":"2024-04-26T23:43:03.288219+0000","last_unstale":"2024-04-26T23:45:00.541254+0000","last_undegraded":"2024-04-26T23:45:00.541254+0000","last_fullsized":"2024-04-26T23:45:00.541254+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:54:32.107146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.547419+0000","last_change":"2024-04-26T23:43:03.296840+0000","last_active":"2024-04-26T23:45:00.547419+0000","last_peered":"2024-04-26T23:45:00.547419+0000","last_clean":"2024-04-26T23:45:00.547419+0000","last_became_active":"2024-04-26T23:43:03.296501+0000","last_became_peered":"2024-04-26T23:43:03.296501+0000","last_unstale":"2024-04-26T23:45:00.547419+0000","last_undegraded":"2024-04-26T23:45:00.547419+0000","last_fullsized":"2024-04-26T23:45:00.547419+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:07:05.293388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.546627+0000","last_change":"2024-04-26T23:43:03.291739+0000","last_active":"2024-04-26T23:45:00.546627+0000","last_peered":"2024-04-26T23:45:00.546627+0000","last_clean":"2024-04-26T23:45:00.546627+0000","last_became_active":"2024-04-26T23:43:03.287839+0000","last_became_peered":"2024-04-26T23:43:03.287839+0000","last_unstale":"2024-04-26T23:45:00.546627+0000","last_undegraded":"2024-04-26T23:45:00.546627+0000","last_fullsized":"2024-04-26T23:45:00.546627+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:10:56.698324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.546051+0000","last_change":"2024-04-26T23:43:03.295767+0000","last_active":"2024-04-26T23:45:00.546051+0000","last_peered":"2024-04-26T23:45:00.546051+0000","last_clean":"2024-04-26T23:45:00.546051+0000","last_became_active":"2024-04-26T23:43:03.295339+0000","last_became_peered":"2024-04-26T23:43:03.295339+0000","last_unstale":"2024-04-26T23:45:00.546051+0000","last_undegraded":"2024-04-26T23:45:00.546051+0000","last_fullsized":"2024-04-26T23:45:00.546051+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:21:44.410953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.542951+0000","last_change":"2024-04-26T23:43:03.301687+0000","last_active":"2024-04-26T23:45:00.542951+0000","last_peered":"2024-04-26T23:45:00.542951+0000","last_clean":"2024-04-26T23:45:00.542951+0000","last_became_active":"2024-04-26T23:43:03.301355+0000","last_became_peered":"2024-04-26T23:43:03.301355+0000","last_unstale":"2024-04-26T23:45:00.542951+0000","last_undegraded":"2024-04-26T23:45:00.542951+0000","last_fullsized":"2024-04-26T23:45:00.542951+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:00:57.991022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.541916+0000","last_change":"2024-04-26T23:43:03.302109+0000","last_active":"2024-04-26T23:45:00.541916+0000","last_peered":"2024-04-26T23:45:00.541916+0000","last_clean":"2024-04-26T23:45:00.541916+0000","last_became_active":"2024-04-26T23:43:03.301725+0000","last_became_peered":"2024-04-26T23:43:03.301725+0000","last_unstale":"2024-04-26T23:45:00.541916+0000","last_undegraded":"2024-04-26T23:45:00.541916+0000","last_fullsized":"2024-04-26T23:45:00.541916+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:36:44.630125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.557182+0000","last_change":"2024-04-26T23:43:03.298880+0000","last_active":"2024-04-26T23:45:00.557182+0000","last_peered":"2024-04-26T23:45:00.557182+0000","last_clean":"2024-04-26T23:45:00.557182+0000","last_became_active":"2024-04-26T23:43:03.298464+0000","last_became_peered":"2024-04-26T23:43:03.298464+0000","last_unstale":"2024-04-26T23:45:00.557182+0000","last_undegraded":"2024-04-26T23:45:00.557182+0000","last_fullsized":"2024-04-26T23:45:00.557182+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:42:54.745002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544317+0000","last_change":"2024-04-26T23:43:03.309454+0000","last_active":"2024-04-26T23:45:00.544317+0000","last_peered":"2024-04-26T23:45:00.544317+0000","last_clean":"2024-04-26T23:45:00.544317+0000","last_became_active":"2024-04-26T23:43:03.309091+0000","last_became_peered":"2024-04-26T23:43:03.309091+0000","last_unstale":"2024-04-26T23:45:00.544317+0000","last_undegraded":"2024-04-26T23:45:00.544317+0000","last_fullsized":"2024-04-26T23:45:00.544317+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:46:12.333410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.543829+0000","last_change":"2024-04-26T23:43:07.541317+0000","last_active":"2024-04-26T23:45:00.543829+0000","last_peered":"2024-04-26T23:45:00.543829+0000","last_clean":"2024-04-26T23:45:00.543829+0000","last_became_active":"2024-04-26T23:43:07.540760+0000","last_became_peered":"2024-04-26T23:43:07.540760+0000","last_unstale":"2024-04-26T23:45:00.543829+0000","last_undegraded":"2024-04-26T23:45:00.543829+0000","last_fullsized":"2024-04-26T23:45:00.543829+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:28:55.545416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.557174+0000","last_change":"2024-04-26T23:43:02.450956+0000","last_active":"2024-04-26T23:45:00.557174+0000","last_peered":"2024-04-26T23:45:00.557174+0000","last_clean":"2024-04-26T23:45:00.557174+0000","last_became_active":"2024-04-26T23:43:02.446733+0000","last_became_peered":"2024-04-26T23:43:02.446733+0000","last_unstale":"2024-04-26T23:45:00.557174+0000","last_undegraded":"2024-04-26T23:45:00.557174+0000","last_fullsized":"2024-04-26T23:45:00.557174+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:58:47.613575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.551893+0000","last_change":"2024-04-26T23:43:03.311681+0000","last_active":"2024-04-26T23:45:00.551893+0000","last_peered":"2024-04-26T23:45:00.551893+0000","last_clean":"2024-04-26T23:45:00.551893+0000","last_became_active":"2024-04-26T23:43:03.311087+0000","last_became_peered":"2024-04-26T23:43:03.311087+0000","last_unstale":"2024-04-26T23:45:00.551893+0000","last_undegraded":"2024-04-26T23:45:00.551893+0000","last_fullsized":"2024-04-26T23:45:00.551893+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:13:00.744062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.540679+0000","last_change":"2024-04-26T23:43:07.543547+0000","last_active":"2024-04-26T23:45:00.540679+0000","last_peered":"2024-04-26T23:45:00.540679+0000","last_clean":"2024-04-26T23:45:00.540679+0000","last_became_active":"2024-04-26T23:43:07.543073+0000","last_became_peered":"2024-04-26T23:43:07.543073+0000","last_unstale":"2024-04-26T23:45:00.540679+0000","last_undegraded":"2024-04-26T23:45:00.540679+0000","last_fullsized":"2024-04-26T23:45:00.540679+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:30:05.418875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.552100+0000","last_change":"2024-04-26T23:43:02.460459+0000","last_active":"2024-04-26T23:45:00.552100+0000","last_peered":"2024-04-26T23:45:00.552100+0000","last_clean":"2024-04-26T23:45:00.552100+0000","last_became_active":"2024-04-26T23:43:02.459652+0000","last_became_peered":"2024-04-26T23:43:02.459652+0000","last_unstale":"2024-04-26T23:45:00.552100+0000","last_undegraded":"2024-04-26T23:45:00.552100+0000","last_fullsized":"2024-04-26T23:45:00.552100+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:27:19.166748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.549681+0000","last_change":"2024-04-26T23:43:03.300078+0000","last_active":"2024-04-26T23:45:00.549681+0000","last_peered":"2024-04-26T23:45:00.549681+0000","last_clean":"2024-04-26T23:45:00.549681+0000","last_became_active":"2024-04-26T23:43:03.296319+0000","last_became_peered":"2024-04-26T23:43:03.296319+0000","last_unstale":"2024-04-26T23:45:00.549681+0000","last_undegraded":"2024-04-26T23:45:00.549681+0000","last_fullsized":"2024-04-26T23:45:00.549681+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:33:51.859539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.540906+0000","last_change":"2024-04-26T23:43:07.543494+0000","last_active":"2024-04-26T23:45:00.540906+0000","last_peered":"2024-04-26T23:45:00.540906+0000","last_clean":"2024-04-26T23:45:00.540906+0000","last_became_active":"2024-04-26T23:43:07.542806+0000","last_became_peered":"2024-04-26T23:43:07.542806+0000","last_unstale":"2024-04-26T23:45:00.540906+0000","last_undegraded":"2024-04-26T23:45:00.540906+0000","last_fullsized":"2024-04-26T23:45:00.540906+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:54:40.522615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544910+0000","last_change":"2024-04-26T23:43:02.438943+0000","last_active":"2024-04-26T23:45:00.544910+0000","last_peered":"2024-04-26T23:45:00.544910+0000","last_clean":"2024-04-26T23:45:00.544910+0000","last_became_active":"2024-04-26T23:43:02.438604+0000","last_became_peered":"2024-04-26T23:43:02.438604+0000","last_unstale":"2024-04-26T23:45:00.544910+0000","last_undegraded":"2024-04-26T23:45:00.544910+0000","last_fullsized":"2024-04-26T23:45:00.544910+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:06:51.054342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.543352+0000","last_change":"2024-04-26T23:43:03.293856+0000","last_active":"2024-04-26T23:45:00.543352+0000","last_peered":"2024-04-26T23:45:00.543352+0000","last_clean":"2024-04-26T23:45:00.543352+0000","last_became_active":"2024-04-26T23:43:03.293189+0000","last_became_peered":"2024-04-26T23:43:03.293189+0000","last_unstale":"2024-04-26T23:45:00.543352+0000","last_undegraded":"2024-04-26T23:45:00.543352+0000","last_fullsized":"2024-04-26T23:45:00.543352+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:50:27.545567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.540743+0000","last_change":"2024-04-26T23:43:07.535396+0000","last_active":"2024-04-26T23:45:00.540743+0000","last_peered":"2024-04-26T23:45:00.540743+0000","last_clean":"2024-04-26T23:45:00.540743+0000","last_became_active":"2024-04-26T23:43:07.534914+0000","last_became_peered":"2024-04-26T23:43:07.534914+0000","last_unstale":"2024-04-26T23:45:00.540743+0000","last_undegraded":"2024-04-26T23:45:00.540743+0000","last_fullsized":"2024-04-26T23:45:00.540743+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:27:31.199079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544008+0000","last_change":"2024-04-26T23:43:02.442935+0000","last_active":"2024-04-26T23:45:00.544008+0000","last_peered":"2024-04-26T23:45:00.544008+0000","last_clean":"2024-04-26T23:45:00.544008+0000","last_became_active":"2024-04-26T23:43:02.442588+0000","last_became_peered":"2024-04-26T23:43:02.442588+0000","last_unstale":"2024-04-26T23:45:00.544008+0000","last_undegraded":"2024-04-26T23:45:00.544008+0000","last_fullsized":"2024-04-26T23:45:00.544008+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:05:26.442001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.538765+0000","last_change":"2024-04-26T23:43:03.289407+0000","last_active":"2024-04-26T23:45:00.538765+0000","last_peered":"2024-04-26T23:45:00.538765+0000","last_clean":"2024-04-26T23:45:00.538765+0000","last_became_active":"2024-04-26T23:43:03.287951+0000","last_became_peered":"2024-04-26T23:43:03.287951+0000","last_unstale":"2024-04-26T23:45:00.538765+0000","last_undegraded":"2024-04-26T23:45:00.538765+0000","last_fullsized":"2024-04-26T23:45:00.538765+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:24:42.687653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544134+0000","last_change":"2024-04-26T23:43:07.527019+0000","last_active":"2024-04-26T23:45:00.544134+0000","last_peered":"2024-04-26T23:45:00.544134+0000","last_clean":"2024-04-26T23:45:00.544134+0000","last_became_active":"2024-04-26T23:43:07.526516+0000","last_became_peered":"2024-04-26T23:43:07.526516+0000","last_unstale":"2024-04-26T23:45:00.544134+0000","last_undegraded":"2024-04-26T23:45:00.544134+0000","last_fullsized":"2024-04-26T23:45:00.544134+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:44:41.267144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'6","reported_seq":144,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:12.359494+0000","last_change":"2024-04-26T23:43:02.445159+0000","last_active":"2024-04-26T23:45:12.359494+0000","last_peered":"2024-04-26T23:45:12.359494+0000","last_clean":"2024-04-26T23:45:12.359494+0000","last_became_active":"2024-04-26T23:43:02.444781+0000","last_became_peered":"2024-04-26T23:43:02.444781+0000","last_unstale":"2024-04-26T23:45:12.359494+0000","last_undegraded":"2024-04-26T23:45:12.359494+0000","last_fullsized":"2024-04-26T23:45:12.359494+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:45:16.509790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":107,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.543904+0000","last_change":"2024-04-26T23:43:03.292717+0000","last_active":"2024-04-26T23:45:00.543904+0000","last_peered":"2024-04-26T23:45:00.543904+0000","last_clean":"2024-04-26T23:45:00.543904+0000","last_became_active":"2024-04-26T23:43:03.292355+0000","last_became_peered":"2024-04-26T23:43:03.292355+0000","last_unstale":"2024-04-26T23:45:00.543904+0000","last_undegraded":"2024-04-26T23:45:00.543904+0000","last_fullsized":"2024-04-26T23:45:00.543904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:06:02.063165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544064+0000","last_change":"2024-04-26T23:43:07.540758+0000","last_active":"2024-04-26T23:45:00.544064+0000","last_peered":"2024-04-26T23:45:00.544064+0000","last_clean":"2024-04-26T23:45:00.544064+0000","last_became_active":"2024-04-26T23:43:07.540305+0000","last_became_peered":"2024-04-26T23:43:07.540305+0000","last_unstale":"2024-04-26T23:45:00.544064+0000","last_undegraded":"2024-04-26T23:45:00.544064+0000","last_fullsized":"2024-04-26T23:45:00.544064+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:08:04.491544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.548197+0000","last_change":"2024-04-26T23:43:02.453650+0000","last_active":"2024-04-26T23:45:00.548197+0000","last_peered":"2024-04-26T23:45:00.548197+0000","last_clean":"2024-04-26T23:45:00.548197+0000","last_became_active":"2024-04-26T23:43:02.453319+0000","last_became_peered":"2024-04-26T23:43:02.453319+0000","last_unstale":"2024-04-26T23:45:00.548197+0000","last_undegraded":"2024-04-26T23:45:00.548197+0000","last_fullsized":"2024-04-26T23:45:00.548197+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:23:27.570603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.538674+0000","last_change":"2024-04-26T23:43:59.870417+0000","last_active":"2024-04-26T23:45:00.538674+0000","last_peered":"2024-04-26T23:45:00.538674+0000","last_clean":"2024-04-26T23:45:00.538674+0000","last_became_active":"2024-04-26T23:43:59.870053+0000","last_became_peered":"2024-04-26T23:43:59.870053+0000","last_unstale":"2024-04-26T23:45:00.538674+0000","last_undegraded":"2024-04-26T23:45:00.538674+0000","last_fullsized":"2024-04-26T23:45:00.538674+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:07:31.020611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.541968+0000","last_change":"2024-04-26T23:43:07.546026+0000","last_active":"2024-04-26T23:45:00.541968+0000","last_peered":"2024-04-26T23:45:00.541968+0000","last_clean":"2024-04-26T23:45:00.541968+0000","last_became_active":"2024-04-26T23:43:07.545668+0000","last_became_peered":"2024-04-26T23:43:07.545668+0000","last_unstale":"2024-04-26T23:45:00.541968+0000","last_undegraded":"2024-04-26T23:45:00.541968+0000","last_fullsized":"2024-04-26T23:45:00.541968+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:54:20.580267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.548684+0000","last_change":"2024-04-26T23:43:02.454937+0000","last_active":"2024-04-26T23:45:00.548684+0000","last_peered":"2024-04-26T23:45:00.548684+0000","last_clean":"2024-04-26T23:45:00.548684+0000","last_became_active":"2024-04-26T23:43:02.454585+0000","last_became_peered":"2024-04-26T23:43:02.454585+0000","last_unstale":"2024-04-26T23:45:00.548684+0000","last_undegraded":"2024-04-26T23:45:00.548684+0000","last_fullsized":"2024-04-26T23:45:00.548684+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:55:54.999411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.554062+0000","last_change":"2024-04-26T23:43:59.870821+0000","last_active":"2024-04-26T23:45:00.554062+0000","last_peered":"2024-04-26T23:45:00.554062+0000","last_clean":"2024-04-26T23:45:00.554062+0000","last_became_active":"2024-04-26T23:43:59.870240+0000","last_became_peered":"2024-04-26T23:43:59.870240+0000","last_unstale":"2024-04-26T23:45:00.554062+0000","last_undegraded":"2024-04-26T23:45:00.554062+0000","last_fullsized":"2024-04-26T23:45:00.554062+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:33:00.284232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,6],"acting":[9,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.548674+0000","last_change":"2024-04-26T23:43:07.535682+0000","last_active":"2024-04-26T23:45:00.548674+0000","last_peered":"2024-04-26T23:45:00.548674+0000","last_clean":"2024-04-26T23:45:00.548674+0000","last_became_active":"2024-04-26T23:43:07.535221+0000","last_became_peered":"2024-04-26T23:43:07.535221+0000","last_unstale":"2024-04-26T23:45:00.548674+0000","last_undegraded":"2024-04-26T23:45:00.548674+0000","last_fullsized":"2024-04-26T23:45:00.548674+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:33:54.105329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.538382+0000","last_change":"2024-04-26T23:43:02.443541+0000","last_active":"2024-04-26T23:45:00.538382+0000","last_peered":"2024-04-26T23:45:00.538382+0000","last_clean":"2024-04-26T23:45:00.538382+0000","last_became_active":"2024-04-26T23:43:02.442948+0000","last_became_peered":"2024-04-26T23:43:02.442948+0000","last_unstale":"2024-04-26T23:45:00.538382+0000","last_undegraded":"2024-04-26T23:45:00.538382+0000","last_fullsized":"2024-04-26T23:45:00.538382+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:58:44.595575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.555477+0000","last_change":"2024-04-26T23:43:03.311029+0000","last_active":"2024-04-26T23:45:00.555477+0000","last_peered":"2024-04-26T23:45:00.555477+0000","last_clean":"2024-04-26T23:45:00.555477+0000","last_became_active":"2024-04-26T23:43:03.310722+0000","last_became_peered":"2024-04-26T23:43:03.310722+0000","last_unstale":"2024-04-26T23:45:00.555477+0000","last_undegraded":"2024-04-26T23:45:00.555477+0000","last_fullsized":"2024-04-26T23:45:00.555477+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:07:54.449492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.543870+0000","last_change":"2024-04-26T23:43:07.537391+0000","last_active":"2024-04-26T23:45:00.543870+0000","last_peered":"2024-04-26T23:45:00.543870+0000","last_clean":"2024-04-26T23:45:00.543870+0000","last_became_active":"2024-04-26T23:43:07.536216+0000","last_became_peered":"2024-04-26T23:43:07.536216+0000","last_unstale":"2024-04-26T23:45:00.543870+0000","last_undegraded":"2024-04-26T23:45:00.543870+0000","last_fullsized":"2024-04-26T23:45:00.543870+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:45:55.326283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":137,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:12.358693+0000","last_change":"2024-04-26T23:43:02.451929+0000","last_active":"2024-04-26T23:45:12.358693+0000","last_peered":"2024-04-26T23:45:12.358693+0000","last_clean":"2024-04-26T23:45:12.358693+0000","last_became_active":"2024-04-26T23:43:02.451700+0000","last_became_peered":"2024-04-26T23:43:02.451700+0000","last_unstale":"2024-04-26T23:45:12.358693+0000","last_undegraded":"2024-04-26T23:45:12.358693+0000","last_fullsized":"2024-04-26T23:45:12.358693+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:21:31.279681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":104,"num_read_kb":104,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.543772+0000","last_change":"2024-04-26T23:43:03.295462+0000","last_active":"2024-04-26T23:45:00.543772+0000","last_peered":"2024-04-26T23:45:00.543772+0000","last_clean":"2024-04-26T23:45:00.543772+0000","last_became_active":"2024-04-26T23:43:03.295044+0000","last_became_peered":"2024-04-26T23:43:03.295044+0000","last_unstale":"2024-04-26T23:45:00.543772+0000","last_undegraded":"2024-04-26T23:45:00.543772+0000","last_fullsized":"2024-04-26T23:45:00.543772+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:56:19.445166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.554405+0000","last_change":"2024-04-26T23:43:07.539397+0000","last_active":"2024-04-26T23:45:00.554405+0000","last_peered":"2024-04-26T23:45:00.554405+0000","last_clean":"2024-04-26T23:45:00.554405+0000","last_became_active":"2024-04-26T23:43:07.533081+0000","last_became_peered":"2024-04-26T23:43:07.533081+0000","last_unstale":"2024-04-26T23:45:00.554405+0000","last_undegraded":"2024-04-26T23:45:00.554405+0000","last_fullsized":"2024-04-26T23:45:00.554405+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:24:39.693721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544804+0000","last_change":"2024-04-26T23:43:02.439919+0000","last_active":"2024-04-26T23:45:00.544804+0000","last_peered":"2024-04-26T23:45:00.544804+0000","last_clean":"2024-04-26T23:45:00.544804+0000","last_became_active":"2024-04-26T23:43:02.439561+0000","last_became_peered":"2024-04-26T23:43:02.439561+0000","last_unstale":"2024-04-26T23:45:00.544804+0000","last_undegraded":"2024-04-26T23:45:00.544804+0000","last_fullsized":"2024-04-26T23:45:00.544804+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:30:22.905464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.549199+0000","last_change":"2024-04-26T23:43:03.294088+0000","last_active":"2024-04-26T23:45:00.549199+0000","last_peered":"2024-04-26T23:45:00.549199+0000","last_clean":"2024-04-26T23:45:00.549199+0000","last_became_active":"2024-04-26T23:43:03.293726+0000","last_became_peered":"2024-04-26T23:43:03.293726+0000","last_unstale":"2024-04-26T23:45:00.549199+0000","last_undegraded":"2024-04-26T23:45:00.549199+0000","last_fullsized":"2024-04-26T23:45:00.549199+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:19:42.575193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.541076+0000","last_change":"2024-04-26T23:43:07.543056+0000","last_active":"2024-04-26T23:45:00.541076+0000","last_peered":"2024-04-26T23:45:00.541076+0000","last_clean":"2024-04-26T23:45:00.541076+0000","last_became_active":"2024-04-26T23:43:07.542324+0000","last_became_peered":"2024-04-26T23:43:07.542324+0000","last_unstale":"2024-04-26T23:45:00.541076+0000","last_undegraded":"2024-04-26T23:45:00.541076+0000","last_fullsized":"2024-04-26T23:45:00.541076+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:23:46.415932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.538595+0000","last_change":"2024-04-26T23:43:02.454135+0000","last_active":"2024-04-26T23:45:00.538595+0000","last_peered":"2024-04-26T23:45:00.538595+0000","last_clean":"2024-04-26T23:45:00.538595+0000","last_became_active":"2024-04-26T23:43:02.453798+0000","last_became_peered":"2024-04-26T23:43:02.453798+0000","last_unstale":"2024-04-26T23:45:00.538595+0000","last_undegraded":"2024-04-26T23:45:00.538595+0000","last_fullsized":"2024-04-26T23:45:00.538595+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:16:25.286569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.543474+0000","last_change":"2024-04-26T23:43:03.293520+0000","last_active":"2024-04-26T23:45:00.543474+0000","last_peered":"2024-04-26T23:45:00.543474+0000","last_clean":"2024-04-26T23:45:00.543474+0000","last_became_active":"2024-04-26T23:43:03.292537+0000","last_became_peered":"2024-04-26T23:43:03.292537+0000","last_unstale":"2024-04-26T23:45:00.543474+0000","last_undegraded":"2024-04-26T23:45:00.543474+0000","last_fullsized":"2024-04-26T23:45:00.543474+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:19:28.808217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.555126+0000","last_change":"2024-04-26T23:43:07.528116+0000","last_active":"2024-04-26T23:45:00.555126+0000","last_peered":"2024-04-26T23:45:00.555126+0000","last_clean":"2024-04-26T23:45:00.555126+0000","last_became_active":"2024-04-26T23:43:07.527766+0000","last_became_peered":"2024-04-26T23:43:07.527766+0000","last_unstale":"2024-04-26T23:45:00.555126+0000","last_undegraded":"2024-04-26T23:45:00.555126+0000","last_fullsized":"2024-04-26T23:45:00.555126+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:39:23.949964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'33","reported_seq":278,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:13.372317+0000","last_change":"2024-04-26T23:43:02.437454+0000","last_active":"2024-04-26T23:45:13.372317+0000","last_peered":"2024-04-26T23:45:13.372317+0000","last_clean":"2024-04-26T23:45:13.372317+0000","last_became_active":"2024-04-26T23:43:02.437142+0000","last_became_peered":"2024-04-26T23:43:02.437142+0000","last_unstale":"2024-04-26T23:45:13.372317+0000","last_undegraded":"2024-04-26T23:45:13.372317+0000","last_fullsized":"2024-04-26T23:45:13.372317+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:06:37.751066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54826,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":215,"num_read_kb":160,"num_write":34,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544594+0000","last_change":"2024-04-26T23:43:03.287831+0000","last_active":"2024-04-26T23:45:00.544594+0000","last_peered":"2024-04-26T23:45:00.544594+0000","last_clean":"2024-04-26T23:45:00.544594+0000","last_became_active":"2024-04-26T23:43:03.287479+0000","last_became_peered":"2024-04-26T23:43:03.287479+0000","last_unstale":"2024-04-26T23:45:00.544594+0000","last_undegraded":"2024-04-26T23:45:00.544594+0000","last_fullsized":"2024-04-26T23:45:00.544594+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:39:41.726893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.539031+0000","last_change":"2024-04-26T23:43:07.543985+0000","last_active":"2024-04-26T23:45:00.539031+0000","last_peered":"2024-04-26T23:45:00.539031+0000","last_clean":"2024-04-26T23:45:00.539031+0000","last_became_active":"2024-04-26T23:43:07.543327+0000","last_became_peered":"2024-04-26T23:43:07.543327+0000","last_unstale":"2024-04-26T23:45:00.539031+0000","last_undegraded":"2024-04-26T23:45:00.539031+0000","last_fullsized":"2024-04-26T23:45:00.539031+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:24:10.577822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.543860+0000","last_change":"2024-04-26T23:43:02.447627+0000","last_active":"2024-04-26T23:45:00.543860+0000","last_peered":"2024-04-26T23:45:00.543860+0000","last_clean":"2024-04-26T23:45:00.543860+0000","last_became_active":"2024-04-26T23:43:02.447325+0000","last_became_peered":"2024-04-26T23:43:02.447325+0000","last_unstale":"2024-04-26T23:45:00.543860+0000","last_undegraded":"2024-04-26T23:45:00.543860+0000","last_fullsized":"2024-04-26T23:45:00.543860+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:19:03.757202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.547818+0000","last_change":"2024-04-26T23:43:03.313388+0000","last_active":"2024-04-26T23:45:00.547818+0000","last_peered":"2024-04-26T23:45:00.547818+0000","last_clean":"2024-04-26T23:45:00.547818+0000","last_became_active":"2024-04-26T23:43:03.312899+0000","last_became_peered":"2024-04-26T23:43:03.312899+0000","last_unstale":"2024-04-26T23:45:00.547818+0000","last_undegraded":"2024-04-26T23:45:00.547818+0000","last_fullsized":"2024-04-26T23:45:00.547818+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:58:37.126485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.541551+0000","last_change":"2024-04-26T23:43:07.536706+0000","last_active":"2024-04-26T23:45:00.541551+0000","last_peered":"2024-04-26T23:45:00.541551+0000","last_clean":"2024-04-26T23:45:00.541551+0000","last_became_active":"2024-04-26T23:43:07.536316+0000","last_became_peered":"2024-04-26T23:43:07.536316+0000","last_unstale":"2024-04-26T23:45:00.541551+0000","last_undegraded":"2024-04-26T23:45:00.541551+0000","last_fullsized":"2024-04-26T23:45:00.541551+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:36:46.397725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.538435+0000","last_change":"2024-04-26T23:43:02.448647+0000","last_active":"2024-04-26T23:45:00.538435+0000","last_peered":"2024-04-26T23:45:00.538435+0000","last_clean":"2024-04-26T23:45:00.538435+0000","last_became_active":"2024-04-26T23:43:02.448392+0000","last_became_peered":"2024-04-26T23:43:02.448392+0000","last_unstale":"2024-04-26T23:45:00.538435+0000","last_undegraded":"2024-04-26T23:45:00.538435+0000","last_fullsized":"2024-04-26T23:45:00.538435+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:19:01.367361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544605+0000","last_change":"2024-04-26T23:43:03.302980+0000","last_active":"2024-04-26T23:45:00.544605+0000","last_peered":"2024-04-26T23:45:00.544605+0000","last_clean":"2024-04-26T23:45:00.544605+0000","last_became_active":"2024-04-26T23:43:03.302589+0000","last_became_peered":"2024-04-26T23:43:03.302589+0000","last_unstale":"2024-04-26T23:45:00.544605+0000","last_undegraded":"2024-04-26T23:45:00.544605+0000","last_fullsized":"2024-04-26T23:45:00.544605+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:57:18.641063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.556728+0000","last_change":"2024-04-26T23:43:07.547842+0000","last_active":"2024-04-26T23:45:00.556728+0000","last_peered":"2024-04-26T23:45:00.556728+0000","last_clean":"2024-04-26T23:45:00.556728+0000","last_became_active":"2024-04-26T23:43:07.541774+0000","last_became_peered":"2024-04-26T23:43:07.541774+0000","last_unstale":"2024-04-26T23:45:00.556728+0000","last_undegraded":"2024-04-26T23:45:00.556728+0000","last_fullsized":"2024-04-26T23:45:00.556728+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:25:49.999465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'4","reported_seq":142,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:12.418812+0000","last_change":"2024-04-26T23:43:02.437210+0000","last_active":"2024-04-26T23:45:12.418812+0000","last_peered":"2024-04-26T23:45:12.418812+0000","last_clean":"2024-04-26T23:45:12.418812+0000","last_became_active":"2024-04-26T23:43:02.436886+0000","last_became_peered":"2024-04-26T23:43:02.436886+0000","last_unstale":"2024-04-26T23:45:12.418812+0000","last_undegraded":"2024-04-26T23:45:12.418812+0000","last_fullsized":"2024-04-26T23:45:12.418812+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:07:03.663757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":108,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544319+0000","last_change":"2024-04-26T23:43:59.868608+0000","last_active":"2024-04-26T23:45:00.544319+0000","last_peered":"2024-04-26T23:45:00.544319+0000","last_clean":"2024-04-26T23:45:00.544319+0000","last_became_active":"2024-04-26T23:43:59.868231+0000","last_became_peered":"2024-04-26T23:43:59.868231+0000","last_unstale":"2024-04-26T23:45:00.544319+0000","last_undegraded":"2024-04-26T23:45:00.544319+0000","last_fullsized":"2024-04-26T23:45:00.544319+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:28:42.932733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.559597+0000","last_change":"2024-04-26T23:45:00.559597+0000","last_active":"2024-04-26T23:45:00.559597+0000","last_peered":"2024-04-26T23:45:00.559597+0000","last_clean":"2024-04-26T23:45:00.559597+0000","last_became_active":"2024-04-26T23:45:00.558473+0000","last_became_peered":"2024-04-26T23:45:00.558473+0000","last_unstale":"2024-04-26T23:45:00.559597+0000","last_undegraded":"2024-04-26T23:45:00.559597+0000","last_fullsized":"2024-04-26T23:45:00.559597+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:07:29.856136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":70,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.543200+0000","last_change":"2024-04-26T23:43:59.589239+0000","last_active":"2024-04-26T23:45:00.543200+0000","last_peered":"2024-04-26T23:45:00.543200+0000","last_clean":"2024-04-26T23:45:00.543200+0000","last_became_active":"2024-04-26T23:43:59.576239+0000","last_became_peered":"2024-04-26T23:43:59.576239+0000","last_unstale":"2024-04-26T23:45:00.543200+0000","last_undegraded":"2024-04-26T23:45:00.543200+0000","last_fullsized":"2024-04-26T23:45:00.543200+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:28:53.326961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.538251+0000","last_change":"2024-04-26T23:43:03.290567+0000","last_active":"2024-04-26T23:45:00.538251+0000","last_peered":"2024-04-26T23:45:00.538251+0000","last_clean":"2024-04-26T23:45:00.538251+0000","last_became_active":"2024-04-26T23:43:03.287807+0000","last_became_peered":"2024-04-26T23:43:03.287807+0000","last_unstale":"2024-04-26T23:45:00.538251+0000","last_undegraded":"2024-04-26T23:45:00.538251+0000","last_fullsized":"2024-04-26T23:45:00.538251+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:25:24.393978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.538200+0000","last_change":"2024-04-26T23:43:07.535402+0000","last_active":"2024-04-26T23:45:00.538200+0000","last_peered":"2024-04-26T23:45:00.538200+0000","last_clean":"2024-04-26T23:45:00.538200+0000","last_became_active":"2024-04-26T23:43:07.534870+0000","last_became_peered":"2024-04-26T23:43:07.534870+0000","last_unstale":"2024-04-26T23:45:00.538200+0000","last_undegraded":"2024-04-26T23:45:00.538200+0000","last_fullsized":"2024-04-26T23:45:00.538200+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:26:37.703831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.549038+0000","last_change":"2024-04-26T23:45:00.549038+0000","last_active":"2024-04-26T23:45:00.549038+0000","last_peered":"2024-04-26T23:45:00.549038+0000","last_clean":"2024-04-26T23:45:00.549038+0000","last_became_active":"2024-04-26T23:45:00.543141+0000","last_became_peered":"2024-04-26T23:45:00.543141+0000","last_unstale":"2024-04-26T23:45:00.549038+0000","last_undegraded":"2024-04-26T23:45:00.549038+0000","last_fullsized":"2024-04-26T23:45:00.549038+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:03:27.079264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.541735+0000","last_change":"2024-04-26T23:43:59.576561+0000","last_active":"2024-04-26T23:45:00.541735+0000","last_peered":"2024-04-26T23:45:00.541735+0000","last_clean":"2024-04-26T23:45:00.541735+0000","last_became_active":"2024-04-26T23:43:59.574898+0000","last_became_peered":"2024-04-26T23:43:59.574898+0000","last_unstale":"2024-04-26T23:45:00.541735+0000","last_undegraded":"2024-04-26T23:45:00.541735+0000","last_fullsized":"2024-04-26T23:45:00.541735+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:57:04.569103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.550300+0000","last_change":"2024-04-26T23:45:00.549787+0000","last_active":"2024-04-26T23:45:00.550300+0000","last_peered":"2024-04-26T23:45:00.550300+0000","last_clean":"2024-04-26T23:45:00.550300+0000","last_became_active":"2024-04-26T23:45:00.549316+0000","last_became_peered":"2024-04-26T23:45:00.549316+0000","last_unstale":"2024-04-26T23:45:00.550300+0000","last_undegraded":"2024-04-26T23:45:00.550300+0000","last_fullsized":"2024-04-26T23:45:00.550300+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:21:36.188388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.555587+0000","last_change":"2024-04-26T23:43:02.457971+0000","last_active":"2024-04-26T23:45:00.555587+0000","last_peered":"2024-04-26T23:45:00.555587+0000","last_clean":"2024-04-26T23:45:00.555587+0000","last_became_active":"2024-04-26T23:43:02.457627+0000","last_became_peered":"2024-04-26T23:43:02.457627+0000","last_unstale":"2024-04-26T23:45:00.555587+0000","last_undegraded":"2024-04-26T23:45:00.555587+0000","last_fullsized":"2024-04-26T23:45:00.555587+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:25:51.749774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.548424+0000","last_change":"2024-04-26T23:43:03.305569+0000","last_active":"2024-04-26T23:45:00.548424+0000","last_peered":"2024-04-26T23:45:00.548424+0000","last_clean":"2024-04-26T23:45:00.548424+0000","last_became_active":"2024-04-26T23:43:03.305211+0000","last_became_peered":"2024-04-26T23:43:03.305211+0000","last_unstale":"2024-04-26T23:45:00.548424+0000","last_undegraded":"2024-04-26T23:45:00.548424+0000","last_fullsized":"2024-04-26T23:45:00.548424+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:54:55.847680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.553471+0000","last_change":"2024-04-26T23:43:07.535408+0000","last_active":"2024-04-26T23:45:00.553471+0000","last_peered":"2024-04-26T23:45:00.553471+0000","last_clean":"2024-04-26T23:45:00.553471+0000","last_became_active":"2024-04-26T23:43:07.534957+0000","last_became_peered":"2024-04-26T23:43:07.534957+0000","last_unstale":"2024-04-26T23:45:00.553471+0000","last_undegraded":"2024-04-26T23:45:00.553471+0000","last_fullsized":"2024-04-26T23:45:00.553471+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:35:31.103532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544403+0000","last_change":"2024-04-26T23:43:02.453636+0000","last_active":"2024-04-26T23:45:00.544403+0000","last_peered":"2024-04-26T23:45:00.544403+0000","last_clean":"2024-04-26T23:45:00.544403+0000","last_became_active":"2024-04-26T23:43:02.453310+0000","last_became_peered":"2024-04-26T23:43:02.453310+0000","last_unstale":"2024-04-26T23:45:00.544403+0000","last_undegraded":"2024-04-26T23:45:00.544403+0000","last_fullsized":"2024-04-26T23:45:00.544403+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:42:29.295303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.548563+0000","last_change":"2024-04-26T23:43:03.283184+0000","last_active":"2024-04-26T23:45:00.548563+0000","last_peered":"2024-04-26T23:45:00.548563+0000","last_clean":"2024-04-26T23:45:00.548563+0000","last_became_active":"2024-04-26T23:43:03.282780+0000","last_became_peered":"2024-04-26T23:43:03.282780+0000","last_unstale":"2024-04-26T23:45:00.548563+0000","last_undegraded":"2024-04-26T23:45:00.548563+0000","last_fullsized":"2024-04-26T23:45:00.548563+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:00:13.907101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.555563+0000","last_change":"2024-04-26T23:43:07.859107+0000","last_active":"2024-04-26T23:45:00.555563+0000","last_peered":"2024-04-26T23:45:00.555563+0000","last_clean":"2024-04-26T23:45:00.555563+0000","last_became_active":"2024-04-26T23:43:07.856492+0000","last_became_peered":"2024-04-26T23:43:07.856492+0000","last_unstale":"2024-04-26T23:45:00.555563+0000","last_undegraded":"2024-04-26T23:45:00.555563+0000","last_fullsized":"2024-04-26T23:45:00.555563+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:13:36.570398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.538547+0000","last_change":"2024-04-26T23:43:03.289078+0000","last_active":"2024-04-26T23:45:00.538547+0000","last_peered":"2024-04-26T23:45:00.538547+0000","last_clean":"2024-04-26T23:45:00.538547+0000","last_became_active":"2024-04-26T23:43:03.288729+0000","last_became_peered":"2024-04-26T23:43:03.288729+0000","last_unstale":"2024-04-26T23:45:00.538547+0000","last_undegraded":"2024-04-26T23:45:00.538547+0000","last_fullsized":"2024-04-26T23:45:00.538547+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:32:36.346472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":140,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:12.417976+0000","last_change":"2024-04-26T23:43:02.463355+0000","last_active":"2024-04-26T23:45:12.417976+0000","last_peered":"2024-04-26T23:45:12.417976+0000","last_clean":"2024-04-26T23:45:12.417976+0000","last_became_active":"2024-04-26T23:43:02.455832+0000","last_became_peered":"2024-04-26T23:43:02.455832+0000","last_unstale":"2024-04-26T23:45:12.417976+0000","last_undegraded":"2024-04-26T23:45:12.417976+0000","last_fullsized":"2024-04-26T23:45:12.417976+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:33:33.837999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":104,"num_read_kb":104,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544820+0000","last_change":"2024-04-26T23:43:07.540777+0000","last_active":"2024-04-26T23:45:00.544820+0000","last_peered":"2024-04-26T23:45:00.544820+0000","last_clean":"2024-04-26T23:45:00.544820+0000","last_became_active":"2024-04-26T23:43:07.540389+0000","last_became_peered":"2024-04-26T23:43:07.540389+0000","last_unstale":"2024-04-26T23:45:00.544820+0000","last_undegraded":"2024-04-26T23:45:00.544820+0000","last_fullsized":"2024-04-26T23:45:00.544820+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:57:28.674051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544824+0000","last_change":"2024-04-26T23:43:03.308391+0000","last_active":"2024-04-26T23:45:00.544824+0000","last_peered":"2024-04-26T23:45:00.544824+0000","last_clean":"2024-04-26T23:45:00.544824+0000","last_became_active":"2024-04-26T23:43:03.306762+0000","last_became_peered":"2024-04-26T23:43:03.306762+0000","last_unstale":"2024-04-26T23:45:00.544824+0000","last_undegraded":"2024-04-26T23:45:00.544824+0000","last_fullsized":"2024-04-26T23:45:00.544824+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:34:01.083036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.541054+0000","last_change":"2024-04-26T23:43:02.448037+0000","last_active":"2024-04-26T23:45:00.541054+0000","last_peered":"2024-04-26T23:45:00.541054+0000","last_clean":"2024-04-26T23:45:00.541054+0000","last_became_active":"2024-04-26T23:43:02.447706+0000","last_became_peered":"2024-04-26T23:43:02.447706+0000","last_unstale":"2024-04-26T23:45:00.541054+0000","last_undegraded":"2024-04-26T23:45:00.541054+0000","last_fullsized":"2024-04-26T23:45:00.541054+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:43:56.295191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":352,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544151+0000","last_change":"2024-04-26T23:41:42.266448+0000","last_active":"2024-04-26T23:45:00.544151+0000","last_peered":"2024-04-26T23:45:00.544151+0000","last_clean":"2024-04-26T23:45:00.544151+0000","last_became_active":"2024-04-26T23:41:42.265898+0000","last_became_peered":"2024-04-26T23:41:42.265898+0000","last_unstale":"2024-04-26T23:45:00.544151+0000","last_undegraded":"2024-04-26T23:45:00.544151+0000","last_fullsized":"2024-04-26T23:45:00.544151+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:37:44.460409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:37:44.460409+0000","last_clean_scrub_stamp":"2024-04-26T23:37:44.460409+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:40:48.177191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.557115+0000","last_change":"2024-04-26T23:45:00.557115+0000","last_active":"2024-04-26T23:45:00.557115+0000","last_peered":"2024-04-26T23:45:00.557115+0000","last_clean":"2024-04-26T23:45:00.557115+0000","last_became_active":"2024-04-26T23:45:00.556698+0000","last_became_peered":"2024-04-26T23:45:00.556698+0000","last_unstale":"2024-04-26T23:45:00.557115+0000","last_undegraded":"2024-04-26T23:45:00.557115+0000","last_fullsized":"2024-04-26T23:45:00.557115+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:23:11.253779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.548195+0000","last_change":"2024-04-26T23:43:03.301989+0000","last_active":"2024-04-26T23:45:00.548195+0000","last_peered":"2024-04-26T23:45:00.548195+0000","last_clean":"2024-04-26T23:45:00.548195+0000","last_became_active":"2024-04-26T23:43:03.301179+0000","last_became_peered":"2024-04-26T23:43:03.301179+0000","last_unstale":"2024-04-26T23:45:00.548195+0000","last_undegraded":"2024-04-26T23:45:00.548195+0000","last_fullsized":"2024-04-26T23:45:00.548195+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:29:33.012009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544372+0000","last_change":"2024-04-26T23:43:02.439648+0000","last_active":"2024-04-26T23:45:00.544372+0000","last_peered":"2024-04-26T23:45:00.544372+0000","last_clean":"2024-04-26T23:45:00.544372+0000","last_became_active":"2024-04-26T23:43:02.439340+0000","last_became_peered":"2024-04-26T23:43:02.439340+0000","last_unstale":"2024-04-26T23:45:00.544372+0000","last_undegraded":"2024-04-26T23:45:00.544372+0000","last_fullsized":"2024-04-26T23:45:00.544372+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:10:01.096657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.541352+0000","last_change":"2024-04-26T23:43:07.534317+0000","last_active":"2024-04-26T23:45:00.541352+0000","last_peered":"2024-04-26T23:45:00.541352+0000","last_clean":"2024-04-26T23:45:00.541352+0000","last_became_active":"2024-04-26T23:43:07.533808+0000","last_became_peered":"2024-04-26T23:43:07.533808+0000","last_unstale":"2024-04-26T23:45:00.541352+0000","last_undegraded":"2024-04-26T23:45:00.541352+0000","last_fullsized":"2024-04-26T23:45:00.541352+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:54:29.952089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.545460+0000","last_change":"2024-04-26T23:43:02.457501+0000","last_active":"2024-04-26T23:45:00.545460+0000","last_peered":"2024-04-26T23:45:00.545460+0000","last_clean":"2024-04-26T23:45:00.545460+0000","last_became_active":"2024-04-26T23:43:02.457024+0000","last_became_peered":"2024-04-26T23:43:02.457024+0000","last_unstale":"2024-04-26T23:45:00.545460+0000","last_undegraded":"2024-04-26T23:45:00.545460+0000","last_fullsized":"2024-04-26T23:45:00.545460+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:21:58.542186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.553642+0000","last_change":"2024-04-26T23:43:03.303434+0000","last_active":"2024-04-26T23:45:00.553642+0000","last_peered":"2024-04-26T23:45:00.553642+0000","last_clean":"2024-04-26T23:45:00.553642+0000","last_became_active":"2024-04-26T23:43:03.302455+0000","last_became_peered":"2024-04-26T23:43:03.302455+0000","last_unstale":"2024-04-26T23:45:00.553642+0000","last_undegraded":"2024-04-26T23:45:00.553642+0000","last_fullsized":"2024-04-26T23:45:00.553642+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:57:05.530635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.542546+0000","last_change":"2024-04-26T23:43:07.856932+0000","last_active":"2024-04-26T23:45:00.542546+0000","last_peered":"2024-04-26T23:45:00.542546+0000","last_clean":"2024-04-26T23:45:00.542546+0000","last_became_active":"2024-04-26T23:43:07.855699+0000","last_became_peered":"2024-04-26T23:43:07.855699+0000","last_unstale":"2024-04-26T23:45:00.542546+0000","last_undegraded":"2024-04-26T23:45:00.542546+0000","last_fullsized":"2024-04-26T23:45:00.542546+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:20:28.994301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'8","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.557245+0000","last_change":"2024-04-26T23:43:02.449852+0000","last_active":"2024-04-26T23:45:00.557245+0000","last_peered":"2024-04-26T23:45:00.557245+0000","last_clean":"2024-04-26T23:45:00.557245+0000","last_became_active":"2024-04-26T23:43:02.449422+0000","last_became_peered":"2024-04-26T23:43:02.449422+0000","last_unstale":"2024-04-26T23:45:00.557245+0000","last_undegraded":"2024-04-26T23:45:00.557245+0000","last_fullsized":"2024-04-26T23:45:00.557245+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:07:25.453872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.551799+0000","last_change":"2024-04-26T23:43:03.304733+0000","last_active":"2024-04-26T23:45:00.551799+0000","last_peered":"2024-04-26T23:45:00.551799+0000","last_clean":"2024-04-26T23:45:00.551799+0000","last_became_active":"2024-04-26T23:43:03.304383+0000","last_became_peered":"2024-04-26T23:43:03.304383+0000","last_unstale":"2024-04-26T23:45:00.551799+0000","last_undegraded":"2024-04-26T23:45:00.551799+0000","last_fullsized":"2024-04-26T23:45:00.551799+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:49:28.679255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.543687+0000","last_change":"2024-04-26T23:43:07.525310+0000","last_active":"2024-04-26T23:45:00.543687+0000","last_peered":"2024-04-26T23:45:00.543687+0000","last_clean":"2024-04-26T23:45:00.543687+0000","last_became_active":"2024-04-26T23:43:07.524502+0000","last_became_peered":"2024-04-26T23:43:07.524502+0000","last_unstale":"2024-04-26T23:45:00.543687+0000","last_undegraded":"2024-04-26T23:45:00.543687+0000","last_fullsized":"2024-04-26T23:45:00.543687+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:07:45.927837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.541434+0000","last_change":"2024-04-26T23:43:02.446238+0000","last_active":"2024-04-26T23:45:00.541434+0000","last_peered":"2024-04-26T23:45:00.541434+0000","last_clean":"2024-04-26T23:45:00.541434+0000","last_became_active":"2024-04-26T23:43:02.445959+0000","last_became_peered":"2024-04-26T23:43:02.445959+0000","last_unstale":"2024-04-26T23:45:00.541434+0000","last_undegraded":"2024-04-26T23:45:00.541434+0000","last_fullsized":"2024-04-26T23:45:00.541434+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:25:10.007481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.548968+0000","last_change":"2024-04-26T23:43:03.306035+0000","last_active":"2024-04-26T23:45:00.548968+0000","last_peered":"2024-04-26T23:45:00.548968+0000","last_clean":"2024-04-26T23:45:00.548968+0000","last_became_active":"2024-04-26T23:43:03.305604+0000","last_became_peered":"2024-04-26T23:43:03.305604+0000","last_unstale":"2024-04-26T23:45:00.548968+0000","last_undegraded":"2024-04-26T23:45:00.548968+0000","last_fullsized":"2024-04-26T23:45:00.548968+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:02:38.567754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.557767+0000","last_change":"2024-04-26T23:45:00.557767+0000","last_active":"2024-04-26T23:45:00.557767+0000","last_peered":"2024-04-26T23:45:00.557767+0000","last_clean":"2024-04-26T23:45:00.557767+0000","last_became_active":"2024-04-26T23:45:00.557207+0000","last_became_peered":"2024-04-26T23:45:00.557207+0000","last_unstale":"2024-04-26T23:45:00.557767+0000","last_undegraded":"2024-04-26T23:45:00.557767+0000","last_fullsized":"2024-04-26T23:45:00.557767+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:18:06.396984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,10,9],"acting":[4,8,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.543066+0000","last_change":"2024-04-26T23:43:02.443464+0000","last_active":"2024-04-26T23:45:00.543066+0000","last_peered":"2024-04-26T23:45:00.543066+0000","last_clean":"2024-04-26T23:45:00.543066+0000","last_became_active":"2024-04-26T23:43:02.443170+0000","last_became_peered":"2024-04-26T23:43:02.443170+0000","last_unstale":"2024-04-26T23:45:00.543066+0000","last_undegraded":"2024-04-26T23:45:00.543066+0000","last_fullsized":"2024-04-26T23:45:00.543066+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:45:51.431082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544741+0000","last_change":"2024-04-26T23:43:03.282714+0000","last_active":"2024-04-26T23:45:00.544741+0000","last_peered":"2024-04-26T23:45:00.544741+0000","last_clean":"2024-04-26T23:45:00.544741+0000","last_became_active":"2024-04-26T23:43:03.282353+0000","last_became_peered":"2024-04-26T23:43:03.282353+0000","last_unstale":"2024-04-26T23:45:00.544741+0000","last_undegraded":"2024-04-26T23:45:00.544741+0000","last_fullsized":"2024-04-26T23:45:00.544741+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T10:18:24.082946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.556962+0000","last_change":"2024-04-26T23:43:07.543301+0000","last_active":"2024-04-26T23:45:00.556962+0000","last_peered":"2024-04-26T23:45:00.556962+0000","last_clean":"2024-04-26T23:45:00.556962+0000","last_became_active":"2024-04-26T23:43:07.540434+0000","last_became_peered":"2024-04-26T23:43:07.540434+0000","last_unstale":"2024-04-26T23:45:00.556962+0000","last_undegraded":"2024-04-26T23:45:00.556962+0000","last_fullsized":"2024-04-26T23:45:00.556962+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T11:15:18.542595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.543478+0000","last_change":"2024-04-26T23:43:02.451391+0000","last_active":"2024-04-26T23:45:00.543478+0000","last_peered":"2024-04-26T23:45:00.543478+0000","last_clean":"2024-04-26T23:45:00.543478+0000","last_became_active":"2024-04-26T23:43:02.450735+0000","last_became_peered":"2024-04-26T23:43:02.450735+0000","last_unstale":"2024-04-26T23:45:00.543478+0000","last_undegraded":"2024-04-26T23:45:00.543478+0000","last_fullsized":"2024-04-26T23:45:00.543478+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:57:48.658671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.541441+0000","last_change":"2024-04-26T23:43:03.301858+0000","last_active":"2024-04-26T23:45:00.541441+0000","last_peered":"2024-04-26T23:45:00.541441+0000","last_clean":"2024-04-26T23:45:00.541441+0000","last_became_active":"2024-04-26T23:43:03.301474+0000","last_became_peered":"2024-04-26T23:43:03.301474+0000","last_unstale":"2024-04-26T23:45:00.541441+0000","last_undegraded":"2024-04-26T23:45:00.541441+0000","last_fullsized":"2024-04-26T23:45:00.541441+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:24:56.386127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.541095+0000","last_change":"2024-04-26T23:43:07.544512+0000","last_active":"2024-04-26T23:45:00.541095+0000","last_peered":"2024-04-26T23:45:00.541095+0000","last_clean":"2024-04-26T23:45:00.541095+0000","last_became_active":"2024-04-26T23:43:07.544029+0000","last_became_peered":"2024-04-26T23:43:07.544029+0000","last_unstale":"2024-04-26T23:45:00.541095+0000","last_undegraded":"2024-04-26T23:45:00.541095+0000","last_fullsized":"2024-04-26T23:45:00.541095+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:38:08.149123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":19,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.541148+0000","last_change":"2024-04-26T23:43:59.579721+0000","last_active":"2024-04-26T23:45:00.541148+0000","last_peered":"2024-04-26T23:45:00.541148+0000","last_clean":"2024-04-26T23:45:00.541148+0000","last_became_active":"2024-04-26T23:43:59.576675+0000","last_became_peered":"2024-04-26T23:43:59.576675+0000","last_unstale":"2024-04-26T23:45:00.541148+0000","last_undegraded":"2024-04-26T23:45:00.541148+0000","last_fullsized":"2024-04-26T23:45:00.541148+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:59:30.124137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.549817+0000","last_change":"2024-04-26T23:43:03.307895+0000","last_active":"2024-04-26T23:45:00.549817+0000","last_peered":"2024-04-26T23:45:00.549817+0000","last_clean":"2024-04-26T23:45:00.549817+0000","last_became_active":"2024-04-26T23:43:03.307478+0000","last_became_peered":"2024-04-26T23:43:03.307478+0000","last_unstale":"2024-04-26T23:45:00.549817+0000","last_undegraded":"2024-04-26T23:45:00.549817+0000","last_fullsized":"2024-04-26T23:45:00.549817+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:57:14.356234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.541058+0000","last_change":"2024-04-26T23:43:07.855803+0000","last_active":"2024-04-26T23:45:00.541058+0000","last_peered":"2024-04-26T23:45:00.541058+0000","last_clean":"2024-04-26T23:45:00.541058+0000","last_became_active":"2024-04-26T23:43:07.855200+0000","last_became_peered":"2024-04-26T23:43:07.855200+0000","last_unstale":"2024-04-26T23:45:00.541058+0000","last_undegraded":"2024-04-26T23:45:00.541058+0000","last_fullsized":"2024-04-26T23:45:00.541058+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:02:46.223330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.543821+0000","last_change":"2024-04-26T23:43:02.437513+0000","last_active":"2024-04-26T23:45:00.543821+0000","last_peered":"2024-04-26T23:45:00.543821+0000","last_clean":"2024-04-26T23:45:00.543821+0000","last_became_active":"2024-04-26T23:43:02.437153+0000","last_became_peered":"2024-04-26T23:43:02.437153+0000","last_unstale":"2024-04-26T23:45:00.543821+0000","last_undegraded":"2024-04-26T23:45:00.543821+0000","last_fullsized":"2024-04-26T23:45:00.543821+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:11:45.477579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544529+0000","last_change":"2024-04-26T23:43:03.297835+0000","last_active":"2024-04-26T23:45:00.544529+0000","last_peered":"2024-04-26T23:45:00.544529+0000","last_clean":"2024-04-26T23:45:00.544529+0000","last_became_active":"2024-04-26T23:43:03.297481+0000","last_became_peered":"2024-04-26T23:43:03.297481+0000","last_unstale":"2024-04-26T23:45:00.544529+0000","last_undegraded":"2024-04-26T23:45:00.544529+0000","last_fullsized":"2024-04-26T23:45:00.544529+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:52:54.757461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.543077+0000","last_change":"2024-04-26T23:43:07.536849+0000","last_active":"2024-04-26T23:45:00.543077+0000","last_peered":"2024-04-26T23:45:00.543077+0000","last_clean":"2024-04-26T23:45:00.543077+0000","last_became_active":"2024-04-26T23:43:07.536214+0000","last_became_peered":"2024-04-26T23:43:07.536214+0000","last_unstale":"2024-04-26T23:45:00.543077+0000","last_undegraded":"2024-04-26T23:45:00.543077+0000","last_fullsized":"2024-04-26T23:45:00.543077+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:33:16.782368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544263+0000","last_change":"2024-04-26T23:43:02.464181+0000","last_active":"2024-04-26T23:45:00.544263+0000","last_peered":"2024-04-26T23:45:00.544263+0000","last_clean":"2024-04-26T23:45:00.544263+0000","last_became_active":"2024-04-26T23:43:02.463830+0000","last_became_peered":"2024-04-26T23:43:02.463830+0000","last_unstale":"2024-04-26T23:45:00.544263+0000","last_undegraded":"2024-04-26T23:45:00.544263+0000","last_fullsized":"2024-04-26T23:45:00.544263+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:14:52.831302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544954+0000","last_change":"2024-04-26T23:43:03.298497+0000","last_active":"2024-04-26T23:45:00.544954+0000","last_peered":"2024-04-26T23:45:00.544954+0000","last_clean":"2024-04-26T23:45:00.544954+0000","last_became_active":"2024-04-26T23:43:03.298194+0000","last_became_peered":"2024-04-26T23:43:03.298194+0000","last_unstale":"2024-04-26T23:45:00.544954+0000","last_undegraded":"2024-04-26T23:45:00.544954+0000","last_fullsized":"2024-04-26T23:45:00.544954+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:39:19.007294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544441+0000","last_change":"2024-04-26T23:43:07.527962+0000","last_active":"2024-04-26T23:45:00.544441+0000","last_peered":"2024-04-26T23:45:00.544441+0000","last_clean":"2024-04-26T23:45:00.544441+0000","last_became_active":"2024-04-26T23:43:07.527386+0000","last_became_peered":"2024-04-26T23:43:07.527386+0000","last_unstale":"2024-04-26T23:45:00.544441+0000","last_undegraded":"2024-04-26T23:45:00.544441+0000","last_fullsized":"2024-04-26T23:45:00.544441+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:57:21.711670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.538734+0000","last_change":"2024-04-26T23:43:02.453988+0000","last_active":"2024-04-26T23:45:00.538734+0000","last_peered":"2024-04-26T23:45:00.538734+0000","last_clean":"2024-04-26T23:45:00.538734+0000","last_became_active":"2024-04-26T23:43:02.453654+0000","last_became_peered":"2024-04-26T23:43:02.453654+0000","last_unstale":"2024-04-26T23:45:00.538734+0000","last_undegraded":"2024-04-26T23:45:00.538734+0000","last_fullsized":"2024-04-26T23:45:00.538734+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:46:08.718705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544559+0000","last_change":"2024-04-26T23:43:03.305861+0000","last_active":"2024-04-26T23:45:00.544559+0000","last_peered":"2024-04-26T23:45:00.544559+0000","last_clean":"2024-04-26T23:45:00.544559+0000","last_became_active":"2024-04-26T23:43:03.305301+0000","last_became_peered":"2024-04-26T23:43:03.305301+0000","last_unstale":"2024-04-26T23:45:00.544559+0000","last_undegraded":"2024-04-26T23:45:00.544559+0000","last_fullsized":"2024-04-26T23:45:00.544559+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:08:24.174443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.537998+0000","last_change":"2024-04-26T23:43:07.529914+0000","last_active":"2024-04-26T23:45:00.537998+0000","last_peered":"2024-04-26T23:45:00.537998+0000","last_clean":"2024-04-26T23:45:00.537998+0000","last_became_active":"2024-04-26T23:43:07.529509+0000","last_became_peered":"2024-04-26T23:43:07.529509+0000","last_unstale":"2024-04-26T23:45:00.537998+0000","last_undegraded":"2024-04-26T23:45:00.537998+0000","last_fullsized":"2024-04-26T23:45:00.537998+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:06.488849+0000","last_clean_scrub_stamp":"2024-04-26T23:43:06.488849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:50:23.293672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.552220+0000","last_change":"2024-04-26T23:43:02.452096+0000","last_active":"2024-04-26T23:45:00.552220+0000","last_peered":"2024-04-26T23:45:00.552220+0000","last_clean":"2024-04-26T23:45:00.552220+0000","last_became_active":"2024-04-26T23:43:02.450039+0000","last_became_peered":"2024-04-26T23:43:02.450039+0000","last_unstale":"2024-04-26T23:45:00.552220+0000","last_undegraded":"2024-04-26T23:45:00.552220+0000","last_fullsized":"2024-04-26T23:45:00.552220+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:01.410793+0000","last_clean_scrub_stamp":"2024-04-26T23:43:01.410793+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:47:03.062575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.545023+0000","last_change":"2024-04-26T23:43:03.294790+0000","last_active":"2024-04-26T23:45:00.545023+0000","last_peered":"2024-04-26T23:45:00.545023+0000","last_clean":"2024-04-26T23:45:00.545023+0000","last_became_active":"2024-04-26T23:43:03.294460+0000","last_became_peered":"2024-04-26T23:43:03.294460+0000","last_unstale":"2024-04-26T23:45:00.545023+0000","last_undegraded":"2024-04-26T23:45:00.545023+0000","last_fullsized":"2024-04-26T23:45:00.545023+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:57:40.879315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.541289+0000","last_change":"2024-04-26T23:43:03.284344+0000","last_active":"2024-04-26T23:45:00.541289+0000","last_peered":"2024-04-26T23:45:00.541289+0000","last_clean":"2024-04-26T23:45:00.541289+0000","last_became_active":"2024-04-26T23:43:03.283954+0000","last_became_peered":"2024-04-26T23:43:03.283954+0000","last_unstale":"2024-04-26T23:45:00.541289+0000","last_undegraded":"2024-04-26T23:45:00.541289+0000","last_fullsized":"2024-04-26T23:45:00.541289+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:12:56.516067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544580+0000","last_change":"2024-04-26T23:43:59.574959+0000","last_active":"2024-04-26T23:45:00.544580+0000","last_peered":"2024-04-26T23:45:00.544580+0000","last_clean":"2024-04-26T23:45:00.544580+0000","last_became_active":"2024-04-26T23:43:59.574588+0000","last_became_peered":"2024-04-26T23:43:59.574588+0000","last_unstale":"2024-04-26T23:45:00.544580+0000","last_undegraded":"2024-04-26T23:45:00.544580+0000","last_fullsized":"2024-04-26T23:45:00.544580+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:52:56.689439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.543637+0000","last_change":"2024-04-26T23:43:59.574946+0000","last_active":"2024-04-26T23:45:00.543637+0000","last_peered":"2024-04-26T23:45:00.543637+0000","last_clean":"2024-04-26T23:45:00.543637+0000","last_became_active":"2024-04-26T23:43:59.574238+0000","last_became_peered":"2024-04-26T23:43:59.574238+0000","last_unstale":"2024-04-26T23:45:00.543637+0000","last_undegraded":"2024-04-26T23:45:00.543637+0000","last_fullsized":"2024-04-26T23:45:00.543637+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:58:41.619868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,0],"acting":[2,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.545984+0000","last_change":"2024-04-26T23:43:03.304580+0000","last_active":"2024-04-26T23:45:00.545984+0000","last_peered":"2024-04-26T23:45:00.545984+0000","last_clean":"2024-04-26T23:45:00.545984+0000","last_became_active":"2024-04-26T23:43:03.303128+0000","last_became_peered":"2024-04-26T23:43:03.303128+0000","last_unstale":"2024-04-26T23:45:00.545984+0000","last_undegraded":"2024-04-26T23:45:00.545984+0000","last_fullsized":"2024-04-26T23:45:00.545984+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:11:36.868298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.553794+0000","last_change":"2024-04-26T23:43:03.306441+0000","last_active":"2024-04-26T23:45:00.553794+0000","last_peered":"2024-04-26T23:45:00.553794+0000","last_clean":"2024-04-26T23:45:00.553794+0000","last_became_active":"2024-04-26T23:43:03.305209+0000","last_became_peered":"2024-04-26T23:43:03.305209+0000","last_unstale":"2024-04-26T23:45:00.553794+0000","last_undegraded":"2024-04-26T23:45:00.553794+0000","last_fullsized":"2024-04-26T23:45:00.553794+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:41:39.757001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.541812+0000","last_change":"2024-04-26T23:43:03.284291+0000","last_active":"2024-04-26T23:45:00.541812+0000","last_peered":"2024-04-26T23:45:00.541812+0000","last_clean":"2024-04-26T23:45:00.541812+0000","last_became_active":"2024-04-26T23:43:03.284006+0000","last_became_peered":"2024-04-26T23:43:03.284006+0000","last_unstale":"2024-04-26T23:45:00.541812+0000","last_undegraded":"2024-04-26T23:45:00.541812+0000","last_fullsized":"2024-04-26T23:45:00.541812+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:24:01.386322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.544332+0000","last_change":"2024-04-26T23:43:03.286478+0000","last_active":"2024-04-26T23:45:00.544332+0000","last_peered":"2024-04-26T23:45:00.544332+0000","last_clean":"2024-04-26T23:45:00.544332+0000","last_became_active":"2024-04-26T23:43:03.286115+0000","last_became_peered":"2024-04-26T23:43:03.286115+0000","last_unstale":"2024-04-26T23:45:00.544332+0000","last_undegraded":"2024-04-26T23:45:00.544332+0000","last_fullsized":"2024-04-26T23:45:00.544332+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:34:57.534124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.552259+0000","last_change":"2024-04-26T23:43:59.872042+0000","last_active":"2024-04-26T23:45:00.552259+0000","last_peered":"2024-04-26T23:45:00.552259+0000","last_clean":"2024-04-26T23:45:00.552259+0000","last_became_active":"2024-04-26T23:43:59.870288+0000","last_became_peered":"2024-04-26T23:43:59.870288+0000","last_unstale":"2024-04-26T23:45:00.552259+0000","last_undegraded":"2024-04-26T23:45:00.552259+0000","last_fullsized":"2024-04-26T23:45:00.552259+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:16:38.518370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.538313+0000","last_change":"2024-04-26T23:43:03.300602+0000","last_active":"2024-04-26T23:45:00.538313+0000","last_peered":"2024-04-26T23:45:00.538313+0000","last_clean":"2024-04-26T23:45:00.538313+0000","last_became_active":"2024-04-26T23:43:03.300225+0000","last_became_peered":"2024-04-26T23:43:03.300225+0000","last_unstale":"2024-04-26T23:45:00.538313+0000","last_undegraded":"2024-04-26T23:45:00.538313+0000","last_fullsized":"2024-04-26T23:45:00.538313+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:52:36.638153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.543535+0000","last_change":"2024-04-26T23:43:03.305631+0000","last_active":"2024-04-26T23:45:00.543535+0000","last_peered":"2024-04-26T23:45:00.543535+0000","last_clean":"2024-04-26T23:45:00.543535+0000","last_became_active":"2024-04-26T23:43:03.300193+0000","last_became_peered":"2024-04-26T23:43:03.300193+0000","last_unstale":"2024-04-26T23:45:00.543535+0000","last_undegraded":"2024-04-26T23:45:00.543535+0000","last_fullsized":"2024-04-26T23:45:00.543535+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:27:08.355753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.543271+0000","last_change":"2024-04-26T23:43:03.309514+0000","last_active":"2024-04-26T23:45:00.543271+0000","last_peered":"2024-04-26T23:45:00.543271+0000","last_clean":"2024-04-26T23:45:00.543271+0000","last_became_active":"2024-04-26T23:43:03.309085+0000","last_became_peered":"2024-04-26T23:43:03.309085+0000","last_unstale":"2024-04-26T23:45:00.543271+0000","last_undegraded":"2024-04-26T23:45:00.543271+0000","last_fullsized":"2024-04-26T23:45:00.543271+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:08:45.649513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.547848+0000","last_change":"2024-04-26T23:43:03.303553+0000","last_active":"2024-04-26T23:45:00.547848+0000","last_peered":"2024-04-26T23:45:00.547848+0000","last_clean":"2024-04-26T23:45:00.547848+0000","last_became_active":"2024-04-26T23:43:03.302979+0000","last_became_peered":"2024-04-26T23:43:03.302979+0000","last_unstale":"2024-04-26T23:45:00.547848+0000","last_undegraded":"2024-04-26T23:45:00.547848+0000","last_fullsized":"2024-04-26T23:45:00.547848+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:44:25.809138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T23:45:00.541360+0000","last_change":"2024-04-26T23:43:03.288433+0000","last_active":"2024-04-26T23:45:00.541360+0000","last_peered":"2024-04-26T23:45:00.541360+0000","last_clean":"2024-04-26T23:45:00.541360+0000","last_became_active":"2024-04-26T23:43:03.288125+0000","last_became_peered":"2024-04-26T23:43:03.288125+0000","last_unstale":"2024-04-26T23:45:00.541360+0000","last_undegraded":"2024-04-26T23:45:00.541360+0000","last_fullsized":"2024-04-26T23:45:00.541360+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T23:43:02.419736+0000","last_clean_scrub_stamp":"2024-04-26T23:43:02.419736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:53:21.673755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":282161,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":59218,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":650,"num_read_kb":387,"num_write":164,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":134,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":177654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141,"ondisk_log_size":141,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383722,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1024000,"data_stored":519912,"data_compressed":26528,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645358,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32316,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712836,"statfs":{"total":95995035648,"available":95961944064,"internally_reserved":0,"allocated":1159168,"data_stored":1162375,"data_compressed":62415,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808884,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1040384,"data_stored":524222,"data_compressed":26528,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070522,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1032192,"data_stored":520024,"data_compressed":26528,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332160,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1085440,"data_stored":1116346,"data_compressed":62415,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495685,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1015808,"data_stored":515302,"data_compressed":26528,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574946,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1077248,"data_stored":570038,"data_compressed":26528,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738470,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":516690,"data_compressed":26528,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411421,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1093632,"data_stored":571667,"data_compressed":26528,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182486,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1028096,"data_stored":515496,"data_compressed":26528,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920849,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1036288,"data_stored":520455,"data_compressed":26528,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757324,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32248,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712904,"statfs":{"total":95995035648,"available":95962013696,"internally_reserved":0,"allocated":1093632,"data_stored":1111543,"data_compressed":62415,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34242,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34242,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34242,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35887,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35887,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35887,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35887,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35887,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":54826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":56455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":56795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T23:45:21.016 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-26T23:45:21.016 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-26T23:45:21.016 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-26T23:45:21.016 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-26T23:45:21.487 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:45:21.815 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:21 smithi072 ceph-mon[28184]: from='client.25054 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T23:45:21.815 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:21 smithi072 ceph-mon[28184]: pgmap v394: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:45:21.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:21 smithi076 ceph-mon[32774]: from='client.25054 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T23:45:21.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:21 smithi076 ceph-mon[32774]: pgmap v394: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:45:22.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:21 smithi136 ceph-mon[33050]: from='client.25054 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T23:45:22.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:21 smithi136 ceph-mon[33050]: pgmap v394: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:45:23.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.679+0000 7fe20a000700 1 -- 172.21.15.72:0/2919637797 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe204078270 msgr2=0x7fe204113d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:23.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.679+0000 7fe20a000700 1 --2- 172.21.15.72:0/2919637797 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe204078270 0x7fe204113d70 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec009990 tx=0x7fe1ec009ca0 comp rx=0 tx=0).stop 2024-04-26T23:45:23.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.679+0000 7fe20a000700 1 -- 172.21.15.72:0/2919637797 shutdown_connections 2024-04-26T23:45:23.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.679+0000 7fe20a000700 1 --2- 172.21.15.72:0/2919637797 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2041142b0 0x7fe2041166a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:23.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.679+0000 7fe20a000700 1 --2- 172.21.15.72:0/2919637797 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe204077950 0x7fe204077d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:23.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.679+0000 7fe20a000700 1 --2- 172.21.15.72:0/2919637797 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe204078270 0x7fe204113d70 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:23.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.679+0000 7fe20a000700 1 -- 172.21.15.72:0/2919637797 >> 172.21.15.72:0/2919637797 conn(0x7fe204101100 msgr2=0x7fe204103520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:23.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.679+0000 7fe20a000700 1 -- 172.21.15.72:0/2919637797 shutdown_connections 2024-04-26T23:45:23.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.679+0000 7fe20a000700 1 -- 172.21.15.72:0/2919637797 wait complete. 2024-04-26T23:45:23.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.680+0000 7fe20a000700 1 Processor -- start 2024-04-26T23:45:23.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.680+0000 7fe20a000700 1 -- start start 2024-04-26T23:45:23.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.681+0000 7fe20a000700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe204077950 0x7fe2041a0100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:23.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.681+0000 7fe20a000700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe204078270 0x7fe2041a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:23.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.681+0000 7fe20a000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe2041142b0 0x7fe2041a49d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:23.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.681+0000 7fe20a000700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fe204119c70 con 0x7fe204078270 2024-04-26T23:45:23.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.681+0000 7fe20a000700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe204119af0 con 0x7fe2041142b0 2024-04-26T23:45:23.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.681+0000 7fe20a000700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fe204119df0 con 0x7fe204077950 2024-04-26T23:45:23.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.681+0000 7fe1faa00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe204078270 0x7fe2041a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:23.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.681+0000 7fe1faa00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe204078270 0x7fe2041a0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37150/0 (socket says 172.21.15.72:37150) 2024-04-26T23:45:23.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.681+0000 7fe1faa00700 1 -- 172.21.15.72:0/489021016 learned_addr learned my addr 172.21.15.72:0/489021016 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:45:23.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.681+0000 7fe203400700 1 --2- 172.21.15.72:0/489021016 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe204077950 0x7fe2041a0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:23.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.681+0000 7fe203e00700 1 --2- 172.21.15.72:0/489021016 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe2041142b0 0x7fe2041a49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:23.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.682+0000 7fe203400700 1 -- 172.21.15.72:0/489021016 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe2041142b0 msgr2=0x7fe2041a49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:23.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.682+0000 7fe203400700 1 --2- 172.21.15.72:0/489021016 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe2041142b0 0x7fe2041a49d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:23.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.682+0000 7fe203400700 1 -- 172.21.15.72:0/489021016 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe204078270 msgr2=0x7fe2041a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:23.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.682+0000 7fe203400700 1 --2- 172.21.15.72:0/489021016 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe204078270 0x7fe2041a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:23.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.682+0000 7fe203400700 1 -- 172.21.15.72:0/489021016 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2041a5110 con 0x7fe204077950 2024-04-26T23:45:23.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.682+0000 7fe1faa00700 1 --2- 172.21.15.72:0/489021016 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe204078270 0x7fe2041a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:45:23.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.682+0000 7fe203e00700 1 --2- 172.21.15.72:0/489021016 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe2041142b0 0x7fe2041a49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:45:23.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.682+0000 7fe203400700 1 --2- 172.21.15.72:0/489021016 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe204077950 0x7fe2041a0100 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe1f400ef10 tx=0x7fe1f400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:23.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.682+0000 7fe200c00700 1 -- 172.21.15.72:0/489021016 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f400c9c0 con 0x7fe204077950 2024-04-26T23:45:23.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.682+0000 7fe200c00700 1 -- 172.21.15.72:0/489021016 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1f4019070 con 0x7fe204077950 2024-04-26T23:45:23.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.682+0000 7fe20a000700 1 -- 172.21.15.72:0/489021016 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2041a53a0 con 0x7fe204077950 2024-04-26T23:45:23.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.682+0000 7fe200c00700 1 -- 172.21.15.72:0/489021016 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f4014480 con 0x7fe204077950 2024-04-26T23:45:23.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.682+0000 7fe20a000700 1 -- 172.21.15.72:0/489021016 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2041acc80 con 0x7fe204077950 2024-04-26T23:45:23.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.683+0000 7fe20a000700 1 -- 172.21.15.72:0/489021016 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe20406a110 con 0x7fe204077950 2024-04-26T23:45:23.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.685+0000 7fe200c00700 1 -- 172.21.15.72:0/489021016 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fe1f4012050 con 0x7fe204077950 2024-04-26T23:45:23.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.687+0000 7fe200c00700 1 --2- 172.21.15.72:0/489021016 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fe1e80a3cd0 0x7fe1e80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:23.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.687+0000 7fe200c00700 1 -- 172.21.15.72:0/489021016 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7fe1f40d0940 con 0x7fe204077950 2024-04-26T23:45:23.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.687+0000 7fe1faa00700 1 --2- 172.21.15.72:0/489021016 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fe1e80a3cd0 0x7fe1e80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:23.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.688+0000 7fe1faa00700 1 --2- 172.21.15.72:0/489021016 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fe1e80a3cd0 0x7fe1e80a6190 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fe204104c30 tx=0x7fe1ec00b280 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:23.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.691+0000 7fe200c00700 1 -- 172.21.15.72:0/489021016 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe1f4010040 con 0x7fe204077950 2024-04-26T23:45:23.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:23 smithi072 ceph-mon[28184]: pgmap v395: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:45:23.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:23 smithi076 ceph-mon[32774]: pgmap v395: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:45:23.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.968+0000 7fe20a000700 1 -- 172.21.15.72:0/489021016 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fe20402e490 con 0x7fe204077950 2024-04-26T23:45:23.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.969+0000 7fe200c00700 1 -- 172.21.15.72:0/489021016 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fe1f409cd70 con 0x7fe204077950 2024-04-26T23:45:23.973 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:45:23.974 INFO:teuthology.orchestra.run.smithi072.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-26T23:45:23.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.972+0000 7fe20a000700 1 -- 172.21.15.72:0/489021016 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fe1e80a3cd0 msgr2=0x7fe1e80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:23.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.972+0000 7fe20a000700 1 --2- 172.21.15.72:0/489021016 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fe1e80a3cd0 0x7fe1e80a6190 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fe204104c30 tx=0x7fe1ec00b280 comp rx=0 tx=0).stop 2024-04-26T23:45:23.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.972+0000 7fe20a000700 1 -- 172.21.15.72:0/489021016 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe204077950 msgr2=0x7fe2041a0100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:23.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.972+0000 7fe20a000700 1 --2- 172.21.15.72:0/489021016 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe204077950 0x7fe2041a0100 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe1f400ef10 tx=0x7fe1f400c3a0 comp rx=0 tx=0).stop 2024-04-26T23:45:23.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.972+0000 7fe20a000700 1 -- 172.21.15.72:0/489021016 shutdown_connections 2024-04-26T23:45:23.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.972+0000 7fe20a000700 1 --2- 172.21.15.72:0/489021016 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fe1e80a3cd0 0x7fe1e80a6190 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:23.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.972+0000 7fe20a000700 1 --2- 172.21.15.72:0/489021016 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe2041142b0 0x7fe2041a49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:23.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.972+0000 7fe20a000700 1 --2- 172.21.15.72:0/489021016 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe204077950 0x7fe2041a0100 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:23.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.972+0000 7fe20a000700 1 --2- 172.21.15.72:0/489021016 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe204078270 0x7fe2041a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:23.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.972+0000 7fe20a000700 1 -- 172.21.15.72:0/489021016 >> 172.21.15.72:0/489021016 conn(0x7fe204101100 msgr2=0x7fe204114f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:23.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.972+0000 7fe20a000700 1 -- 172.21.15.72:0/489021016 shutdown_connections 2024-04-26T23:45:23.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:23.972+0000 7fe20a000700 1 -- 172.21.15.72:0/489021016 wait complete. 2024-04-26T23:45:24.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:23 smithi136 ceph-mon[33050]: pgmap v395: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:45:24.789 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:24 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/489021016' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-26T23:45:24.837 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-26T23:45:24.838 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:45:24.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:24 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/489021016' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-26T23:45:25.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.037+0000 7f1857200640 1 -- 172.21.15.72:0/755900544 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1858104990 msgr2=0x7f1858104db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:25.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.037+0000 7f1857200640 1 --2- 172.21.15.72:0/755900544 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1858104990 0x7f1858104db0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1848009860 tx=0x7f184802a340 comp rx=0 tx=0).stop 2024-04-26T23:45:25.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.037+0000 7f1857200640 1 -- 172.21.15.72:0/755900544 shutdown_connections 2024-04-26T23:45:25.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.037+0000 7f1857200640 1 --2- 172.21.15.72:0/755900544 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1858106540 0x7f185810cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:25.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.037+0000 7f1857200640 1 --2- 172.21.15.72:0/755900544 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1858105b80 0x7f1858106000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:25.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.037+0000 7f1857200640 1 --2- 172.21.15.72:0/755900544 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1858104990 0x7f1858104db0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:25.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.037+0000 7f1857200640 1 -- 172.21.15.72:0/755900544 >> 172.21.15.72:0/755900544 conn(0x7f18580fff00 msgr2=0x7f1858102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:25.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.037+0000 7f1857200640 1 -- 172.21.15.72:0/755900544 shutdown_connections 2024-04-26T23:45:25.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.037+0000 7f1857200640 1 -- 172.21.15.72:0/755900544 wait complete. 2024-04-26T23:45:25.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.038+0000 7f1857200640 1 Processor -- start 2024-04-26T23:45:25.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.039+0000 7f1857200640 1 -- start start 2024-04-26T23:45:25.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.039+0000 7f1857200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1858104990 0x7f185819abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:25.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.039+0000 7f1857200640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1858105b80 0x7f185819b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:25.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.039+0000 7f1857200640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1858106540 0x7f18581a2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:25.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.039+0000 7f1857200640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f1858110490 con 0x7f1858105b80 2024-04-26T23:45:25.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.039+0000 7f1857200640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f1858110310 con 0x7f1858104990 2024-04-26T23:45:25.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.039+0000 7f1857200640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f1858110610 con 0x7f1858106540 2024-04-26T23:45:25.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.039+0000 7f1855e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1858104990 0x7f185819abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:25.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.039+0000 7f1855400640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1858105b80 0x7f185819b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:25.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.039+0000 7f1855e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1858104990 0x7f185819abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:44510/0 (socket says 172.21.15.72:44510) 2024-04-26T23:45:25.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.039+0000 7f1855e00640 1 -- 172.21.15.72:0/1733628415 learned_addr learned my addr 172.21.15.72:0/1733628415 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:45:25.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.039+0000 7f1856800640 1 --2- 172.21.15.72:0/1733628415 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1858106540 0x7f18581a2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:25.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.040+0000 7f1855e00640 1 -- 172.21.15.72:0/1733628415 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1858106540 msgr2=0x7f18581a2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:25.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.040+0000 7f1855e00640 1 --2- 172.21.15.72:0/1733628415 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1858106540 0x7f18581a2120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:25.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.040+0000 7f1855e00640 1 -- 172.21.15.72:0/1733628415 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1858105b80 msgr2=0x7f185819b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:25.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.040+0000 7f1855e00640 1 --2- 172.21.15.72:0/1733628415 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1858105b80 0x7f185819b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:25.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.040+0000 7f1855e00640 1 -- 172.21.15.72:0/1733628415 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18581a2790 con 0x7f1858104990 2024-04-26T23:45:25.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.040+0000 7f1855e00640 1 --2- 172.21.15.72:0/1733628415 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1858104990 0x7f185819abc0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f18480057c0 tx=0x7f1848032da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:25.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.040+0000 7f184ea00640 1 -- 172.21.15.72:0/1733628415 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1848037040 con 0x7f1858104990 2024-04-26T23:45:25.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.040+0000 7f1857200640 1 -- 172.21.15.72:0/1733628415 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18581a2a20 con 0x7f1858104990 2024-04-26T23:45:25.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.040+0000 7f184ea00640 1 -- 172.21.15.72:0/1733628415 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1848030bd0 con 0x7f1858104990 2024-04-26T23:45:25.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.040+0000 7f184ea00640 1 -- 172.21.15.72:0/1733628415 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1848005280 con 0x7f1858104990 2024-04-26T23:45:25.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.041+0000 7f1857200640 1 -- 172.21.15.72:0/1733628415 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18581a2f00 con 0x7f1858104990 2024-04-26T23:45:25.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.042+0000 7f1857200640 1 -- 172.21.15.72:0/1733628415 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f185810be20 con 0x7f1858104990 2024-04-26T23:45:25.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.044+0000 7f184ea00640 1 -- 172.21.15.72:0/1733628415 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1848043020 con 0x7f1858104990 2024-04-26T23:45:25.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.045+0000 7f184ea00640 1 --2- 172.21.15.72:0/1733628415 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f18380a3c40 0x7f18380a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:25.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.045+0000 7f184ea00640 1 -- 172.21.15.72:0/1733628415 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f18480efde0 con 0x7f1858104990 2024-04-26T23:45:25.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.045+0000 7f1855400640 1 --2- 172.21.15.72:0/1733628415 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f18380a3c40 0x7f18380a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:25.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.046+0000 7f1855400640 1 --2- 172.21.15.72:0/1733628415 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f18380a3c40 0x7f18380a6100 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f185819c080 tx=0x7f1840010040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:25.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:24 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/489021016' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-26T23:45:25.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.050+0000 7f184ea00640 1 -- 172.21.15.72:0/1733628415 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f18480bc320 con 0x7f1858104990 2024-04-26T23:45:25.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.251+0000 7f1857200640 1 -- 172.21.15.72:0/1733628415 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1858063530 con 0x7f1858104990 2024-04-26T23:45:25.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.255+0000 7f184ea00640 1 -- 172.21.15.72:0/1733628415 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24841 (secure 0 0 0) 0x7f18480a6020 con 0x7f1858104990 2024-04-26T23:45:25.259 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:45:25.259 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":93,"fsid":"3c8453ae-0425-11ef-bc93-c7b262605968","created":"2024-04-26T23:33:48.088300+0000","modified":"2024-04-26T23:45:00.532006+0000","last_up_change":"2024-04-26T23:42:04.482891+0000","last_in_change":"2024-04-26T23:41:44.784415+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:37:44.428727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:43:00.431829+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:43:01.872977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:43:05.512050+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e20c9ebf-56f6-47c1-a11b-e1e615af042b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6803","nonce":1639298875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6805","nonce":1639298875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6809","nonce":1639298875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6807","nonce":1639298875}]},"public_addr":"172.21.15.72:6803/1639298875","cluster_addr":"172.21.15.72:6805/1639298875","heartbeat_back_addr":"172.21.15.72:6809/1639298875","heartbeat_front_addr":"172.21.15.72:6807/1639298875","state":["exists","up"]},{"osd":1,"uuid":"e6cac469-780c-4d66-b560-b686072371fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6801","nonce":4185342193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6803","nonce":4185342193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6807","nonce":4185342193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6805","nonce":4185342193}]},"public_addr":"172.21.15.76:6801/4185342193","cluster_addr":"172.21.15.76:6803/4185342193","heartbeat_back_addr":"172.21.15.76:6807/4185342193","heartbeat_front_addr":"172.21.15.76:6805/4185342193","state":["exists","up"]},{"osd":2,"uuid":"838a49fc-18de-446c-8336-67b3e05da231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6801","nonce":3033860799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6803","nonce":3033860799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6807","nonce":3033860799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6805","nonce":3033860799}]},"public_addr":"172.21.15.136:6801/3033860799","cluster_addr":"172.21.15.136:6803/3033860799","heartbeat_back_addr":"172.21.15.136:6807/3033860799","heartbeat_front_addr":"172.21.15.136:6805/3033860799","state":["exists","up"]},{"osd":3,"uuid":"e6da89be-5ba7-48a4-9306-f75498660e0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6811","nonce":2071107815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6813","nonce":2071107815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6817","nonce":2071107815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6815","nonce":2071107815}]},"public_addr":"172.21.15.72:6811/2071107815","cluster_addr":"172.21.15.72:6813/2071107815","heartbeat_back_addr":"172.21.15.72:6817/2071107815","heartbeat_front_addr":"172.21.15.72:6815/2071107815","state":["exists","up"]},{"osd":4,"uuid":"0b17c491-d798-488f-892d-521ef2fbc51e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6809","nonce":147449238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6811","nonce":147449238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6815","nonce":147449238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6813","nonce":147449238}]},"public_addr":"172.21.15.76:6809/147449238","cluster_addr":"172.21.15.76:6811/147449238","heartbeat_back_addr":"172.21.15.76:6815/147449238","heartbeat_front_addr":"172.21.15.76:6813/147449238","state":["exists","up"]},{"osd":5,"uuid":"c7e887b3-ab38-4608-9bfd-1ed49956e8af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6809","nonce":2320149212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6811","nonce":2320149212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6815","nonce":2320149212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6813","nonce":2320149212}]},"public_addr":"172.21.15.136:6809/2320149212","cluster_addr":"172.21.15.136:6811/2320149212","heartbeat_back_addr":"172.21.15.136:6815/2320149212","heartbeat_front_addr":"172.21.15.136:6813/2320149212","state":["exists","up"]},{"osd":6,"uuid":"f0bd604b-0f57-4a38-9295-eb200264f1ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6819","nonce":389499207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6821","nonce":389499207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6825","nonce":389499207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6823","nonce":389499207}]},"public_addr":"172.21.15.72:6819/389499207","cluster_addr":"172.21.15.72:6821/389499207","heartbeat_back_addr":"172.21.15.72:6825/389499207","heartbeat_front_addr":"172.21.15.72:6823/389499207","state":["exists","up"]},{"osd":7,"uuid":"1879d4e4-038e-4cd6-a3ec-3a940bcc3797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6817","nonce":375888575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6819","nonce":375888575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6823","nonce":375888575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6821","nonce":375888575}]},"public_addr":"172.21.15.76:6817/375888575","cluster_addr":"172.21.15.76:6819/375888575","heartbeat_back_addr":"172.21.15.76:6823/375888575","heartbeat_front_addr":"172.21.15.76:6821/375888575","state":["exists","up"]},{"osd":8,"uuid":"6f404062-97e7-4ccb-a582-76dc1ffae548","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6817","nonce":3446774052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6819","nonce":3446774052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6823","nonce":3446774052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6821","nonce":3446774052}]},"public_addr":"172.21.15.136:6817/3446774052","cluster_addr":"172.21.15.136:6819/3446774052","heartbeat_back_addr":"172.21.15.136:6823/3446774052","heartbeat_front_addr":"172.21.15.136:6821/3446774052","state":["exists","up"]},{"osd":9,"uuid":"2989ca68-3164-4667-9bcf-230dcbb7b8f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6827","nonce":3423789152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6829","nonce":3423789152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6832","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6833","nonce":3423789152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6831","nonce":3423789152}]},"public_addr":"172.21.15.72:6827/3423789152","cluster_addr":"172.21.15.72:6829/3423789152","heartbeat_back_addr":"172.21.15.72:6833/3423789152","heartbeat_front_addr":"172.21.15.72:6831/3423789152","state":["exists","up"]},{"osd":10,"uuid":"9ec6d46a-0913-4208-97b5-167188ee8cb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6825","nonce":1120991574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6827","nonce":1120991574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6831","nonce":1120991574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6829","nonce":1120991574}]},"public_addr":"172.21.15.76:6825/1120991574","cluster_addr":"172.21.15.76:6827/1120991574","heartbeat_back_addr":"172.21.15.76:6831/1120991574","heartbeat_front_addr":"172.21.15.76:6829/1120991574","state":["exists","up"]},{"osd":11,"uuid":"bdd0774f-6e58-4918-b4cc-afd4f1118d3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6825","nonce":1135995193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6827","nonce":1135995193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6831","nonce":1135995193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6829","nonce":1135995193}]},"public_addr":"172.21.15.136:6825/1135995193","cluster_addr":"172.21.15.136:6827/1135995193","heartbeat_back_addr":"172.21.15.136:6831/1135995193","heartbeat_front_addr":"172.21.15.136:6829/1135995193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:36:52.128788+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:37:17.523654+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:37:41.634677+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:38:15.060797+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:38:41.234111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:39:06.783560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:39:40.884386+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:40:08.340465+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:40:34.344858+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:41:08.525309+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:41:36.782624+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:42:02.756888+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":6}]},{"pgid":"3.37","mappings":[{"from":5,"to":7}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:0/3625071330":"2024-04-27T23:34:58.266218+0000","172.21.15.72:6801/2903979134":"2024-04-27T23:34:58.266218+0000","172.21.15.72:6800/2903979134":"2024-04-27T23:34:58.266218+0000","172.21.15.72:0/1519142576":"2024-04-27T23:34:58.266218+0000","172.21.15.72:0/3669626460":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/945623297":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/3479150073":"2024-04-27T23:34:24.233319+0000","172.21.15.72:6800/4175896290":"2024-04-27T23:34:24.233319+0000","172.21.15.72:6801/4175896290":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/3783175012":"2024-04-27T23:34:58.266218+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:45:25.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.256+0000 7f1857200640 1 -- 172.21.15.72:0/1733628415 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f18380a3c40 msgr2=0x7f18380a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:25.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.256+0000 7f1857200640 1 --2- 172.21.15.72:0/1733628415 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f18380a3c40 0x7f18380a6100 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f185819c080 tx=0x7f1840010040 comp rx=0 tx=0).stop 2024-04-26T23:45:25.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.256+0000 7f1857200640 1 -- 172.21.15.72:0/1733628415 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1858104990 msgr2=0x7f185819abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:25.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.256+0000 7f1857200640 1 --2- 172.21.15.72:0/1733628415 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1858104990 0x7f185819abc0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f18480057c0 tx=0x7f1848032da0 comp rx=0 tx=0).stop 2024-04-26T23:45:25.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.257+0000 7f1857200640 1 -- 172.21.15.72:0/1733628415 shutdown_connections 2024-04-26T23:45:25.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.257+0000 7f1857200640 1 --2- 172.21.15.72:0/1733628415 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f18380a3c40 0x7f18380a6100 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:25.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.257+0000 7f1857200640 1 --2- 172.21.15.72:0/1733628415 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1858106540 0x7f18581a2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:25.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.257+0000 7f1857200640 1 --2- 172.21.15.72:0/1733628415 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1858105b80 0x7f185819b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:25.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.257+0000 7f1857200640 1 --2- 172.21.15.72:0/1733628415 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1858104990 0x7f185819abc0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:25.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.257+0000 7f1857200640 1 -- 172.21.15.72:0/1733628415 >> 172.21.15.72:0/1733628415 conn(0x7f18580fff00 msgr2=0x7f1858108ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:25.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.257+0000 7f1857200640 1 -- 172.21.15.72:0/1733628415 shutdown_connections 2024-04-26T23:45:25.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.257+0000 7f1857200640 1 -- 172.21.15.72:0/1733628415 wait complete. 2024-04-26T23:45:25.281 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:37:44.428727+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T23:43:00.431829+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T23:43:01.872977+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T23:43:05.512050+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-26T23:45:25.281 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:45:25.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.484+0000 7f53e3200640 1 -- 172.21.15.72:0/2487185891 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53e41028c0 msgr2=0x7f53e4102ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:25.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.484+0000 7f53e3200640 1 --2- 172.21.15.72:0/2487185891 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53e41028c0 0x7f53e4102ce0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f53d4009900 tx=0x7f53d402a410 comp rx=0 tx=0).stop 2024-04-26T23:45:25.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.484+0000 7f53e3200640 1 -- 172.21.15.72:0/2487185891 shutdown_connections 2024-04-26T23:45:25.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.484+0000 7f53e3200640 1 --2- 172.21.15.72:0/2487185891 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f53e4103ab0 0x7f53e4103f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:25.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.484+0000 7f53e3200640 1 --2- 172.21.15.72:0/2487185891 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53e4104470 0x7f53e410ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:25.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.484+0000 7f53e3200640 1 --2- 172.21.15.72:0/2487185891 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53e41028c0 0x7f53e4102ce0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:25.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.484+0000 7f53e3200640 1 -- 172.21.15.72:0/2487185891 >> 172.21.15.72:0/2487185891 conn(0x7f53e40fde30 msgr2=0x7f53e4100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:25.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.484+0000 7f53e3200640 1 -- 172.21.15.72:0/2487185891 shutdown_connections 2024-04-26T23:45:25.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.484+0000 7f53e3200640 1 -- 172.21.15.72:0/2487185891 wait complete. 2024-04-26T23:45:25.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.485+0000 7f53e3200640 1 Processor -- start 2024-04-26T23:45:25.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.485+0000 7f53e3200640 1 -- start start 2024-04-26T23:45:25.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.486+0000 7f53e3200640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53e41028c0 0x7f53e419ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:25.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.486+0000 7f53e3200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f53e4103ab0 0x7f53e419b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:25.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.486+0000 7f53e3200640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53e4104470 0x7f53e41a2260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:25.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.486+0000 7f53e3200640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f53e410e370 con 0x7f53e41028c0 2024-04-26T23:45:25.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.486+0000 7f53e3200640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f53e410e1f0 con 0x7f53e4103ab0 2024-04-26T23:45:25.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.486+0000 7f53e3200640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f53e410e4f0 con 0x7f53e4104470 2024-04-26T23:45:25.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.486+0000 7f53e1e00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53e41028c0 0x7f53e419ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:25.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.486+0000 7f53e1e00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53e41028c0 0x7f53e419ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37180/0 (socket says 172.21.15.72:37180) 2024-04-26T23:45:25.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.486+0000 7f53e1e00640 1 -- 172.21.15.72:0/560879408 learned_addr learned my addr 172.21.15.72:0/560879408 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:45:25.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.486+0000 7f53e2800640 1 --2- 172.21.15.72:0/560879408 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53e4104470 0x7f53e41a2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:25.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.486+0000 7f53d9400640 1 --2- 172.21.15.72:0/560879408 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f53e4103ab0 0x7f53e419b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:25.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.487+0000 7f53e1e00640 1 -- 172.21.15.72:0/560879408 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53e4104470 msgr2=0x7f53e41a2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:25.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.487+0000 7f53e1e00640 1 --2- 172.21.15.72:0/560879408 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53e4104470 0x7f53e41a2260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:25.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.487+0000 7f53e1e00640 1 -- 172.21.15.72:0/560879408 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f53e4103ab0 msgr2=0x7f53e419b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:25.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.487+0000 7f53e1e00640 1 --2- 172.21.15.72:0/560879408 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f53e4103ab0 0x7f53e419b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:25.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.487+0000 7f53e1e00640 1 -- 172.21.15.72:0/560879408 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53e41a2900 con 0x7f53e41028c0 2024-04-26T23:45:25.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.487+0000 7f53e1e00640 1 --2- 172.21.15.72:0/560879408 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53e41028c0 0x7f53e419ad40 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f53d4006010 tx=0x7f53d40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:25.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.487+0000 7f53db400640 1 -- 172.21.15.72:0/560879408 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53d403f030 con 0x7f53e41028c0 2024-04-26T23:45:25.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.487+0000 7f53e3200640 1 -- 172.21.15.72:0/560879408 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53e41a2b90 con 0x7f53e41028c0 2024-04-26T23:45:25.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.487+0000 7f53db400640 1 -- 172.21.15.72:0/560879408 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53d4031910 con 0x7f53e41028c0 2024-04-26T23:45:25.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.487+0000 7f53db400640 1 -- 172.21.15.72:0/560879408 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53d4035070 con 0x7f53e41028c0 2024-04-26T23:45:25.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.488+0000 7f53e3200640 1 -- 172.21.15.72:0/560879408 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53e41a3080 con 0x7f53e41028c0 2024-04-26T23:45:25.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.489+0000 7f53e3200640 1 -- 172.21.15.72:0/560879408 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53e41a3250 con 0x7f53e41028c0 2024-04-26T23:45:25.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.491+0000 7f53db400640 1 -- 172.21.15.72:0/560879408 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f53d4042030 con 0x7f53e41028c0 2024-04-26T23:45:25.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.492+0000 7f53db400640 1 --2- 172.21.15.72:0/560879408 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f53c80a3c40 0x7f53c80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:25.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.492+0000 7f53db400640 1 -- 172.21.15.72:0/560879408 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f53d40f0f60 con 0x7f53e41028c0 2024-04-26T23:45:25.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.492+0000 7f53d9400640 1 --2- 172.21.15.72:0/560879408 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f53c80a3c40 0x7f53c80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:25.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.493+0000 7f53d9400640 1 --2- 172.21.15.72:0/560879408 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f53c80a3c40 0x7f53c80a6100 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f53cc009bd0 tx=0x7f53cc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:25.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.497+0000 7f53db400640 1 -- 172.21.15.72:0/560879408 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f53d4003e60 con 0x7f53e41028c0 2024-04-26T23:45:25.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.700+0000 7f53e3200640 1 -- 172.21.15.72:0/560879408 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f53e41a3420 con 0x7f53e41028c0 2024-04-26T23:45:25.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.701+0000 7f53db400640 1 -- 172.21.15.72:0/560879408 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f53d40bcd40 con 0x7f53e41028c0 2024-04-26T23:45:25.705 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-04-26T23:45:25.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.703+0000 7f53e3200640 1 -- 172.21.15.72:0/560879408 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f53c80a3c40 msgr2=0x7f53c80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:25.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.703+0000 7f53e3200640 1 --2- 172.21.15.72:0/560879408 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f53c80a3c40 0x7f53c80a6100 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f53cc009bd0 tx=0x7f53cc005f90 comp rx=0 tx=0).stop 2024-04-26T23:45:25.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.704+0000 7f53e3200640 1 -- 172.21.15.72:0/560879408 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53e41028c0 msgr2=0x7f53e419ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:25.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.704+0000 7f53e3200640 1 --2- 172.21.15.72:0/560879408 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53e41028c0 0x7f53e419ad40 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f53d4006010 tx=0x7f53d40042f0 comp rx=0 tx=0).stop 2024-04-26T23:45:25.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.704+0000 7f53e3200640 1 -- 172.21.15.72:0/560879408 shutdown_connections 2024-04-26T23:45:25.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.704+0000 7f53e3200640 1 --2- 172.21.15.72:0/560879408 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f53c80a3c40 0x7f53c80a6100 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:25.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.704+0000 7f53e3200640 1 --2- 172.21.15.72:0/560879408 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f53e4103ab0 0x7f53e419b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:25.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.704+0000 7f53e3200640 1 --2- 172.21.15.72:0/560879408 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53e4104470 0x7f53e41a2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:25.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.704+0000 7f53e3200640 1 --2- 172.21.15.72:0/560879408 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53e41028c0 0x7f53e419ad40 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:25.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.704+0000 7f53e3200640 1 -- 172.21.15.72:0/560879408 >> 172.21.15.72:0/560879408 conn(0x7f53e40fde30 msgr2=0x7f53e4074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:25.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.704+0000 7f53e3200640 1 -- 172.21.15.72:0/560879408 shutdown_connections 2024-04-26T23:45:25.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.704+0000 7f53e3200640 1 -- 172.21.15.72:0/560879408 wait complete. 2024-04-26T23:45:25.727 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T23:45:25.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:25 smithi076 ceph-mon[32774]: pgmap v396: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:45:25.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:25 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/1733628415' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:45:25.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.931+0000 7f2b52200640 1 -- 172.21.15.72:0/2510538431 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2b4c078530 msgr2=0x7f2b4c078990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:25.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.931+0000 7f2b52200640 1 --2- 172.21.15.72:0/2510538431 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2b4c078530 0x7f2b4c078990 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2b38009860 tx=0x7f2b3802a340 comp rx=0 tx=0).stop 2024-04-26T23:45:25.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.932+0000 7f2b52200640 1 -- 172.21.15.72:0/2510538431 shutdown_connections 2024-04-26T23:45:25.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.932+0000 7f2b52200640 1 --2- 172.21.15.72:0/2510538431 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2b4c078530 0x7f2b4c078990 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:25.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.932+0000 7f2b52200640 1 --2- 172.21.15.72:0/2510538431 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2b4c078ed0 0x7f2b4c079770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:25.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.932+0000 7f2b52200640 1 --2- 172.21.15.72:0/2510538431 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b4c0772f0 0x7f2b4c077710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:25.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.932+0000 7f2b52200640 1 -- 172.21.15.72:0/2510538431 >> 172.21.15.72:0/2510538431 conn(0x7f2b4c100120 msgr2=0x7f2b4c102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:25.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.932+0000 7f2b52200640 1 -- 172.21.15.72:0/2510538431 shutdown_connections 2024-04-26T23:45:25.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.932+0000 7f2b52200640 1 -- 172.21.15.72:0/2510538431 wait complete. 2024-04-26T23:45:25.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.933+0000 7f2b52200640 1 Processor -- start 2024-04-26T23:45:25.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.933+0000 7f2b52200640 1 -- start start 2024-04-26T23:45:25.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.933+0000 7f2b52200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2b4c0772f0 0x7f2b4c19f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:25.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.933+0000 7f2b52200640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b4c078530 0x7f2b4c19f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:25.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.933+0000 7f2b52200640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2b4c078ed0 0x7f2b4c1a6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:25.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.933+0000 7f2b52200640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f2b4c114ad0 con 0x7f2b4c078530 2024-04-26T23:45:25.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.933+0000 7f2b52200640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2b4c114950 con 0x7f2b4c0772f0 2024-04-26T23:45:25.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.933+0000 7f2b52200640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f2b4c114c50 con 0x7f2b4c078ed0 2024-04-26T23:45:25.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.933+0000 7f2b4aa00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b4c078530 0x7f2b4c19f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:25.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.934+0000 7f2b4aa00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b4c078530 0x7f2b4c19f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37188/0 (socket says 172.21.15.72:37188) 2024-04-26T23:45:25.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.934+0000 7f2b4aa00640 1 -- 172.21.15.72:0/3805238447 learned_addr learned my addr 172.21.15.72:0/3805238447 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:45:25.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.934+0000 7f2b4b400640 1 --2- 172.21.15.72:0/3805238447 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2b4c0772f0 0x7f2b4c19f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:25.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.934+0000 7f2b4be00640 1 --2- 172.21.15.72:0/3805238447 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2b4c078ed0 0x7f2b4c1a6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:25.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.934+0000 7f2b4aa00640 1 -- 172.21.15.72:0/3805238447 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2b4c078ed0 msgr2=0x7f2b4c1a6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:25.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.934+0000 7f2b4aa00640 1 --2- 172.21.15.72:0/3805238447 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2b4c078ed0 0x7f2b4c1a6790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:25.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.934+0000 7f2b4aa00640 1 -- 172.21.15.72:0/3805238447 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2b4c0772f0 msgr2=0x7f2b4c19f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:25.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.934+0000 7f2b4aa00640 1 --2- 172.21.15.72:0/3805238447 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2b4c0772f0 0x7f2b4c19f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:25.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.934+0000 7f2b4aa00640 1 -- 172.21.15.72:0/3805238447 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b4c1a6e00 con 0x7f2b4c078530 2024-04-26T23:45:25.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.934+0000 7f2b4b400640 1 --2- 172.21.15.72:0/3805238447 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2b4c0772f0 0x7f2b4c19f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:45:25.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.934+0000 7f2b4aa00640 1 --2- 172.21.15.72:0/3805238447 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b4c078530 0x7f2b4c19f720 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2b380057c0 tx=0x7f2b38032da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:25.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.934+0000 7f2b43e00640 1 -- 172.21.15.72:0/3805238447 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b38037040 con 0x7f2b4c078530 2024-04-26T23:45:25.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.935+0000 7f2b52200640 1 -- 172.21.15.72:0/3805238447 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b4c1a7090 con 0x7f2b4c078530 2024-04-26T23:45:25.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.935+0000 7f2b43e00640 1 -- 172.21.15.72:0/3805238447 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b38030bd0 con 0x7f2b4c078530 2024-04-26T23:45:25.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.935+0000 7f2b43e00640 1 -- 172.21.15.72:0/3805238447 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b38005280 con 0x7f2b4c078530 2024-04-26T23:45:25.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.935+0000 7f2b52200640 1 -- 172.21.15.72:0/3805238447 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b4c1a7570 con 0x7f2b4c078530 2024-04-26T23:45:25.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.936+0000 7f2b52200640 1 -- 172.21.15.72:0/3805238447 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b4c111500 con 0x7f2b4c078530 2024-04-26T23:45:25.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.938+0000 7f2b43e00640 1 -- 172.21.15.72:0/3805238447 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f2b38043020 con 0x7f2b4c078530 2024-04-26T23:45:25.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.939+0000 7f2b43e00640 1 --2- 172.21.15.72:0/3805238447 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f2b300a3c40 0x7f2b300a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:25.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.940+0000 7f2b43e00640 1 -- 172.21.15.72:0/3805238447 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f2b380eff30 con 0x7f2b4c078530 2024-04-26T23:45:25.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.940+0000 7f2b4b400640 1 --2- 172.21.15.72:0/3805238447 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f2b300a3c40 0x7f2b300a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:25.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.940+0000 7f2b4b400640 1 --2- 172.21.15.72:0/3805238447 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f2b300a3c40 0x7f2b300a6100 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c0096c0 tx=0x7f2b3c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:25.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:25.944+0000 7f2b43e00640 1 -- 172.21.15.72:0/3805238447 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f2b380bc470 con 0x7f2b4c078530 2024-04-26T23:45:26.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:25 smithi072 ceph-mon[28184]: pgmap v396: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:45:26.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:25 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/1733628415' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:45:26.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:25 smithi136 ceph-mon[33050]: pgmap v396: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:45:26.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:25 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/1733628415' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:45:26.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.146+0000 7f2b52200640 1 -- 172.21.15.72:0/3805238447 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f2b4c02dcd0 con 0x7f2b4c078530 2024-04-26T23:45:26.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.147+0000 7f2b43e00640 1 -- 172.21.15.72:0/3805238447 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f2b380bbd70 con 0x7f2b4c078530 2024-04-26T23:45:26.152 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-26T23:45:26.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.150+0000 7f2b52200640 1 -- 172.21.15.72:0/3805238447 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f2b300a3c40 msgr2=0x7f2b300a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:26.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.150+0000 7f2b52200640 1 --2- 172.21.15.72:0/3805238447 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f2b300a3c40 0x7f2b300a6100 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c0096c0 tx=0x7f2b3c009260 comp rx=0 tx=0).stop 2024-04-26T23:45:26.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.150+0000 7f2b52200640 1 -- 172.21.15.72:0/3805238447 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b4c078530 msgr2=0x7f2b4c19f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:26.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.150+0000 7f2b52200640 1 --2- 172.21.15.72:0/3805238447 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b4c078530 0x7f2b4c19f720 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2b380057c0 tx=0x7f2b38032da0 comp rx=0 tx=0).stop 2024-04-26T23:45:26.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.150+0000 7f2b52200640 1 -- 172.21.15.72:0/3805238447 shutdown_connections 2024-04-26T23:45:26.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.150+0000 7f2b52200640 1 --2- 172.21.15.72:0/3805238447 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f2b300a3c40 0x7f2b300a6100 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:26.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.150+0000 7f2b52200640 1 --2- 172.21.15.72:0/3805238447 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2b4c078ed0 0x7f2b4c1a6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:26.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.150+0000 7f2b52200640 1 --2- 172.21.15.72:0/3805238447 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b4c078530 0x7f2b4c19f720 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:26.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.150+0000 7f2b52200640 1 --2- 172.21.15.72:0/3805238447 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2b4c0772f0 0x7f2b4c19f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:26.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.150+0000 7f2b52200640 1 -- 172.21.15.72:0/3805238447 >> 172.21.15.72:0/3805238447 conn(0x7f2b4c100120 msgr2=0x7f2b4c101c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:26.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.150+0000 7f2b52200640 1 -- 172.21.15.72:0/3805238447 shutdown_connections 2024-04-26T23:45:26.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.150+0000 7f2b52200640 1 -- 172.21.15.72:0/3805238447 wait complete. 2024-04-26T23:45:26.174 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T23:45:26.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.377+0000 7f7825200640 1 -- 172.21.15.72:0/1755103842 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7820105b80 msgr2=0x7f7820106000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:26.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.377+0000 7f7825200640 1 --2- 172.21.15.72:0/1755103842 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7820105b80 0x7f7820106000 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f780c009900 tx=0x7f780c02a3f0 comp rx=0 tx=0).stop 2024-04-26T23:45:26.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.377+0000 7f7825200640 1 -- 172.21.15.72:0/1755103842 shutdown_connections 2024-04-26T23:45:26.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.377+0000 7f7825200640 1 --2- 172.21.15.72:0/1755103842 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7820105b80 0x7f7820106000 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:26.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.377+0000 7f7825200640 1 --2- 172.21.15.72:0/1755103842 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7820106540 0x7f782010cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:26.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.377+0000 7f7825200640 1 --2- 172.21.15.72:0/1755103842 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7820104990 0x7f7820104db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:26.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.377+0000 7f7825200640 1 -- 172.21.15.72:0/1755103842 >> 172.21.15.72:0/1755103842 conn(0x7f78200fff00 msgr2=0x7f7820102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:26.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.377+0000 7f7825200640 1 -- 172.21.15.72:0/1755103842 shutdown_connections 2024-04-26T23:45:26.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.377+0000 7f7825200640 1 -- 172.21.15.72:0/1755103842 wait complete. 2024-04-26T23:45:26.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.378+0000 7f7825200640 1 Processor -- start 2024-04-26T23:45:26.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.378+0000 7f7825200640 1 -- start start 2024-04-26T23:45:26.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.378+0000 7f7825200640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7820104990 0x7f782019aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:26.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.378+0000 7f7825200640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7820105b80 0x7f782019b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:26.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.378+0000 7f7825200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7820106540 0x7f78201a2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:26.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.378+0000 7f7825200640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f78201104a0 con 0x7f7820104990 2024-04-26T23:45:26.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.378+0000 7f7825200640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f7820110320 con 0x7f7820106540 2024-04-26T23:45:26.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.378+0000 7f7825200640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f7820110620 con 0x7f7820105b80 2024-04-26T23:45:26.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.379+0000 7f781ea00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7820105b80 0x7f782019b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:26.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.379+0000 7f781fe00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7820106540 0x7f78201a2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:26.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.379+0000 7f781ea00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7820105b80 0x7f782019b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.72:48368/0 (socket says 172.21.15.72:48368) 2024-04-26T23:45:26.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.379+0000 7f781ea00640 1 -- 172.21.15.72:0/1092940433 learned_addr learned my addr 172.21.15.72:0/1092940433 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:45:26.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.379+0000 7f781f400640 1 --2- 172.21.15.72:0/1092940433 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7820104990 0x7f782019aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:26.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.379+0000 7f781fe00640 1 -- 172.21.15.72:0/1092940433 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7820105b80 msgr2=0x7f782019b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:26.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.379+0000 7f781fe00640 1 --2- 172.21.15.72:0/1092940433 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7820105b80 0x7f782019b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:26.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.379+0000 7f781fe00640 1 -- 172.21.15.72:0/1092940433 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7820104990 msgr2=0x7f782019aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:26.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.379+0000 7f781fe00640 1 --2- 172.21.15.72:0/1092940433 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7820104990 0x7f782019aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:26.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.379+0000 7f781fe00640 1 -- 172.21.15.72:0/1092940433 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78201a2770 con 0x7f7820106540 2024-04-26T23:45:26.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.380+0000 7f781fe00640 1 --2- 172.21.15.72:0/1092940433 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7820106540 0x7f78201a2100 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7818009fd0 tx=0x7f781800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:26.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.380+0000 7f7817e00640 1 -- 172.21.15.72:0/1092940433 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f781800ce20 con 0x7f7820106540 2024-04-26T23:45:26.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.380+0000 7f7817e00640 1 -- 172.21.15.72:0/1092940433 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7818005020 con 0x7f7820106540 2024-04-26T23:45:26.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.380+0000 7f7825200640 1 -- 172.21.15.72:0/1092940433 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78201a2a60 con 0x7f7820106540 2024-04-26T23:45:26.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.380+0000 7f7817e00640 1 -- 172.21.15.72:0/1092940433 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7818010690 con 0x7f7820106540 2024-04-26T23:45:26.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.380+0000 7f7825200640 1 -- 172.21.15.72:0/1092940433 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78201a2f30 con 0x7f7820106540 2024-04-26T23:45:26.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.381+0000 7f7825200640 1 -- 172.21.15.72:0/1092940433 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f782010bd80 con 0x7f7820106540 2024-04-26T23:45:26.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.383+0000 7f7817e00640 1 -- 172.21.15.72:0/1092940433 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f78180054d0 con 0x7f7820106540 2024-04-26T23:45:26.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.385+0000 7f7817e00640 1 --2- 172.21.15.72:0/1092940433 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f78040a3c40 0x7f78040a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:26.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.385+0000 7f7817e00640 1 -- 172.21.15.72:0/1092940433 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f78180d1530 con 0x7f7820106540 2024-04-26T23:45:26.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.385+0000 7f781f400640 1 --2- 172.21.15.72:0/1092940433 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f78040a3c40 0x7f78040a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:26.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.386+0000 7f781f400640 1 --2- 172.21.15.72:0/1092940433 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f78040a3c40 0x7f78040a6100 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f78201059e0 tx=0x7f7810005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:26.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.387+0000 7f7817e00640 1 -- 172.21.15.72:0/1092940433 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f781809d960 con 0x7f7820106540 2024-04-26T23:45:26.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.594+0000 7f7825200640 1 -- 172.21.15.72:0/1092940433 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f7820063530 con 0x7f7820106540 2024-04-26T23:45:26.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.595+0000 7f7817e00640 1 -- 172.21.15.72:0/1092940433 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f781809d260 con 0x7f7820106540 2024-04-26T23:45:26.599 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-04-26T23:45:26.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.597+0000 7f7825200640 1 -- 172.21.15.72:0/1092940433 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f78040a3c40 msgr2=0x7f78040a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:26.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.597+0000 7f7825200640 1 --2- 172.21.15.72:0/1092940433 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f78040a3c40 0x7f78040a6100 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f78201059e0 tx=0x7f7810005f90 comp rx=0 tx=0).stop 2024-04-26T23:45:26.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.597+0000 7f7825200640 1 -- 172.21.15.72:0/1092940433 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7820106540 msgr2=0x7f78201a2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:26.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.597+0000 7f7825200640 1 --2- 172.21.15.72:0/1092940433 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7820106540 0x7f78201a2100 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7818009fd0 tx=0x7f781800ef10 comp rx=0 tx=0).stop 2024-04-26T23:45:26.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.597+0000 7f7825200640 1 -- 172.21.15.72:0/1092940433 shutdown_connections 2024-04-26T23:45:26.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.597+0000 7f7825200640 1 --2- 172.21.15.72:0/1092940433 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f78040a3c40 0x7f78040a6100 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:26.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.597+0000 7f7825200640 1 --2- 172.21.15.72:0/1092940433 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7820106540 0x7f78201a2100 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:26.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.597+0000 7f7825200640 1 --2- 172.21.15.72:0/1092940433 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7820105b80 0x7f782019b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:26.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.597+0000 7f7825200640 1 --2- 172.21.15.72:0/1092940433 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7820104990 0x7f782019aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:26.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.597+0000 7f7825200640 1 -- 172.21.15.72:0/1092940433 >> 172.21.15.72:0/1092940433 conn(0x7f78200fff00 msgr2=0x7f7820108f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:26.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.597+0000 7f7825200640 1 -- 172.21.15.72:0/1092940433 shutdown_connections 2024-04-26T23:45:26.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.597+0000 7f7825200640 1 -- 172.21.15.72:0/1092940433 wait complete. 2024-04-26T23:45:26.619 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-26T23:45:26.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.814+0000 7f9e4ca00640 1 -- 172.21.15.72:0/1389662982 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e4810ee10 msgr2=0x7f9e48111200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:26.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.814+0000 7f9e4ca00640 1 --2- 172.21.15.72:0/1389662982 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e4810ee10 0x7f9e48111200 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9e40009ca0 tx=0x7f9e4002a8c0 comp rx=0 tx=0).stop 2024-04-26T23:45:26.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.814+0000 7f9e4ca00640 1 -- 172.21.15.72:0/1389662982 shutdown_connections 2024-04-26T23:45:26.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.814+0000 7f9e4ca00640 1 --2- 172.21.15.72:0/1389662982 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e4810ee10 0x7f9e48111200 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:26.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.814+0000 7f9e4ca00640 1 --2- 172.21.15.72:0/1389662982 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9e48075390 0x7f9e48075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:26.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.814+0000 7f9e4ca00640 1 --2- 172.21.15.72:0/1389662982 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e48076970 0x7f9e48074dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:26.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.814+0000 7f9e4ca00640 1 -- 172.21.15.72:0/1389662982 >> 172.21.15.72:0/1389662982 conn(0x7f9e480fddd0 msgr2=0x7f9e481001f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:26.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.815+0000 7f9e4ca00640 1 -- 172.21.15.72:0/1389662982 shutdown_connections 2024-04-26T23:45:26.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.815+0000 7f9e4ca00640 1 -- 172.21.15.72:0/1389662982 wait complete. 2024-04-26T23:45:26.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.816+0000 7f9e4ca00640 1 Processor -- start 2024-04-26T23:45:26.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.816+0000 7f9e4ca00640 1 -- start start 2024-04-26T23:45:26.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.816+0000 7f9e4ca00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e48075390 0x7f9e4819efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:26.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.816+0000 7f9e4ca00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e48076970 0x7f9e4819f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:26.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.816+0000 7f9e4ca00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9e4810ee10 0x7f9e481a6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:26.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.816+0000 7f9e4ca00640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f9e48114990 con 0x7f9e4810ee10 2024-04-26T23:45:26.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.816+0000 7f9e4ca00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f9e48114810 con 0x7f9e48076970 2024-04-26T23:45:26.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.816+0000 7f9e4ca00640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f9e48114b10 con 0x7f9e48075390 2024-04-26T23:45:26.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.816+0000 7f9e47e00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9e4810ee10 0x7f9e481a6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:26.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.816+0000 7f9e47400640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e48075390 0x7f9e4819efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:26.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.816+0000 7f9e3fe00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e48076970 0x7f9e4819f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:26.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.816+0000 7f9e47400640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e48075390 0x7f9e4819efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.72:44344/0 (socket says 172.21.15.72:44344) 2024-04-26T23:45:26.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.816+0000 7f9e47400640 1 -- 172.21.15.72:0/2125589396 learned_addr learned my addr 172.21.15.72:0/2125589396 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:45:26.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.817+0000 7f9e47e00640 1 -- 172.21.15.72:0/2125589396 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e48075390 msgr2=0x7f9e4819efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:26.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.817+0000 7f9e47e00640 1 --2- 172.21.15.72:0/2125589396 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e48075390 0x7f9e4819efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:26.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.817+0000 7f9e47e00640 1 -- 172.21.15.72:0/2125589396 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e48076970 msgr2=0x7f9e4819f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:26.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.817+0000 7f9e47e00640 1 --2- 172.21.15.72:0/2125589396 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e48076970 0x7f9e4819f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:26.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.817+0000 7f9e47e00640 1 -- 172.21.15.72:0/2125589396 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e481a6bd0 con 0x7f9e4810ee10 2024-04-26T23:45:26.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.817+0000 7f9e3fe00640 1 --2- 172.21.15.72:0/2125589396 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e48076970 0x7f9e4819f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:45:26.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.817+0000 7f9e47e00640 1 --2- 172.21.15.72:0/2125589396 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9e4810ee10 0x7f9e481a6560 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9e40031a10 tx=0x7f9e400316f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:26.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.817+0000 7f9e44c00640 1 -- 172.21.15.72:0/2125589396 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e40003b70 con 0x7f9e4810ee10 2024-04-26T23:45:26.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.818+0000 7f9e44c00640 1 -- 172.21.15.72:0/2125589396 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e40003d10 con 0x7f9e4810ee10 2024-04-26T23:45:26.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.818+0000 7f9e4ca00640 1 -- 172.21.15.72:0/2125589396 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e481a6e60 con 0x7f9e4810ee10 2024-04-26T23:45:26.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.818+0000 7f9e44c00640 1 -- 172.21.15.72:0/2125589396 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e4002fb70 con 0x7f9e4810ee10 2024-04-26T23:45:26.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.818+0000 7f9e4ca00640 1 -- 172.21.15.72:0/2125589396 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e481a7370 con 0x7f9e4810ee10 2024-04-26T23:45:26.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.819+0000 7f9e4ca00640 1 -- 172.21.15.72:0/2125589396 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e48104fe0 con 0x7f9e4810ee10 2024-04-26T23:45:26.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.821+0000 7f9e44c00640 1 -- 172.21.15.72:0/2125589396 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f9e400046a0 con 0x7f9e4810ee10 2024-04-26T23:45:26.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.822+0000 7f9e44c00640 1 --2- 172.21.15.72:0/2125589396 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f9e2c0a3cc0 0x7f9e2c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:26.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.822+0000 7f9e44c00640 1 -- 172.21.15.72:0/2125589396 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f9e400f1b40 con 0x7f9e4810ee10 2024-04-26T23:45:26.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.823+0000 7f9e47400640 1 --2- 172.21.15.72:0/2125589396 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f9e2c0a3cc0 0x7f9e2c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:26.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.823+0000 7f9e47400640 1 --2- 172.21.15.72:0/2125589396 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f9e2c0a3cc0 0x7f9e2c0a6180 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f9e38009c90 tx=0x7f9e38005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:26.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:26.826+0000 7f9e44c00640 1 -- 172.21.15.72:0/2125589396 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9e400be080 con 0x7f9e4810ee10 2024-04-26T23:45:26.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:26 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/560879408' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T23:45:26.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:26 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/3805238447' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T23:45:27.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.032+0000 7f9e4ca00640 1 -- 172.21.15.72:0/2125589396 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f9e4802dd20 con 0x7f9e4810ee10 2024-04-26T23:45:27.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.033+0000 7f9e44c00640 1 -- 172.21.15.72:0/2125589396 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f9e400bd980 con 0x7f9e4810ee10 2024-04-26T23:45:27.037 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-26T23:45:27.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.035+0000 7f9e4ca00640 1 -- 172.21.15.72:0/2125589396 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f9e2c0a3cc0 msgr2=0x7f9e2c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:27.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.035+0000 7f9e4ca00640 1 --2- 172.21.15.72:0/2125589396 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f9e2c0a3cc0 0x7f9e2c0a6180 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f9e38009c90 tx=0x7f9e38005f90 comp rx=0 tx=0).stop 2024-04-26T23:45:27.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.035+0000 7f9e4ca00640 1 -- 172.21.15.72:0/2125589396 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9e4810ee10 msgr2=0x7f9e481a6560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:27.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.035+0000 7f9e4ca00640 1 --2- 172.21.15.72:0/2125589396 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9e4810ee10 0x7f9e481a6560 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9e40031a10 tx=0x7f9e400316f0 comp rx=0 tx=0).stop 2024-04-26T23:45:27.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.036+0000 7f9e4ca00640 1 -- 172.21.15.72:0/2125589396 shutdown_connections 2024-04-26T23:45:27.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.036+0000 7f9e4ca00640 1 --2- 172.21.15.72:0/2125589396 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f9e2c0a3cc0 0x7f9e2c0a6180 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:27.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.036+0000 7f9e4ca00640 1 --2- 172.21.15.72:0/2125589396 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e48076970 0x7f9e4819f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:27.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.036+0000 7f9e4ca00640 1 --2- 172.21.15.72:0/2125589396 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9e4810ee10 0x7f9e481a6560 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:27.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.036+0000 7f9e4ca00640 1 --2- 172.21.15.72:0/2125589396 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e48075390 0x7f9e4819efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:27.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.036+0000 7f9e4ca00640 1 -- 172.21.15.72:0/2125589396 >> 172.21.15.72:0/2125589396 conn(0x7f9e480fddd0 msgr2=0x7f9e481021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:27.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.036+0000 7f9e4ca00640 1 -- 172.21.15.72:0/2125589396 shutdown_connections 2024-04-26T23:45:27.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.036+0000 7f9e4ca00640 1 -- 172.21.15.72:0/2125589396 wait complete. 2024-04-26T23:45:27.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:26 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/560879408' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T23:45:27.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:26 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/3805238447' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T23:45:27.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:26 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/560879408' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T23:45:27.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:26 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/3805238447' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T23:45:27.059 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:45:27.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.250+0000 7f50d8c00640 1 -- 172.21.15.72:0/1496264854 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50d4108450 msgr2=0x7f50d410a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:27.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.250+0000 7f50d8c00640 1 --2- 172.21.15.72:0/1496264854 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50d4108450 0x7f50d410a8a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f50c4009900 tx=0x7f50c402a400 comp rx=0 tx=0).stop 2024-04-26T23:45:27.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.251+0000 7f50d8c00640 1 -- 172.21.15.72:0/1496264854 shutdown_connections 2024-04-26T23:45:27.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.251+0000 7f50d8c00640 1 --2- 172.21.15.72:0/1496264854 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50d4105b20 0x7f50d4107f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:27.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.251+0000 7f50d8c00640 1 --2- 172.21.15.72:0/1496264854 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50d4108450 0x7f50d410a8a0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:27.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.251+0000 7f50d8c00640 1 --2- 172.21.15.72:0/1496264854 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f50d41031f0 0x7f50d41055e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:27.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.251+0000 7f50d8c00640 1 -- 172.21.15.72:0/1496264854 >> 172.21.15.72:0/1496264854 conn(0x7f50d40fcbc0 msgr2=0x7f50d40ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:27.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.251+0000 7f50d8c00640 1 -- 172.21.15.72:0/1496264854 shutdown_connections 2024-04-26T23:45:27.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.251+0000 7f50d8c00640 1 -- 172.21.15.72:0/1496264854 wait complete. 2024-04-26T23:45:27.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.252+0000 7f50d8c00640 1 Processor -- start 2024-04-26T23:45:27.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.252+0000 7f50d8c00640 1 -- start start 2024-04-26T23:45:27.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.252+0000 7f50d8c00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f50d41031f0 0x7f50d419ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:27.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.252+0000 7f50d8c00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50d4105b20 0x7f50d419b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:27.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.252+0000 7f50d8c00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50d4108450 0x7f50d41a2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:27.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.252+0000 7f50d8c00640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f50d410e2c0 con 0x7f50d4108450 2024-04-26T23:45:27.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.252+0000 7f50d8c00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f50d410e140 con 0x7f50d4105b20 2024-04-26T23:45:27.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.252+0000 7f50d8c00640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f50d410e440 con 0x7f50d41031f0 2024-04-26T23:45:27.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.252+0000 7f50d3e00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50d4108450 0x7f50d41a2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:27.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.252+0000 7f50d2a00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50d4105b20 0x7f50d419b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:27.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.252+0000 7f50d3e00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50d4108450 0x7f50d41a2100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54048/0 (socket says 172.21.15.72:54048) 2024-04-26T23:45:27.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.252+0000 7f50d3e00640 1 -- 172.21.15.72:0/3306525954 learned_addr learned my addr 172.21.15.72:0/3306525954 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:45:27.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.252+0000 7f50d3400640 1 --2- 172.21.15.72:0/3306525954 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f50d41031f0 0x7f50d419ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:27.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.253+0000 7f50d2a00640 1 -- 172.21.15.72:0/3306525954 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f50d41031f0 msgr2=0x7f50d419ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:27.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.253+0000 7f50d2a00640 1 --2- 172.21.15.72:0/3306525954 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f50d41031f0 0x7f50d419ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:27.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.253+0000 7f50d2a00640 1 -- 172.21.15.72:0/3306525954 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50d4108450 msgr2=0x7f50d41a2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:27.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.253+0000 7f50d2a00640 1 --2- 172.21.15.72:0/3306525954 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50d4108450 0x7f50d41a2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:27.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.253+0000 7f50d2a00640 1 -- 172.21.15.72:0/3306525954 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50d41a2770 con 0x7f50d4105b20 2024-04-26T23:45:27.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.253+0000 7f50d2a00640 1 --2- 172.21.15.72:0/3306525954 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50d4105b20 0x7f50d419b090 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f50bc005fd0 tx=0x7f50bc00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:27.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.253+0000 7f50cbe00640 1 -- 172.21.15.72:0/3306525954 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50bc003b70 con 0x7f50d4105b20 2024-04-26T23:45:27.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.253+0000 7f50d8c00640 1 -- 172.21.15.72:0/3306525954 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50d41a2a60 con 0x7f50d4105b20 2024-04-26T23:45:27.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.253+0000 7f50cbe00640 1 -- 172.21.15.72:0/3306525954 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f50bc003d10 con 0x7f50d4105b20 2024-04-26T23:45:27.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.253+0000 7f50cbe00640 1 -- 172.21.15.72:0/3306525954 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50bc0108d0 con 0x7f50d4105b20 2024-04-26T23:45:27.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.253+0000 7f50d8c00640 1 -- 172.21.15.72:0/3306525954 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50d41a3000 con 0x7f50d4105b20 2024-04-26T23:45:27.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.254+0000 7f50d8c00640 1 -- 172.21.15.72:0/3306525954 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50d4109f40 con 0x7f50d4105b20 2024-04-26T23:45:27.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.256+0000 7f50cbe00640 1 -- 172.21.15.72:0/3306525954 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f50bc004b40 con 0x7f50d4105b20 2024-04-26T23:45:27.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.258+0000 7f50cbe00640 1 --2- 172.21.15.72:0/3306525954 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f50b80a3cc0 0x7f50b80a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:27.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.259+0000 7f50cbe00640 1 -- 172.21.15.72:0/3306525954 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f50bc0d0d30 con 0x7f50d4105b20 2024-04-26T23:45:27.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.259+0000 7f50d3400640 1 --2- 172.21.15.72:0/3306525954 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f50b80a3cc0 0x7f50b80a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:27.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.259+0000 7f50d3400640 1 --2- 172.21.15.72:0/3306525954 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f50b80a3cc0 0x7f50b80a6180 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f50cc009bd0 tx=0x7f50cc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:27.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.262+0000 7f50cbe00640 1 -- 172.21.15.72:0/3306525954 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f50bc09d160 con 0x7f50d4105b20 2024-04-26T23:45:27.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.508+0000 7f50d8c00640 1 -- 172.21.15.72:0/3306525954 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f50d402dad0 con 0x7f50d4105b20 2024-04-26T23:45:27.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.511+0000 7f50cbe00640 1 -- 172.21.15.72:0/3306525954 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11874 (secure 0 0 0) 0x7f50bc09ca60 con 0x7f50d4105b20 2024-04-26T23:45:27.514 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:45:27.514 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/2386456166","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2386456166},{"type":"v1","addr":"172.21.15.72:6835","nonce":2386456166}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1528342053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1528342053},{"type":"v1","addr":"172.21.15.136:6833","nonce":1528342053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6833/2427238559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":2427238559},{"type":"v1","addr":"172.21.15.76:6833","nonce":2427238559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24491,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/4050689894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":4050689894},{"type":"v1","addr":"172.21.15.72:6837","nonce":4050689894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3681925367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3681925367},{"type":"v1","addr":"172.21.15.136:6835","nonce":3681925367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4251109085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4251109085},{"type":"v1","addr":"172.21.15.76:6835","nonce":4251109085}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T23:43:02.928649+0000","modified":"2024-04-26T23:43:30.190394+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24590,"mds_1":24571,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24514":{"gid":24514,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6837/3639308555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3639308555},{"type":"v1","addr":"172.21.15.136:6837","nonce":3639308555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.76:6837/2506930945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2506930945},{"type":"v1","addr":"172.21.15.76:6837","nonce":2506930945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6839/3938677630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3938677630},{"type":"v1","addr":"172.21.15.136:6839","nonce":3938677630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.72:6839/1403162102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6838","nonce":1403162102},{"type":"v1","addr":"172.21.15.72:6839","nonce":1403162102}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24571":{"gid":24571,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.72:6841/1356345077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6840","nonce":1356345077},{"type":"v1","addr":"172.21.15.72:6841","nonce":1356345077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.76:6839/3405829058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":3405829058},{"type":"v1","addr":"172.21.15.76:6839","nonce":3405829058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24553,24571,24590]},"id":1}]} 2024-04-26T23:45:27.515 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 23 2024-04-26T23:45:27.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.512+0000 7f50d8c00640 1 -- 172.21.15.72:0/3306525954 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f50b80a3cc0 msgr2=0x7f50b80a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:27.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.512+0000 7f50d8c00640 1 --2- 172.21.15.72:0/3306525954 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f50b80a3cc0 0x7f50b80a6180 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f50cc009bd0 tx=0x7f50cc005f90 comp rx=0 tx=0).stop 2024-04-26T23:45:27.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.512+0000 7f50d8c00640 1 -- 172.21.15.72:0/3306525954 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50d4105b20 msgr2=0x7f50d419b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:27.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.512+0000 7f50d8c00640 1 --2- 172.21.15.72:0/3306525954 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50d4105b20 0x7f50d419b090 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f50bc005fd0 tx=0x7f50bc00c6a0 comp rx=0 tx=0).stop 2024-04-26T23:45:27.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.512+0000 7f50d8c00640 1 -- 172.21.15.72:0/3306525954 shutdown_connections 2024-04-26T23:45:27.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.512+0000 7f50d8c00640 1 --2- 172.21.15.72:0/3306525954 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f50b80a3cc0 0x7f50b80a6180 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:27.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.512+0000 7f50d8c00640 1 --2- 172.21.15.72:0/3306525954 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50d4105b20 0x7f50d419b090 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:27.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.512+0000 7f50d8c00640 1 --2- 172.21.15.72:0/3306525954 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50d4108450 0x7f50d41a2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:27.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.512+0000 7f50d8c00640 1 --2- 172.21.15.72:0/3306525954 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f50d41031f0 0x7f50d419ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:27.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.512+0000 7f50d8c00640 1 -- 172.21.15.72:0/3306525954 >> 172.21.15.72:0/3306525954 conn(0x7f50d40fcbc0 msgr2=0x7f50d4102950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:27.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.513+0000 7f50d8c00640 1 -- 172.21.15.72:0/3306525954 shutdown_connections 2024-04-26T23:45:27.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.513+0000 7f50d8c00640 1 -- 172.21.15.72:0/3306525954 wait complete. 2024-04-26T23:45:27.536 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:45:27.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.732+0000 7f542e600640 1 -- 172.21.15.72:0/3377894463 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5428075390 msgr2=0x7f5428075810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:27.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.732+0000 7f542e600640 1 --2- 172.21.15.72:0/3377894463 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5428075390 0x7f5428075810 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5414009900 tx=0x7f541402a3f0 comp rx=0 tx=0).stop 2024-04-26T23:45:27.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.732+0000 7f542e600640 1 -- 172.21.15.72:0/3377894463 shutdown_connections 2024-04-26T23:45:27.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.732+0000 7f542e600640 1 --2- 172.21.15.72:0/3377894463 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f542810ed20 0x7f5428111110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:27.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.732+0000 7f542e600640 1 --2- 172.21.15.72:0/3377894463 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5428075390 0x7f5428075810 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:27.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.732+0000 7f542e600640 1 --2- 172.21.15.72:0/3377894463 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5428076970 0x7f5428074dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:27.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.732+0000 7f542e600640 1 -- 172.21.15.72:0/3377894463 >> 172.21.15.72:0/3377894463 conn(0x7f54280fdcc0 msgr2=0x7f5428100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:27.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.732+0000 7f542e600640 1 -- 172.21.15.72:0/3377894463 shutdown_connections 2024-04-26T23:45:27.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.732+0000 7f542e600640 1 -- 172.21.15.72:0/3377894463 wait complete. 2024-04-26T23:45:27.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.733+0000 7f542e600640 1 Processor -- start 2024-04-26T23:45:27.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.733+0000 7f542e600640 1 -- start start 2024-04-26T23:45:27.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.733+0000 7f542e600640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5428075390 0x7f542819ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:27.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.733+0000 7f542e600640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5428076970 0x7f542819f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:27.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.733+0000 7f542e600640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f542810ed20 0x7f54281a6520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:27.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.733+0000 7f542e600640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f54281148a0 con 0x7f542810ed20 2024-04-26T23:45:27.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.733+0000 7f542e600640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f5428114720 con 0x7f5428076970 2024-04-26T23:45:27.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.733+0000 7f542e600640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f5428114a20 con 0x7f5428075390 2024-04-26T23:45:27.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.734+0000 7f5427e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5428076970 0x7f542819f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:27.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.734+0000 7f5427e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5428076970 0x7f542819f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:52656/0 (socket says 172.21.15.72:52656) 2024-04-26T23:45:27.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.734+0000 7f542d200640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5428075390 0x7f542819ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:27.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.734+0000 7f5427e00640 1 -- 172.21.15.72:0/1995368815 learned_addr learned my addr 172.21.15.72:0/1995368815 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:45:27.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.734+0000 7f542dc00640 1 --2- 172.21.15.72:0/1995368815 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f542810ed20 0x7f54281a6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:27.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.734+0000 7f542d200640 1 -- 172.21.15.72:0/1995368815 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5428076970 msgr2=0x7f542819f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:27.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.734+0000 7f542d200640 1 --2- 172.21.15.72:0/1995368815 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5428076970 0x7f542819f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:27.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.734+0000 7f542d200640 1 -- 172.21.15.72:0/1995368815 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f542810ed20 msgr2=0x7f54281a6520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:27.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.734+0000 7f542d200640 1 --2- 172.21.15.72:0/1995368815 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f542810ed20 0x7f54281a6520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:27.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.734+0000 7f542d200640 1 -- 172.21.15.72:0/1995368815 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54281a6b90 con 0x7f5428075390 2024-04-26T23:45:27.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.734+0000 7f5427e00640 1 --2- 172.21.15.72:0/1995368815 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5428076970 0x7f542819f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:45:27.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.734+0000 7f542d200640 1 --2- 172.21.15.72:0/1995368815 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5428075390 0x7f542819ef70 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f541c00e9f0 tx=0x7f541c00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:27.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.735+0000 7f5425600640 1 -- 172.21.15.72:0/1995368815 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f541c00ccc0 con 0x7f5428075390 2024-04-26T23:45:27.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.735+0000 7f5425600640 1 -- 172.21.15.72:0/1995368815 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f541c0108e0 con 0x7f5428075390 2024-04-26T23:45:27.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.735+0000 7f542e600640 1 -- 172.21.15.72:0/1995368815 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54281a6e80 con 0x7f5428075390 2024-04-26T23:45:27.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.735+0000 7f5425600640 1 -- 172.21.15.72:0/1995368815 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f541c018840 con 0x7f5428075390 2024-04-26T23:45:27.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.735+0000 7f542e600640 1 -- 172.21.15.72:0/1995368815 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54281a72f0 con 0x7f5428075390 2024-04-26T23:45:27.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.736+0000 7f542e600640 1 -- 172.21.15.72:0/1995368815 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5428104ef0 con 0x7f5428075390 2024-04-26T23:45:27.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.738+0000 7f5425600640 1 -- 172.21.15.72:0/1995368815 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f541c010430 con 0x7f5428075390 2024-04-26T23:45:27.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.740+0000 7f5425600640 1 --2- 172.21.15.72:0/1995368815 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f54100a3cc0 0x7f54100a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:27.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.740+0000 7f5425600640 1 -- 172.21.15.72:0/1995368815 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f541c0d0b90 con 0x7f5428075390 2024-04-26T23:45:27.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.740+0000 7f5427e00640 1 --2- 172.21.15.72:0/1995368815 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f54100a3cc0 0x7f54100a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:27.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.740+0000 7f5427e00640 1 --2- 172.21.15.72:0/1995368815 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f54100a3cc0 0x7f54100a6180 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5414000c00 tx=0x7f5414005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:27.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.743+0000 7f5425600640 1 -- 172.21.15.72:0/1995368815 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f541c09cfc0 con 0x7f5428075390 2024-04-26T23:45:27.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:27 smithi072 ceph-mon[28184]: pgmap v397: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:45:27.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:27 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/1092940433' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T23:45:27.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:27 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/2125589396' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T23:45:27.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:27 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/3306525954' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:45:27.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:27 smithi076 ceph-mon[32774]: pgmap v397: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:45:27.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:27 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/1092940433' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T23:45:27.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:27 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/2125589396' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T23:45:27.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:27 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/3306525954' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:45:27.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.951+0000 7f542e600640 1 -- 172.21.15.72:0/1995368815 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f542802dd10 con 0x7f5428075390 2024-04-26T23:45:27.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.953+0000 7f5425600640 1 -- 172.21.15.72:0/1995368815 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24841 (secure 0 0 0) 0x7f541c09c8c0 con 0x7f5428075390 2024-04-26T23:45:27.957 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:45:27.957 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":93,"fsid":"3c8453ae-0425-11ef-bc93-c7b262605968","created":"2024-04-26T23:33:48.088300+0000","modified":"2024-04-26T23:45:00.532006+0000","last_up_change":"2024-04-26T23:42:04.482891+0000","last_in_change":"2024-04-26T23:41:44.784415+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:37:44.428727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:43:00.431829+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:43:01.872977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:43:05.512050+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e20c9ebf-56f6-47c1-a11b-e1e615af042b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6803","nonce":1639298875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6805","nonce":1639298875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6809","nonce":1639298875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6807","nonce":1639298875}]},"public_addr":"172.21.15.72:6803/1639298875","cluster_addr":"172.21.15.72:6805/1639298875","heartbeat_back_addr":"172.21.15.72:6809/1639298875","heartbeat_front_addr":"172.21.15.72:6807/1639298875","state":["exists","up"]},{"osd":1,"uuid":"e6cac469-780c-4d66-b560-b686072371fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6801","nonce":4185342193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6803","nonce":4185342193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6807","nonce":4185342193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6805","nonce":4185342193}]},"public_addr":"172.21.15.76:6801/4185342193","cluster_addr":"172.21.15.76:6803/4185342193","heartbeat_back_addr":"172.21.15.76:6807/4185342193","heartbeat_front_addr":"172.21.15.76:6805/4185342193","state":["exists","up"]},{"osd":2,"uuid":"838a49fc-18de-446c-8336-67b3e05da231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6801","nonce":3033860799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6803","nonce":3033860799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6807","nonce":3033860799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6805","nonce":3033860799}]},"public_addr":"172.21.15.136:6801/3033860799","cluster_addr":"172.21.15.136:6803/3033860799","heartbeat_back_addr":"172.21.15.136:6807/3033860799","heartbeat_front_addr":"172.21.15.136:6805/3033860799","state":["exists","up"]},{"osd":3,"uuid":"e6da89be-5ba7-48a4-9306-f75498660e0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6811","nonce":2071107815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6813","nonce":2071107815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6817","nonce":2071107815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6815","nonce":2071107815}]},"public_addr":"172.21.15.72:6811/2071107815","cluster_addr":"172.21.15.72:6813/2071107815","heartbeat_back_addr":"172.21.15.72:6817/2071107815","heartbeat_front_addr":"172.21.15.72:6815/2071107815","state":["exists","up"]},{"osd":4,"uuid":"0b17c491-d798-488f-892d-521ef2fbc51e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6809","nonce":147449238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6811","nonce":147449238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6815","nonce":147449238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6813","nonce":147449238}]},"public_addr":"172.21.15.76:6809/147449238","cluster_addr":"172.21.15.76:6811/147449238","heartbeat_back_addr":"172.21.15.76:6815/147449238","heartbeat_front_addr":"172.21.15.76:6813/147449238","state":["exists","up"]},{"osd":5,"uuid":"c7e887b3-ab38-4608-9bfd-1ed49956e8af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6809","nonce":2320149212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6811","nonce":2320149212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6815","nonce":2320149212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6813","nonce":2320149212}]},"public_addr":"172.21.15.136:6809/2320149212","cluster_addr":"172.21.15.136:6811/2320149212","heartbeat_back_addr":"172.21.15.136:6815/2320149212","heartbeat_front_addr":"172.21.15.136:6813/2320149212","state":["exists","up"]},{"osd":6,"uuid":"f0bd604b-0f57-4a38-9295-eb200264f1ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6819","nonce":389499207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6821","nonce":389499207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6825","nonce":389499207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6823","nonce":389499207}]},"public_addr":"172.21.15.72:6819/389499207","cluster_addr":"172.21.15.72:6821/389499207","heartbeat_back_addr":"172.21.15.72:6825/389499207","heartbeat_front_addr":"172.21.15.72:6823/389499207","state":["exists","up"]},{"osd":7,"uuid":"1879d4e4-038e-4cd6-a3ec-3a940bcc3797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6817","nonce":375888575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6819","nonce":375888575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6823","nonce":375888575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6821","nonce":375888575}]},"public_addr":"172.21.15.76:6817/375888575","cluster_addr":"172.21.15.76:6819/375888575","heartbeat_back_addr":"172.21.15.76:6823/375888575","heartbeat_front_addr":"172.21.15.76:6821/375888575","state":["exists","up"]},{"osd":8,"uuid":"6f404062-97e7-4ccb-a582-76dc1ffae548","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6817","nonce":3446774052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6819","nonce":3446774052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6823","nonce":3446774052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6821","nonce":3446774052}]},"public_addr":"172.21.15.136:6817/3446774052","cluster_addr":"172.21.15.136:6819/3446774052","heartbeat_back_addr":"172.21.15.136:6823/3446774052","heartbeat_front_addr":"172.21.15.136:6821/3446774052","state":["exists","up"]},{"osd":9,"uuid":"2989ca68-3164-4667-9bcf-230dcbb7b8f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6827","nonce":3423789152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6829","nonce":3423789152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6832","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6833","nonce":3423789152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6831","nonce":3423789152}]},"public_addr":"172.21.15.72:6827/3423789152","cluster_addr":"172.21.15.72:6829/3423789152","heartbeat_back_addr":"172.21.15.72:6833/3423789152","heartbeat_front_addr":"172.21.15.72:6831/3423789152","state":["exists","up"]},{"osd":10,"uuid":"9ec6d46a-0913-4208-97b5-167188ee8cb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6825","nonce":1120991574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6827","nonce":1120991574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6831","nonce":1120991574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6829","nonce":1120991574}]},"public_addr":"172.21.15.76:6825/1120991574","cluster_addr":"172.21.15.76:6827/1120991574","heartbeat_back_addr":"172.21.15.76:6831/1120991574","heartbeat_front_addr":"172.21.15.76:6829/1120991574","state":["exists","up"]},{"osd":11,"uuid":"bdd0774f-6e58-4918-b4cc-afd4f1118d3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6825","nonce":1135995193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6827","nonce":1135995193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6831","nonce":1135995193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6829","nonce":1135995193}]},"public_addr":"172.21.15.136:6825/1135995193","cluster_addr":"172.21.15.136:6827/1135995193","heartbeat_back_addr":"172.21.15.136:6831/1135995193","heartbeat_front_addr":"172.21.15.136:6829/1135995193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:36:52.128788+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:37:17.523654+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:37:41.634677+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:38:15.060797+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:38:41.234111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:39:06.783560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:39:40.884386+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:40:08.340465+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:40:34.344858+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:41:08.525309+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:41:36.782624+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:42:02.756888+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":6}]},{"pgid":"3.37","mappings":[{"from":5,"to":7}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:0/3625071330":"2024-04-27T23:34:58.266218+0000","172.21.15.72:6801/2903979134":"2024-04-27T23:34:58.266218+0000","172.21.15.72:6800/2903979134":"2024-04-27T23:34:58.266218+0000","172.21.15.72:0/1519142576":"2024-04-27T23:34:58.266218+0000","172.21.15.72:0/3669626460":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/945623297":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/3479150073":"2024-04-27T23:34:24.233319+0000","172.21.15.72:6800/4175896290":"2024-04-27T23:34:24.233319+0000","172.21.15.72:6801/4175896290":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/3783175012":"2024-04-27T23:34:58.266218+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:45:27.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.956+0000 7f542e600640 1 -- 172.21.15.72:0/1995368815 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f54100a3cc0 msgr2=0x7f54100a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:27.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.956+0000 7f542e600640 1 --2- 172.21.15.72:0/1995368815 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f54100a3cc0 0x7f54100a6180 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5414000c00 tx=0x7f5414005a20 comp rx=0 tx=0).stop 2024-04-26T23:45:27.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.956+0000 7f542e600640 1 -- 172.21.15.72:0/1995368815 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5428075390 msgr2=0x7f542819ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:27.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.956+0000 7f542e600640 1 --2- 172.21.15.72:0/1995368815 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5428075390 0x7f542819ef70 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f541c00e9f0 tx=0x7f541c00edf0 comp rx=0 tx=0).stop 2024-04-26T23:45:27.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.956+0000 7f542e600640 1 -- 172.21.15.72:0/1995368815 shutdown_connections 2024-04-26T23:45:27.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.956+0000 7f542e600640 1 --2- 172.21.15.72:0/1995368815 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f54100a3cc0 0x7f54100a6180 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:27.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.956+0000 7f542e600640 1 --2- 172.21.15.72:0/1995368815 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5428076970 0x7f542819f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:27.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.956+0000 7f542e600640 1 --2- 172.21.15.72:0/1995368815 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f542810ed20 0x7f54281a6520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:27.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.956+0000 7f542e600640 1 --2- 172.21.15.72:0/1995368815 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5428075390 0x7f542819ef70 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:27.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.956+0000 7f542e600640 1 -- 172.21.15.72:0/1995368815 >> 172.21.15.72:0/1995368815 conn(0x7f54280fdcc0 msgr2=0x7f54281020c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:27.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.956+0000 7f542e600640 1 -- 172.21.15.72:0/1995368815 shutdown_connections 2024-04-26T23:45:27.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:27.956+0000 7f542e600640 1 -- 172.21.15.72:0/1995368815 wait complete. 2024-04-26T23:45:27.979 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:37:44.428727+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T23:43:00.431829+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T23:43:01.872977+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T23:43:05.512050+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-26T23:45:27.979 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:45:28.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:27 smithi136 ceph-mon[33050]: pgmap v397: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:45:28.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:27 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/1092940433' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T23:45:28.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:27 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/2125589396' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T23:45:28.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:27 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/3306525954' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:45:28.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.180+0000 7f3043200640 1 -- 172.21.15.72:0/3260775315 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f303c109fc0 msgr2=0x7f303c10a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:28.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.180+0000 7f3043200640 1 --2- 172.21.15.72:0/3260775315 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f303c109fc0 0x7f303c10a440 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f302c009900 tx=0x7f302c02a3f0 comp rx=0 tx=0).stop 2024-04-26T23:45:28.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.180+0000 7f3043200640 1 -- 172.21.15.72:0/3260775315 shutdown_connections 2024-04-26T23:45:28.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.180+0000 7f3043200640 1 --2- 172.21.15.72:0/3260775315 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f303c10a980 0x7f303c1112a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:28.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.180+0000 7f3043200640 1 --2- 172.21.15.72:0/3260775315 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f303c109fc0 0x7f303c10a440 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:28.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.180+0000 7f3043200640 1 --2- 172.21.15.72:0/3260775315 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f303c108e70 0x7f303c109290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:28.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.180+0000 7f3043200640 1 -- 172.21.15.72:0/3260775315 >> 172.21.15.72:0/3260775315 conn(0x7f303c077e00 msgr2=0x7f303c07a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:28.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.180+0000 7f3043200640 1 -- 172.21.15.72:0/3260775315 shutdown_connections 2024-04-26T23:45:28.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.180+0000 7f3043200640 1 -- 172.21.15.72:0/3260775315 wait complete. 2024-04-26T23:45:28.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.181+0000 7f3043200640 1 Processor -- start 2024-04-26T23:45:28.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.181+0000 7f3043200640 1 -- start start 2024-04-26T23:45:28.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.181+0000 7f3043200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f303c108e70 0x7f303c19f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:28.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.181+0000 7f3043200640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f303c109fc0 0x7f303c19f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:28.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.181+0000 7f3043200640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f303c10a980 0x7f303c1a6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:28.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.181+0000 7f3043200640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f303c114a10 con 0x7f303c109fc0 2024-04-26T23:45:28.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.181+0000 7f3043200640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f303c114890 con 0x7f303c108e70 2024-04-26T23:45:28.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.181+0000 7f3043200640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f303c114b90 con 0x7f303c10a980 2024-04-26T23:45:28.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.182+0000 7f3041400640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f303c109fc0 0x7f303c19f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:28.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.182+0000 7f3041400640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f303c109fc0 0x7f303c19f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54068/0 (socket says 172.21.15.72:54068) 2024-04-26T23:45:28.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.182+0000 7f3041400640 1 -- 172.21.15.72:0/348120265 learned_addr learned my addr 172.21.15.72:0/348120265 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:45:28.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.182+0000 7f3041e00640 1 --2- 172.21.15.72:0/348120265 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f303c108e70 0x7f303c19f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:28.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.182+0000 7f3042800640 1 --2- 172.21.15.72:0/348120265 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f303c10a980 0x7f303c1a6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:28.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.182+0000 7f3041400640 1 -- 172.21.15.72:0/348120265 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f303c10a980 msgr2=0x7f303c1a6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:28.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.182+0000 7f3041400640 1 --2- 172.21.15.72:0/348120265 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f303c10a980 0x7f303c1a6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:28.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.182+0000 7f3041400640 1 -- 172.21.15.72:0/348120265 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f303c108e70 msgr2=0x7f303c19f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:28.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.182+0000 7f3041400640 1 --2- 172.21.15.72:0/348120265 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f303c108e70 0x7f303c19f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:28.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.182+0000 7f3041400640 1 -- 172.21.15.72:0/348120265 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f303c1a6cb0 con 0x7f303c109fc0 2024-04-26T23:45:28.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.182+0000 7f3041e00640 1 --2- 172.21.15.72:0/348120265 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f303c108e70 0x7f303c19f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:45:28.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.182+0000 7f3041400640 1 --2- 172.21.15.72:0/348120265 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f303c109fc0 0x7f303c19f5d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f302c031500 tx=0x7f302c003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:28.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.183+0000 7f3036a00640 1 -- 172.21.15.72:0/348120265 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f302c03f070 con 0x7f303c109fc0 2024-04-26T23:45:28.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.183+0000 7f3036a00640 1 -- 172.21.15.72:0/348120265 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f302c031c50 con 0x7f303c109fc0 2024-04-26T23:45:28.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.183+0000 7f3036a00640 1 -- 172.21.15.72:0/348120265 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f302c035030 con 0x7f303c109fc0 2024-04-26T23:45:28.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.183+0000 7f3043200640 1 -- 172.21.15.72:0/348120265 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f303c1a6f40 con 0x7f303c109fc0 2024-04-26T23:45:28.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.183+0000 7f3043200640 1 -- 172.21.15.72:0/348120265 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f303c1a7340 con 0x7f303c109fc0 2024-04-26T23:45:28.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.184+0000 7f3043200640 1 -- 172.21.15.72:0/348120265 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f303c110180 con 0x7f303c109fc0 2024-04-26T23:45:28.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.187+0000 7f3036a00640 1 -- 172.21.15.72:0/348120265 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f302c042070 con 0x7f303c109fc0 2024-04-26T23:45:28.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.188+0000 7f3036a00640 1 --2- 172.21.15.72:0/348120265 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f30240a3c40 0x7f30240a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:28.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.189+0000 7f3036a00640 1 -- 172.21.15.72:0/348120265 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f302c0f17b0 con 0x7f303c109fc0 2024-04-26T23:45:28.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.189+0000 7f3041e00640 1 --2- 172.21.15.72:0/348120265 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f30240a3c40 0x7f30240a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:28.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.189+0000 7f3041e00640 1 --2- 172.21.15.72:0/348120265 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f30240a3c40 0x7f30240a6100 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3030009fc0 tx=0x7f3030005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:28.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.191+0000 7f3036a00640 1 -- 172.21.15.72:0/348120265 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f302c0bd590 con 0x7f303c109fc0 2024-04-26T23:45:28.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.378+0000 7f3043200640 1 -- 172.21.15.72:0/348120265 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f303c02dd20 con 0x7f303c109fc0 2024-04-26T23:45:28.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.379+0000 7f3036a00640 1 -- 172.21.15.72:0/348120265 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f302c0c1890 con 0x7f303c109fc0 2024-04-26T23:45:28.382 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-04-26T23:45:28.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.381+0000 7f3043200640 1 -- 172.21.15.72:0/348120265 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f30240a3c40 msgr2=0x7f30240a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:28.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.381+0000 7f3043200640 1 --2- 172.21.15.72:0/348120265 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f30240a3c40 0x7f30240a6100 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3030009fc0 tx=0x7f3030005f90 comp rx=0 tx=0).stop 2024-04-26T23:45:28.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.381+0000 7f3043200640 1 -- 172.21.15.72:0/348120265 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f303c109fc0 msgr2=0x7f303c19f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:28.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.381+0000 7f3043200640 1 --2- 172.21.15.72:0/348120265 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f303c109fc0 0x7f303c19f5d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f302c031500 tx=0x7f302c003c10 comp rx=0 tx=0).stop 2024-04-26T23:45:28.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.381+0000 7f3043200640 1 -- 172.21.15.72:0/348120265 shutdown_connections 2024-04-26T23:45:28.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.381+0000 7f3043200640 1 --2- 172.21.15.72:0/348120265 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f30240a3c40 0x7f30240a6100 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:28.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.381+0000 7f3043200640 1 --2- 172.21.15.72:0/348120265 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f303c10a980 0x7f303c1a6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:28.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.381+0000 7f3043200640 1 --2- 172.21.15.72:0/348120265 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f303c109fc0 0x7f303c19f5d0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:28.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.381+0000 7f3043200640 1 --2- 172.21.15.72:0/348120265 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f303c108e70 0x7f303c19f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:28.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.381+0000 7f3043200640 1 -- 172.21.15.72:0/348120265 >> 172.21.15.72:0/348120265 conn(0x7f303c077e00 msgr2=0x7f303c10d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:28.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.382+0000 7f3043200640 1 -- 172.21.15.72:0/348120265 shutdown_connections 2024-04-26T23:45:28.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.382+0000 7f3043200640 1 -- 172.21.15.72:0/348120265 wait complete. 2024-04-26T23:45:28.404 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T23:45:28.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.599+0000 7f5a33e00640 1 -- 172.21.15.72:0/2799939460 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5a34103320 msgr2=0x7f5a34105710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:28.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.599+0000 7f5a33e00640 1 --2- 172.21.15.72:0/2799939460 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5a34103320 0x7f5a34105710 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5a24009900 tx=0x7f5a2402a3f0 comp rx=0 tx=0).stop 2024-04-26T23:45:28.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.599+0000 7f5a33e00640 1 -- 172.21.15.72:0/2799939460 shutdown_connections 2024-04-26T23:45:28.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.599+0000 7f5a33e00640 1 --2- 172.21.15.72:0/2799939460 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5a34105c50 0x7f5a34108040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:28.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.599+0000 7f5a33e00640 1 --2- 172.21.15.72:0/2799939460 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a34108580 0x7f5a3410a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:28.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.599+0000 7f5a33e00640 1 --2- 172.21.15.72:0/2799939460 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5a34103320 0x7f5a34105710 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:28.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.599+0000 7f5a33e00640 1 -- 172.21.15.72:0/2799939460 >> 172.21.15.72:0/2799939460 conn(0x7f5a340fccd0 msgr2=0x7f5a340ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:28.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.599+0000 7f5a33e00640 1 -- 172.21.15.72:0/2799939460 shutdown_connections 2024-04-26T23:45:28.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.599+0000 7f5a33e00640 1 -- 172.21.15.72:0/2799939460 wait complete. 2024-04-26T23:45:28.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.600+0000 7f5a33e00640 1 Processor -- start 2024-04-26T23:45:28.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.601+0000 7f5a33e00640 1 -- start start 2024-04-26T23:45:28.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.601+0000 7f5a33e00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a34103320 0x7f5a3419ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:28.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.601+0000 7f5a33e00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5a34105c50 0x7f5a3419b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:28.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.601+0000 7f5a33e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5a34108580 0x7f5a341a2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:28.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.601+0000 7f5a33e00640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f5a3410e3e0 con 0x7f5a34105c50 2024-04-26T23:45:28.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.601+0000 7f5a33e00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f5a3410e260 con 0x7f5a34108580 2024-04-26T23:45:28.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.601+0000 7f5a33e00640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f5a3410e560 con 0x7f5a34103320 2024-04-26T23:45:28.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.601+0000 7f5a32a00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a34103320 0x7f5a3419ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:28.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.601+0000 7f5a32000640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5a34105c50 0x7f5a3419b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:28.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.601+0000 7f5a33400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5a34108580 0x7f5a341a2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:28.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.601+0000 7f5a32a00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a34103320 0x7f5a3419ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.72:44430/0 (socket says 172.21.15.72:44430) 2024-04-26T23:45:28.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.601+0000 7f5a32a00640 1 -- 172.21.15.72:0/2220524562 learned_addr learned my addr 172.21.15.72:0/2220524562 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:45:28.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.602+0000 7f5a33400640 1 -- 172.21.15.72:0/2220524562 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a34103320 msgr2=0x7f5a3419ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:28.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.602+0000 7f5a33400640 1 --2- 172.21.15.72:0/2220524562 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a34103320 0x7f5a3419ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:28.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.602+0000 7f5a33400640 1 -- 172.21.15.72:0/2220524562 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5a34105c50 msgr2=0x7f5a3419b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:28.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.602+0000 7f5a33400640 1 --2- 172.21.15.72:0/2220524562 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5a34105c50 0x7f5a3419b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:28.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.602+0000 7f5a33400640 1 -- 172.21.15.72:0/2220524562 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a341a28b0 con 0x7f5a34108580 2024-04-26T23:45:28.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.602+0000 7f5a33400640 1 --2- 172.21.15.72:0/2220524562 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5a34108580 0x7f5a341a2210 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c009fd0 tx=0x7f5a2c00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:28.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.602+0000 7f5a2b400640 1 -- 172.21.15.72:0/2220524562 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a2c00ce20 con 0x7f5a34108580 2024-04-26T23:45:28.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.602+0000 7f5a2b400640 1 -- 172.21.15.72:0/2220524562 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a2c005020 con 0x7f5a34108580 2024-04-26T23:45:28.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.602+0000 7f5a33e00640 1 -- 172.21.15.72:0/2220524562 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a341a2ba0 con 0x7f5a34108580 2024-04-26T23:45:28.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.602+0000 7f5a2b400640 1 -- 172.21.15.72:0/2220524562 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a2c010690 con 0x7f5a34108580 2024-04-26T23:45:28.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.602+0000 7f5a33e00640 1 -- 172.21.15.72:0/2220524562 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a341a3140 con 0x7f5a34108580 2024-04-26T23:45:28.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.603+0000 7f5a33e00640 1 -- 172.21.15.72:0/2220524562 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a3410a070 con 0x7f5a34108580 2024-04-26T23:45:28.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.605+0000 7f5a2b400640 1 -- 172.21.15.72:0/2220524562 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5a2c0054d0 con 0x7f5a34108580 2024-04-26T23:45:28.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.606+0000 7f5a2b400640 1 --2- 172.21.15.72:0/2220524562 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f5a180a3d00 0x7f5a180a61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:28.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.607+0000 7f5a2b400640 1 -- 172.21.15.72:0/2220524562 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f5a2c0d0fd0 con 0x7f5a34108580 2024-04-26T23:45:28.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.607+0000 7f5a32a00640 1 --2- 172.21.15.72:0/2220524562 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f5a180a3d00 0x7f5a180a61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:28.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.607+0000 7f5a32a00640 1 --2- 172.21.15.72:0/2220524562 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f5a180a3d00 0x7f5a180a61c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f5a240096c0 tx=0x7f5a240314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:28.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.611+0000 7f5a2b400640 1 -- 172.21.15.72:0/2220524562 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5a2c09d400 con 0x7f5a34108580 2024-04-26T23:45:28.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:28 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/1995368815' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:45:28.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:28 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/348120265' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T23:45:28.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.799+0000 7f5a33e00640 1 -- 172.21.15.72:0/2220524562 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5a3402dc80 con 0x7f5a34108580 2024-04-26T23:45:28.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.800+0000 7f5a2b400640 1 -- 172.21.15.72:0/2220524562 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f5a2c09cd00 con 0x7f5a34108580 2024-04-26T23:45:28.804 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-26T23:45:28.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.803+0000 7f5a33e00640 1 -- 172.21.15.72:0/2220524562 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f5a180a3d00 msgr2=0x7f5a180a61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:28.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.803+0000 7f5a33e00640 1 --2- 172.21.15.72:0/2220524562 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f5a180a3d00 0x7f5a180a61c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f5a240096c0 tx=0x7f5a240314a0 comp rx=0 tx=0).stop 2024-04-26T23:45:28.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.803+0000 7f5a33e00640 1 -- 172.21.15.72:0/2220524562 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5a34108580 msgr2=0x7f5a341a2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:28.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.803+0000 7f5a33e00640 1 --2- 172.21.15.72:0/2220524562 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5a34108580 0x7f5a341a2210 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c009fd0 tx=0x7f5a2c00ef10 comp rx=0 tx=0).stop 2024-04-26T23:45:28.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.803+0000 7f5a33e00640 1 -- 172.21.15.72:0/2220524562 shutdown_connections 2024-04-26T23:45:28.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.803+0000 7f5a33e00640 1 --2- 172.21.15.72:0/2220524562 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f5a180a3d00 0x7f5a180a61c0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:28.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.803+0000 7f5a33e00640 1 --2- 172.21.15.72:0/2220524562 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5a34108580 0x7f5a341a2210 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:28.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.803+0000 7f5a33e00640 1 --2- 172.21.15.72:0/2220524562 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5a34105c50 0x7f5a3419b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:28.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.803+0000 7f5a33e00640 1 --2- 172.21.15.72:0/2220524562 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5a34103320 0x7f5a3419ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:28.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.803+0000 7f5a33e00640 1 -- 172.21.15.72:0/2220524562 >> 172.21.15.72:0/2220524562 conn(0x7f5a340fccd0 msgr2=0x7f5a341029d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:28.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.804+0000 7f5a33e00640 1 -- 172.21.15.72:0/2220524562 shutdown_connections 2024-04-26T23:45:28.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:28.804+0000 7f5a33e00640 1 -- 172.21.15.72:0/2220524562 wait complete. 2024-04-26T23:45:28.826 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T23:45:28.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:28 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/1995368815' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:45:28.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:28 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/348120265' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T23:45:29.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.011+0000 7f53a5a00640 1 -- 172.21.15.72:0/862026667 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53a010ab40 msgr2=0x7f53a01113d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:29.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.011+0000 7f53a5a00640 1 --2- 172.21.15.72:0/862026667 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53a010ab40 0x7f53a01113d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5398009ca0 tx=0x7f539802a8c0 comp rx=0 tx=0).stop 2024-04-26T23:45:29.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.011+0000 7f53a5a00640 1 -- 172.21.15.72:0/862026667 shutdown_connections 2024-04-26T23:45:29.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.011+0000 7f53a5a00640 1 --2- 172.21.15.72:0/862026667 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53a010ab40 0x7f53a01113d0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:29.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.011+0000 7f53a5a00640 1 --2- 172.21.15.72:0/862026667 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53a010a180 0x7f53a010a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:29.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.011+0000 7f53a5a00640 1 --2- 172.21.15.72:0/862026667 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f53a0108f90 0x7f53a01093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:29.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.011+0000 7f53a5a00640 1 -- 172.21.15.72:0/862026667 >> 172.21.15.72:0/862026667 conn(0x7f53a0077f30 msgr2=0x7f53a007a370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:29.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.012+0000 7f53a5a00640 1 -- 172.21.15.72:0/862026667 shutdown_connections 2024-04-26T23:45:29.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.012+0000 7f53a5a00640 1 -- 172.21.15.72:0/862026667 wait complete. 2024-04-26T23:45:29.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.012+0000 7f53a5a00640 1 Processor -- start 2024-04-26T23:45:29.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.013+0000 7f53a5a00640 1 -- start start 2024-04-26T23:45:29.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.013+0000 7f53a5a00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53a0108f90 0x7f53a019f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:29.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.013+0000 7f53a5a00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53a010a180 0x7f53a019f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:29.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.013+0000 7f53a5a00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f53a010ab40 0x7f53a01a66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:29.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.013+0000 7f53a5a00640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f53a0114ad0 con 0x7f53a010a180 2024-04-26T23:45:29.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.013+0000 7f53a5a00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f53a0114950 con 0x7f53a010ab40 2024-04-26T23:45:29.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.013+0000 7f53a5a00640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f53a0114c50 con 0x7f53a0108f90 2024-04-26T23:45:29.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.013+0000 7f5397e00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53a010a180 0x7f53a019f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:29.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.013+0000 7f5397e00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53a010a180 0x7f53a019f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54100/0 (socket says 172.21.15.72:54100) 2024-04-26T23:45:29.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.013+0000 7f5397e00640 1 -- 172.21.15.72:0/1891441562 learned_addr learned my addr 172.21.15.72:0/1891441562 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:45:29.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.013+0000 7f539ea00640 1 --2- 172.21.15.72:0/1891441562 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53a0108f90 0x7f53a019f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:29.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.013+0000 7f539f400640 1 --2- 172.21.15.72:0/1891441562 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f53a010ab40 0x7f53a01a66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:29.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.014+0000 7f5397e00640 1 -- 172.21.15.72:0/1891441562 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53a0108f90 msgr2=0x7f53a019f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:29.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.014+0000 7f5397e00640 1 --2- 172.21.15.72:0/1891441562 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53a0108f90 0x7f53a019f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:29.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.014+0000 7f5397e00640 1 -- 172.21.15.72:0/1891441562 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f53a010ab40 msgr2=0x7f53a01a66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:29.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.014+0000 7f5397e00640 1 --2- 172.21.15.72:0/1891441562 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f53a010ab40 0x7f53a01a66e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:29.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.014+0000 7f5397e00640 1 -- 172.21.15.72:0/1891441562 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53a01a6d50 con 0x7f53a010a180 2024-04-26T23:45:29.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.014+0000 7f5397e00640 1 --2- 172.21.15.72:0/1891441562 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53a010a180 0x7f53a019f670 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f538800e9f0 tx=0x7f538800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:29.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.014+0000 7f5397400640 1 -- 172.21.15.72:0/1891441562 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f538800cd80 con 0x7f53a010a180 2024-04-26T23:45:29.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.014+0000 7f5397400640 1 -- 172.21.15.72:0/1891441562 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53880054d0 con 0x7f53a010a180 2024-04-26T23:45:29.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.014+0000 7f5397400640 1 -- 172.21.15.72:0/1891441562 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5388010690 con 0x7f53a010a180 2024-04-26T23:45:29.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.014+0000 7f53a5a00640 1 -- 172.21.15.72:0/1891441562 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53a01a6fe0 con 0x7f53a010a180 2024-04-26T23:45:29.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.014+0000 7f53a5a00640 1 -- 172.21.15.72:0/1891441562 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53a01a7500 con 0x7f53a010a180 2024-04-26T23:45:29.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.015+0000 7f53a5a00640 1 -- 172.21.15.72:0/1891441562 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53a01102b0 con 0x7f53a010a180 2024-04-26T23:45:29.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.017+0000 7f5397400640 1 -- 172.21.15.72:0/1891441562 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5388004e30 con 0x7f53a010a180 2024-04-26T23:45:29.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.019+0000 7f5397400640 1 --2- 172.21.15.72:0/1891441562 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f53840a3cc0 0x7f53840a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:29.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.020+0000 7f5397400640 1 -- 172.21.15.72:0/1891441562 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f53880d0f20 con 0x7f53a010a180 2024-04-26T23:45:29.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.020+0000 7f539ea00640 1 --2- 172.21.15.72:0/1891441562 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f53840a3cc0 0x7f53840a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:29.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.020+0000 7f539ea00640 1 --2- 172.21.15.72:0/1891441562 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f53840a3cc0 0x7f53840a6180 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f53900095f0 tx=0x7f5390005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:29.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.023+0000 7f5397400640 1 -- 172.21.15.72:0/1891441562 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5388054060 con 0x7f53a010a180 2024-04-26T23:45:29.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:28 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/1995368815' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:45:29.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:28 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/348120265' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T23:45:29.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.223+0000 7f53a5a00640 1 -- 172.21.15.72:0/1891441562 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f53a002dd20 con 0x7f53a010a180 2024-04-26T23:45:29.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.224+0000 7f5397400640 1 -- 172.21.15.72:0/1891441562 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f538809d350 con 0x7f53a010a180 2024-04-26T23:45:29.227 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-04-26T23:45:29.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.226+0000 7f53a5a00640 1 -- 172.21.15.72:0/1891441562 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f53840a3cc0 msgr2=0x7f53840a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:29.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.227+0000 7f53a5a00640 1 --2- 172.21.15.72:0/1891441562 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f53840a3cc0 0x7f53840a6180 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f53900095f0 tx=0x7f5390005f90 comp rx=0 tx=0).stop 2024-04-26T23:45:29.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.227+0000 7f53a5a00640 1 -- 172.21.15.72:0/1891441562 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53a010a180 msgr2=0x7f53a019f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:29.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.227+0000 7f53a5a00640 1 --2- 172.21.15.72:0/1891441562 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53a010a180 0x7f53a019f670 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f538800e9f0 tx=0x7f538800edf0 comp rx=0 tx=0).stop 2024-04-26T23:45:29.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.227+0000 7f53a5a00640 1 -- 172.21.15.72:0/1891441562 shutdown_connections 2024-04-26T23:45:29.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.227+0000 7f53a5a00640 1 --2- 172.21.15.72:0/1891441562 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f53840a3cc0 0x7f53840a6180 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:29.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.227+0000 7f53a5a00640 1 --2- 172.21.15.72:0/1891441562 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f53a010ab40 0x7f53a01a66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:29.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.227+0000 7f53a5a00640 1 --2- 172.21.15.72:0/1891441562 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53a010a180 0x7f53a019f670 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:29.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.227+0000 7f53a5a00640 1 --2- 172.21.15.72:0/1891441562 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53a0108f90 0x7f53a019f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:29.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.227+0000 7f53a5a00640 1 -- 172.21.15.72:0/1891441562 >> 172.21.15.72:0/1891441562 conn(0x7f53a0077f30 msgr2=0x7f53a010d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:29.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.227+0000 7f53a5a00640 1 -- 172.21.15.72:0/1891441562 shutdown_connections 2024-04-26T23:45:29.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.227+0000 7f53a5a00640 1 -- 172.21.15.72:0/1891441562 wait complete. 2024-04-26T23:45:29.250 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-26T23:45:29.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.457+0000 7fda2e000640 1 -- 172.21.15.72:0/4079498750 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda28106740 msgr2=0x7fda2810cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:29.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.457+0000 7fda2e000640 1 --2- 172.21.15.72:0/4079498750 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda28106740 0x7fda2810cfd0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fda20009ca0 tx=0x7fda2002a8b0 comp rx=0 tx=0).stop 2024-04-26T23:45:29.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.457+0000 7fda2e000640 1 -- 172.21.15.72:0/4079498750 shutdown_connections 2024-04-26T23:45:29.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.457+0000 7fda2e000640 1 --2- 172.21.15.72:0/4079498750 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda28106740 0x7fda2810cfd0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:29.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.457+0000 7fda2e000640 1 --2- 172.21.15.72:0/4079498750 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fda28105d80 0x7fda28106200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:29.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.457+0000 7fda2e000640 1 --2- 172.21.15.72:0/4079498750 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda28104b90 0x7fda28104fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:29.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.457+0000 7fda2e000640 1 -- 172.21.15.72:0/4079498750 >> 172.21.15.72:0/4079498750 conn(0x7fda281000e0 msgr2=0x7fda28102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:29.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.457+0000 7fda2e000640 1 -- 172.21.15.72:0/4079498750 shutdown_connections 2024-04-26T23:45:29.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.457+0000 7fda2e000640 1 -- 172.21.15.72:0/4079498750 wait complete. 2024-04-26T23:45:29.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.458+0000 7fda2e000640 1 Processor -- start 2024-04-26T23:45:29.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.458+0000 7fda2e000640 1 -- start start 2024-04-26T23:45:29.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.459+0000 7fda2e000640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fda28104b90 0x7fda2819ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:29.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.459+0000 7fda2e000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda28105d80 0x7fda2819b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:29.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.459+0000 7fda2e000640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda28106740 0x7fda281a2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:29.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.459+0000 7fda2e000640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fda28110710 con 0x7fda28104b90 2024-04-26T23:45:29.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.459+0000 7fda2e000640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fda28110590 con 0x7fda28105d80 2024-04-26T23:45:29.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.459+0000 7fda2e000640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fda28110890 con 0x7fda28106740 2024-04-26T23:45:29.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.459+0000 7fda27400640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fda28104b90 0x7fda2819ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:29.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.459+0000 7fda27400640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fda28104b90 0x7fda2819ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54132/0 (socket says 172.21.15.72:54132) 2024-04-26T23:45:29.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.459+0000 7fda27e00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda28106740 0x7fda281a2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:29.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.459+0000 7fda27400640 1 -- 172.21.15.72:0/2263244780 learned_addr learned my addr 172.21.15.72:0/2263244780 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:45:29.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.459+0000 7fda26a00640 1 --2- 172.21.15.72:0/2263244780 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda28105d80 0x7fda2819b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:29.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.460+0000 7fda27e00640 1 -- 172.21.15.72:0/2263244780 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda28105d80 msgr2=0x7fda2819b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:29.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.460+0000 7fda27e00640 1 --2- 172.21.15.72:0/2263244780 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda28105d80 0x7fda2819b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:29.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.460+0000 7fda27e00640 1 -- 172.21.15.72:0/2263244780 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fda28104b90 msgr2=0x7fda2819ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:29.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.460+0000 7fda27e00640 1 --2- 172.21.15.72:0/2263244780 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fda28104b90 0x7fda2819ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:29.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.460+0000 7fda27e00640 1 -- 172.21.15.72:0/2263244780 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda281a2980 con 0x7fda28106740 2024-04-26T23:45:29.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.460+0000 7fda27e00640 1 --2- 172.21.15.72:0/2263244780 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda28106740 0x7fda281a2310 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fda2002aea0 tx=0x7fda20003ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:29.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.460+0000 7fda1fe00640 1 -- 172.21.15.72:0/2263244780 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda20004240 con 0x7fda28106740 2024-04-26T23:45:29.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.460+0000 7fda1fe00640 1 -- 172.21.15.72:0/2263244780 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda20040070 con 0x7fda28106740 2024-04-26T23:45:29.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.460+0000 7fda2e000640 1 -- 172.21.15.72:0/2263244780 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda281a2c10 con 0x7fda28106740 2024-04-26T23:45:29.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.460+0000 7fda1fe00640 1 -- 172.21.15.72:0/2263244780 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda20045af0 con 0x7fda28106740 2024-04-26T23:45:29.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.460+0000 7fda2e000640 1 -- 172.21.15.72:0/2263244780 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda281a3160 con 0x7fda28106740 2024-04-26T23:45:29.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.461+0000 7fda2e000640 1 -- 172.21.15.72:0/2263244780 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda2810beb0 con 0x7fda28106740 2024-04-26T23:45:29.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.463+0000 7fda1fe00640 1 -- 172.21.15.72:0/2263244780 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fda20004b40 con 0x7fda28106740 2024-04-26T23:45:29.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.465+0000 7fda1fe00640 1 --2- 172.21.15.72:0/2263244780 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fda0c0a3c40 0x7fda0c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:29.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.465+0000 7fda1fe00640 1 -- 172.21.15.72:0/2263244780 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7fda200f1ff0 con 0x7fda28106740 2024-04-26T23:45:29.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.465+0000 7fda27400640 1 --2- 172.21.15.72:0/2263244780 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fda0c0a3c40 0x7fda0c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:29.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.466+0000 7fda27400640 1 --2- 172.21.15.72:0/2263244780 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fda0c0a3c40 0x7fda0c0a6100 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fda180058c0 tx=0x7fda18005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:29.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.467+0000 7fda1fe00640 1 -- 172.21.15.72:0/2263244780 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fda200be550 con 0x7fda28106740 2024-04-26T23:45:29.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.670+0000 7fda2e000640 1 -- 172.21.15.72:0/2263244780 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fda2802dad0 con 0x7fda28106740 2024-04-26T23:45:29.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.670+0000 7fda1fe00640 1 -- 172.21.15.72:0/2263244780 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7fda200bde50 con 0x7fda28106740 2024-04-26T23:45:29.674 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-26T23:45:29.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.673+0000 7fda2e000640 1 -- 172.21.15.72:0/2263244780 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fda0c0a3c40 msgr2=0x7fda0c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:29.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.673+0000 7fda2e000640 1 --2- 172.21.15.72:0/2263244780 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fda0c0a3c40 0x7fda0c0a6100 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fda180058c0 tx=0x7fda18005f90 comp rx=0 tx=0).stop 2024-04-26T23:45:29.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.673+0000 7fda2e000640 1 -- 172.21.15.72:0/2263244780 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda28106740 msgr2=0x7fda281a2310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:29.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.673+0000 7fda2e000640 1 --2- 172.21.15.72:0/2263244780 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda28106740 0x7fda281a2310 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fda2002aea0 tx=0x7fda20003ba0 comp rx=0 tx=0).stop 2024-04-26T23:45:29.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.674+0000 7fda2e000640 1 -- 172.21.15.72:0/2263244780 shutdown_connections 2024-04-26T23:45:29.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.674+0000 7fda2e000640 1 --2- 172.21.15.72:0/2263244780 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fda0c0a3c40 0x7fda0c0a6100 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:29.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.674+0000 7fda2e000640 1 --2- 172.21.15.72:0/2263244780 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda28105d80 0x7fda2819b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:29.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.674+0000 7fda2e000640 1 --2- 172.21.15.72:0/2263244780 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda28106740 0x7fda281a2310 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:29.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.674+0000 7fda2e000640 1 --2- 172.21.15.72:0/2263244780 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fda28104b90 0x7fda2819ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:29.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.674+0000 7fda2e000640 1 -- 172.21.15.72:0/2263244780 >> 172.21.15.72:0/2263244780 conn(0x7fda281000e0 msgr2=0x7fda28109080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:29.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.674+0000 7fda2e000640 1 -- 172.21.15.72:0/2263244780 shutdown_connections 2024-04-26T23:45:29.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.674+0000 7fda2e000640 1 -- 172.21.15.72:0/2263244780 wait complete. 2024-04-26T23:45:29.698 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:45:29.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.898+0000 7f190c000640 1 -- 172.21.15.72:0/2158709344 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1904076970 msgr2=0x7f1904074dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:29.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.898+0000 7f190c000640 1 --2- 172.21.15.72:0/2158709344 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1904076970 0x7f1904074dc0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f18fc009900 tx=0x7f18fc02a410 comp rx=0 tx=0).stop 2024-04-26T23:45:29.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.899+0000 7f190c000640 1 -- 172.21.15.72:0/2158709344 shutdown_connections 2024-04-26T23:45:29.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.899+0000 7f190c000640 1 --2- 172.21.15.72:0/2158709344 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f190410a840 0x7f190410ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:29.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.899+0000 7f190c000640 1 --2- 172.21.15.72:0/2158709344 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1904075390 0x7f1904075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:29.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.899+0000 7f190c000640 1 --2- 172.21.15.72:0/2158709344 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1904076970 0x7f1904074dc0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:29.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.899+0000 7f190c000640 1 -- 172.21.15.72:0/2158709344 >> 172.21.15.72:0/2158709344 conn(0x7f19040fdf10 msgr2=0x7f1904100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:29.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.899+0000 7f190c000640 1 -- 172.21.15.72:0/2158709344 shutdown_connections 2024-04-26T23:45:29.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.899+0000 7f190c000640 1 -- 172.21.15.72:0/2158709344 wait complete. 2024-04-26T23:45:29.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.900+0000 7f190c000640 1 Processor -- start 2024-04-26T23:45:29.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.900+0000 7f190c000640 1 -- start start 2024-04-26T23:45:29.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.900+0000 7f190c000640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1904075390 0x7f190419add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:29.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.900+0000 7f190c000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1904076970 0x7f190419b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:29.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.900+0000 7f190c000640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f190410a840 0x7f19041a2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:29.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.900+0000 7f190c000640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f19041106e0 con 0x7f1904075390 2024-04-26T23:45:29.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.900+0000 7f190c000640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f1904110560 con 0x7f1904076970 2024-04-26T23:45:29.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.900+0000 7f190c000640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f1904110860 con 0x7f190410a840 2024-04-26T23:45:29.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.900+0000 7f1909600640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1904075390 0x7f190419add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:29.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.900+0000 7f190a000640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f190410a840 0x7f19041a2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:29.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.900+0000 7f1909600640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1904075390 0x7f190419add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54144/0 (socket says 172.21.15.72:54144) 2024-04-26T23:45:29.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.900+0000 7f1909600640 1 -- 172.21.15.72:0/2382228455 learned_addr learned my addr 172.21.15.72:0/2382228455 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:45:29.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.900+0000 7f1908c00640 1 --2- 172.21.15.72:0/2382228455 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1904076970 0x7f190419b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:29.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.901+0000 7f190a000640 1 -- 172.21.15.72:0/2382228455 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1904076970 msgr2=0x7f190419b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:29.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.901+0000 7f190a000640 1 --2- 172.21.15.72:0/2382228455 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1904076970 0x7f190419b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:29.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.901+0000 7f190a000640 1 -- 172.21.15.72:0/2382228455 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1904075390 msgr2=0x7f190419add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:29.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.901+0000 7f190a000640 1 --2- 172.21.15.72:0/2382228455 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1904075390 0x7f190419add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:29.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.901+0000 7f190a000640 1 -- 172.21.15.72:0/2382228455 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19041a29f0 con 0x7f190410a840 2024-04-26T23:45:29.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.901+0000 7f1909600640 1 --2- 172.21.15.72:0/2382228455 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1904075390 0x7f190419add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:45:29.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.901+0000 7f190a000640 1 --2- 172.21.15.72:0/2382228455 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f190410a840 0x7f19041a2380 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f18f400e970 tx=0x7f18f400ee30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:29.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.901+0000 7f1902000640 1 -- 172.21.15.72:0/2382228455 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18f400ccc0 con 0x7f190410a840 2024-04-26T23:45:29.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.901+0000 7f190c000640 1 -- 172.21.15.72:0/2382228455 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19041a2ce0 con 0x7f190410a840 2024-04-26T23:45:29.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.901+0000 7f1902000640 1 -- 172.21.15.72:0/2382228455 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f18f4005020 con 0x7f190410a840 2024-04-26T23:45:29.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.901+0000 7f1902000640 1 -- 172.21.15.72:0/2382228455 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18f4004a00 con 0x7f190410a840 2024-04-26T23:45:29.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.902+0000 7f190c000640 1 -- 172.21.15.72:0/2382228455 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19041a31d0 con 0x7f190410a840 2024-04-26T23:45:29.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.903+0000 7f190c000640 1 -- 172.21.15.72:0/2382228455 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1904109570 con 0x7f190410a840 2024-04-26T23:45:29.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.905+0000 7f1902000640 1 -- 172.21.15.72:0/2382228455 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f18f4004ba0 con 0x7f190410a840 2024-04-26T23:45:29.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.906+0000 7f1902000640 1 --2- 172.21.15.72:0/2382228455 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f18e80a3c40 0x7f18e80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:29.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.906+0000 7f1902000640 1 -- 172.21.15.72:0/2382228455 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f18f40d0a60 con 0x7f190410a840 2024-04-26T23:45:29.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.906+0000 7f1909600640 1 --2- 172.21.15.72:0/2382228455 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f18e80a3c40 0x7f18e80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:29.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.907+0000 7f1909600640 1 --2- 172.21.15.72:0/2382228455 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f18e80a3c40 0x7f18e80a6100 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f18fc0096c0 tx=0x7f18fc005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:29.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:29.909+0000 7f1902000640 1 -- 172.21.15.72:0/2382228455 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f18f409ce90 con 0x7f190410a840 2024-04-26T23:45:29.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:29 smithi076 ceph-mon[32774]: pgmap v398: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:45:29.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:29 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/2220524562' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T23:45:29.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:29 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/1891441562' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T23:45:30.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:29 smithi072 ceph-mon[28184]: pgmap v398: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:45:30.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:29 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/2220524562' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T23:45:30.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:29 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/1891441562' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T23:45:30.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:29 smithi136 ceph-mon[33050]: pgmap v398: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:45:30.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:29 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/2220524562' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T23:45:30.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:29 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/1891441562' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T23:45:30.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.154+0000 7f190c000640 1 -- 172.21.15.72:0/2382228455 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f190402dad0 con 0x7f190410a840 2024-04-26T23:45:30.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.156+0000 7f1902000640 1 -- 172.21.15.72:0/2382228455 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11874 (secure 0 0 0) 0x7f18f409c790 con 0x7f190410a840 2024-04-26T23:45:30.159 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:45:30.159 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/2386456166","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2386456166},{"type":"v1","addr":"172.21.15.72:6835","nonce":2386456166}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1528342053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1528342053},{"type":"v1","addr":"172.21.15.136:6833","nonce":1528342053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6833/2427238559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":2427238559},{"type":"v1","addr":"172.21.15.76:6833","nonce":2427238559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24491,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/4050689894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":4050689894},{"type":"v1","addr":"172.21.15.72:6837","nonce":4050689894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3681925367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3681925367},{"type":"v1","addr":"172.21.15.136:6835","nonce":3681925367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4251109085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4251109085},{"type":"v1","addr":"172.21.15.76:6835","nonce":4251109085}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T23:43:02.928649+0000","modified":"2024-04-26T23:43:30.190394+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24590,"mds_1":24571,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24514":{"gid":24514,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6837/3639308555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3639308555},{"type":"v1","addr":"172.21.15.136:6837","nonce":3639308555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.76:6837/2506930945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2506930945},{"type":"v1","addr":"172.21.15.76:6837","nonce":2506930945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6839/3938677630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3938677630},{"type":"v1","addr":"172.21.15.136:6839","nonce":3938677630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.72:6839/1403162102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6838","nonce":1403162102},{"type":"v1","addr":"172.21.15.72:6839","nonce":1403162102}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24571":{"gid":24571,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.72:6841/1356345077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6840","nonce":1356345077},{"type":"v1","addr":"172.21.15.72:6841","nonce":1356345077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.76:6839/3405829058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":3405829058},{"type":"v1","addr":"172.21.15.76:6839","nonce":3405829058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24553,24571,24590]},"id":1}]} 2024-04-26T23:45:30.159 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 23 2024-04-26T23:45:30.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.158+0000 7f190c000640 1 -- 172.21.15.72:0/2382228455 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f18e80a3c40 msgr2=0x7f18e80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:30.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.158+0000 7f190c000640 1 --2- 172.21.15.72:0/2382228455 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f18e80a3c40 0x7f18e80a6100 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f18fc0096c0 tx=0x7f18fc005a20 comp rx=0 tx=0).stop 2024-04-26T23:45:30.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.158+0000 7f190c000640 1 -- 172.21.15.72:0/2382228455 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f190410a840 msgr2=0x7f19041a2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:30.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.158+0000 7f190c000640 1 --2- 172.21.15.72:0/2382228455 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f190410a840 0x7f19041a2380 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f18f400e970 tx=0x7f18f400ee30 comp rx=0 tx=0).stop 2024-04-26T23:45:30.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.158+0000 7f190c000640 1 -- 172.21.15.72:0/2382228455 shutdown_connections 2024-04-26T23:45:30.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.158+0000 7f190c000640 1 --2- 172.21.15.72:0/2382228455 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f18e80a3c40 0x7f18e80a6100 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:30.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.158+0000 7f190c000640 1 --2- 172.21.15.72:0/2382228455 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1904076970 0x7f190419b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:30.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.158+0000 7f190c000640 1 --2- 172.21.15.72:0/2382228455 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f190410a840 0x7f19041a2380 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:30.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.158+0000 7f190c000640 1 --2- 172.21.15.72:0/2382228455 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1904075390 0x7f190419add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:30.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.159+0000 7f190c000640 1 -- 172.21.15.72:0/2382228455 >> 172.21.15.72:0/2382228455 conn(0x7f19040fdf10 msgr2=0x7f1904106740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:30.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.159+0000 7f190c000640 1 -- 172.21.15.72:0/2382228455 shutdown_connections 2024-04-26T23:45:30.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.159+0000 7f190c000640 1 -- 172.21.15.72:0/2382228455 wait complete. 2024-04-26T23:45:30.181 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:45:30.379 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.376+0000 7f23fec00640 1 -- 172.21.15.72:0/1958033403 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f23f8108f90 msgr2=0x7f23f81093b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:30.379 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.376+0000 7f23fec00640 1 --2- 172.21.15.72:0/1958033403 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f23f8108f90 0x7f23f81093b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f23e8009be0 tx=0x7f23e802a6c0 comp rx=0 tx=0).stop 2024-04-26T23:45:30.379 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.376+0000 7f23fec00640 1 -- 172.21.15.72:0/1958033403 shutdown_connections 2024-04-26T23:45:30.379 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.376+0000 7f23fec00640 1 --2- 172.21.15.72:0/1958033403 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23f810ab40 0x7f23f81113d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:30.380 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.376+0000 7f23fec00640 1 --2- 172.21.15.72:0/1958033403 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f23f810a180 0x7f23f810a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:30.380 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.376+0000 7f23fec00640 1 --2- 172.21.15.72:0/1958033403 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f23f8108f90 0x7f23f81093b0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:30.380 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.376+0000 7f23fec00640 1 -- 172.21.15.72:0/1958033403 >> 172.21.15.72:0/1958033403 conn(0x7f23f8077f40 msgr2=0x7f23f807a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:30.380 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.376+0000 7f23fec00640 1 -- 172.21.15.72:0/1958033403 shutdown_connections 2024-04-26T23:45:30.380 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.376+0000 7f23fec00640 1 -- 172.21.15.72:0/1958033403 wait complete. 2024-04-26T23:45:30.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.377+0000 7f23fec00640 1 Processor -- start 2024-04-26T23:45:30.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.377+0000 7f23fec00640 1 -- start start 2024-04-26T23:45:30.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.377+0000 7f23fec00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f23f8108f90 0x7f23f819f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:30.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.377+0000 7f23fec00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23f810a180 0x7f23f819f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:30.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.377+0000 7f23fec00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f23f810ab40 0x7f23f81a6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:30.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.377+0000 7f23fec00640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f23f8114a90 con 0x7f23f810a180 2024-04-26T23:45:30.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.377+0000 7f23fec00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f23f8114910 con 0x7f23f8108f90 2024-04-26T23:45:30.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.377+0000 7f23fec00640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f23f8114c10 con 0x7f23f810ab40 2024-04-26T23:45:30.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.377+0000 7f23f7400640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23f810a180 0x7f23f819f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:30.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.378+0000 7f23f7e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f23f8108f90 0x7f23f819f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:30.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.378+0000 7f23fcc00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f23f810ab40 0x7f23f81a6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:30.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.378+0000 7f23f7e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f23f8108f90 0x7f23f819f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:52748/0 (socket says 172.21.15.72:52748) 2024-04-26T23:45:30.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.378+0000 7f23f7e00640 1 -- 172.21.15.72:0/3608750048 learned_addr learned my addr 172.21.15.72:0/3608750048 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:45:30.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.378+0000 7f23f7e00640 1 -- 172.21.15.72:0/3608750048 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f23f810ab40 msgr2=0x7f23f81a6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:30.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.378+0000 7f23f7e00640 1 --2- 172.21.15.72:0/3608750048 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f23f810ab40 0x7f23f81a6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:30.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.378+0000 7f23f7e00640 1 -- 172.21.15.72:0/3608750048 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23f810a180 msgr2=0x7f23f819f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:30.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.378+0000 7f23f7e00640 1 --2- 172.21.15.72:0/3608750048 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23f810a180 0x7f23f819f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:30.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.378+0000 7f23f7e00640 1 -- 172.21.15.72:0/3608750048 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23f81a6de0 con 0x7f23f8108f90 2024-04-26T23:45:30.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.378+0000 7f23fcc00640 1 --2- 172.21.15.72:0/3608750048 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f23f810ab40 0x7f23f81a6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:45:30.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.378+0000 7f23f7e00640 1 --2- 172.21.15.72:0/3608750048 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f23f8108f90 0x7f23f819f1c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f23e8000f80 tx=0x7f23e8030d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:30.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.379+0000 7f23f4c00640 1 -- 172.21.15.72:0/3608750048 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23e8041020 con 0x7f23f8108f90 2024-04-26T23:45:30.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.379+0000 7f23f4c00640 1 -- 172.21.15.72:0/3608750048 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f23e80318f0 con 0x7f23f8108f90 2024-04-26T23:45:30.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.379+0000 7f23f4c00640 1 -- 172.21.15.72:0/3608750048 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23e8036040 con 0x7f23f8108f90 2024-04-26T23:45:30.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.379+0000 7f23fec00640 1 -- 172.21.15.72:0/3608750048 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23f81a7070 con 0x7f23f8108f90 2024-04-26T23:45:30.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.379+0000 7f23fec00640 1 -- 172.21.15.72:0/3608750048 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23f81a7510 con 0x7f23f8108f90 2024-04-26T23:45:30.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.380+0000 7f23fec00640 1 -- 172.21.15.72:0/3608750048 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23f81102d0 con 0x7f23f8108f90 2024-04-26T23:45:30.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.383+0000 7f23f4c00640 1 -- 172.21.15.72:0/3608750048 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f23e8044020 con 0x7f23f8108f90 2024-04-26T23:45:30.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.384+0000 7f23f4c00640 1 --2- 172.21.15.72:0/3608750048 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f23dc0a3c40 0x7f23dc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:30.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.384+0000 7f23f4c00640 1 -- 172.21.15.72:0/3608750048 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f23e80f1a30 con 0x7f23f8108f90 2024-04-26T23:45:30.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.384+0000 7f23f7400640 1 --2- 172.21.15.72:0/3608750048 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f23dc0a3c40 0x7f23dc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:30.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.385+0000 7f23f7400640 1 --2- 172.21.15.72:0/3608750048 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f23dc0a3c40 0x7f23dc0a6100 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f23f81a06d0 tx=0x7f23e4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:30.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.386+0000 7f23f4c00640 1 -- 172.21.15.72:0/3608750048 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f23e804b020 con 0x7f23f8108f90 2024-04-26T23:45:30.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.590+0000 7f23fec00640 1 -- 172.21.15.72:0/3608750048 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f23f802dd40 con 0x7f23f8108f90 2024-04-26T23:45:30.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.593+0000 7f23f4c00640 1 -- 172.21.15.72:0/3608750048 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24841 (secure 0 0 0) 0x7f23e80bde60 con 0x7f23f8108f90 2024-04-26T23:45:30.596 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:45:30.596 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":93,"fsid":"3c8453ae-0425-11ef-bc93-c7b262605968","created":"2024-04-26T23:33:48.088300+0000","modified":"2024-04-26T23:45:00.532006+0000","last_up_change":"2024-04-26T23:42:04.482891+0000","last_in_change":"2024-04-26T23:41:44.784415+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:37:44.428727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:43:00.431829+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:43:01.872977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:43:05.512050+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e20c9ebf-56f6-47c1-a11b-e1e615af042b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6803","nonce":1639298875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6805","nonce":1639298875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6809","nonce":1639298875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6807","nonce":1639298875}]},"public_addr":"172.21.15.72:6803/1639298875","cluster_addr":"172.21.15.72:6805/1639298875","heartbeat_back_addr":"172.21.15.72:6809/1639298875","heartbeat_front_addr":"172.21.15.72:6807/1639298875","state":["exists","up"]},{"osd":1,"uuid":"e6cac469-780c-4d66-b560-b686072371fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6801","nonce":4185342193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6803","nonce":4185342193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6807","nonce":4185342193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6805","nonce":4185342193}]},"public_addr":"172.21.15.76:6801/4185342193","cluster_addr":"172.21.15.76:6803/4185342193","heartbeat_back_addr":"172.21.15.76:6807/4185342193","heartbeat_front_addr":"172.21.15.76:6805/4185342193","state":["exists","up"]},{"osd":2,"uuid":"838a49fc-18de-446c-8336-67b3e05da231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6801","nonce":3033860799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6803","nonce":3033860799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6807","nonce":3033860799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6805","nonce":3033860799}]},"public_addr":"172.21.15.136:6801/3033860799","cluster_addr":"172.21.15.136:6803/3033860799","heartbeat_back_addr":"172.21.15.136:6807/3033860799","heartbeat_front_addr":"172.21.15.136:6805/3033860799","state":["exists","up"]},{"osd":3,"uuid":"e6da89be-5ba7-48a4-9306-f75498660e0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6811","nonce":2071107815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6813","nonce":2071107815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6817","nonce":2071107815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6815","nonce":2071107815}]},"public_addr":"172.21.15.72:6811/2071107815","cluster_addr":"172.21.15.72:6813/2071107815","heartbeat_back_addr":"172.21.15.72:6817/2071107815","heartbeat_front_addr":"172.21.15.72:6815/2071107815","state":["exists","up"]},{"osd":4,"uuid":"0b17c491-d798-488f-892d-521ef2fbc51e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6809","nonce":147449238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6811","nonce":147449238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6815","nonce":147449238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6813","nonce":147449238}]},"public_addr":"172.21.15.76:6809/147449238","cluster_addr":"172.21.15.76:6811/147449238","heartbeat_back_addr":"172.21.15.76:6815/147449238","heartbeat_front_addr":"172.21.15.76:6813/147449238","state":["exists","up"]},{"osd":5,"uuid":"c7e887b3-ab38-4608-9bfd-1ed49956e8af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6809","nonce":2320149212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6811","nonce":2320149212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6815","nonce":2320149212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6813","nonce":2320149212}]},"public_addr":"172.21.15.136:6809/2320149212","cluster_addr":"172.21.15.136:6811/2320149212","heartbeat_back_addr":"172.21.15.136:6815/2320149212","heartbeat_front_addr":"172.21.15.136:6813/2320149212","state":["exists","up"]},{"osd":6,"uuid":"f0bd604b-0f57-4a38-9295-eb200264f1ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6819","nonce":389499207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6821","nonce":389499207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6825","nonce":389499207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6823","nonce":389499207}]},"public_addr":"172.21.15.72:6819/389499207","cluster_addr":"172.21.15.72:6821/389499207","heartbeat_back_addr":"172.21.15.72:6825/389499207","heartbeat_front_addr":"172.21.15.72:6823/389499207","state":["exists","up"]},{"osd":7,"uuid":"1879d4e4-038e-4cd6-a3ec-3a940bcc3797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6817","nonce":375888575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6819","nonce":375888575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6823","nonce":375888575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6821","nonce":375888575}]},"public_addr":"172.21.15.76:6817/375888575","cluster_addr":"172.21.15.76:6819/375888575","heartbeat_back_addr":"172.21.15.76:6823/375888575","heartbeat_front_addr":"172.21.15.76:6821/375888575","state":["exists","up"]},{"osd":8,"uuid":"6f404062-97e7-4ccb-a582-76dc1ffae548","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6817","nonce":3446774052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6819","nonce":3446774052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6823","nonce":3446774052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6821","nonce":3446774052}]},"public_addr":"172.21.15.136:6817/3446774052","cluster_addr":"172.21.15.136:6819/3446774052","heartbeat_back_addr":"172.21.15.136:6823/3446774052","heartbeat_front_addr":"172.21.15.136:6821/3446774052","state":["exists","up"]},{"osd":9,"uuid":"2989ca68-3164-4667-9bcf-230dcbb7b8f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6827","nonce":3423789152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6829","nonce":3423789152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6832","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6833","nonce":3423789152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6831","nonce":3423789152}]},"public_addr":"172.21.15.72:6827/3423789152","cluster_addr":"172.21.15.72:6829/3423789152","heartbeat_back_addr":"172.21.15.72:6833/3423789152","heartbeat_front_addr":"172.21.15.72:6831/3423789152","state":["exists","up"]},{"osd":10,"uuid":"9ec6d46a-0913-4208-97b5-167188ee8cb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6825","nonce":1120991574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6827","nonce":1120991574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6831","nonce":1120991574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6829","nonce":1120991574}]},"public_addr":"172.21.15.76:6825/1120991574","cluster_addr":"172.21.15.76:6827/1120991574","heartbeat_back_addr":"172.21.15.76:6831/1120991574","heartbeat_front_addr":"172.21.15.76:6829/1120991574","state":["exists","up"]},{"osd":11,"uuid":"bdd0774f-6e58-4918-b4cc-afd4f1118d3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6825","nonce":1135995193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6827","nonce":1135995193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6831","nonce":1135995193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6829","nonce":1135995193}]},"public_addr":"172.21.15.136:6825/1135995193","cluster_addr":"172.21.15.136:6827/1135995193","heartbeat_back_addr":"172.21.15.136:6831/1135995193","heartbeat_front_addr":"172.21.15.136:6829/1135995193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:36:52.128788+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:37:17.523654+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:37:41.634677+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:38:15.060797+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:38:41.234111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:39:06.783560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:39:40.884386+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:40:08.340465+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:40:34.344858+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:41:08.525309+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:41:36.782624+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:42:02.756888+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":6}]},{"pgid":"3.37","mappings":[{"from":5,"to":7}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:0/3625071330":"2024-04-27T23:34:58.266218+0000","172.21.15.72:6801/2903979134":"2024-04-27T23:34:58.266218+0000","172.21.15.72:6800/2903979134":"2024-04-27T23:34:58.266218+0000","172.21.15.72:0/1519142576":"2024-04-27T23:34:58.266218+0000","172.21.15.72:0/3669626460":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/945623297":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/3479150073":"2024-04-27T23:34:24.233319+0000","172.21.15.72:6800/4175896290":"2024-04-27T23:34:24.233319+0000","172.21.15.72:6801/4175896290":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/3783175012":"2024-04-27T23:34:58.266218+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:45:30.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.596+0000 7f23fec00640 1 -- 172.21.15.72:0/3608750048 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f23dc0a3c40 msgr2=0x7f23dc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:30.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.596+0000 7f23fec00640 1 --2- 172.21.15.72:0/3608750048 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f23dc0a3c40 0x7f23dc0a6100 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f23f81a06d0 tx=0x7f23e4009260 comp rx=0 tx=0).stop 2024-04-26T23:45:30.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.596+0000 7f23fec00640 1 -- 172.21.15.72:0/3608750048 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f23f8108f90 msgr2=0x7f23f819f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:30.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.596+0000 7f23fec00640 1 --2- 172.21.15.72:0/3608750048 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f23f8108f90 0x7f23f819f1c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f23e8000f80 tx=0x7f23e8030d10 comp rx=0 tx=0).stop 2024-04-26T23:45:30.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.596+0000 7f23fec00640 1 -- 172.21.15.72:0/3608750048 shutdown_connections 2024-04-26T23:45:30.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.596+0000 7f23fec00640 1 --2- 172.21.15.72:0/3608750048 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f23dc0a3c40 0x7f23dc0a6100 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:30.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.596+0000 7f23fec00640 1 --2- 172.21.15.72:0/3608750048 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f23f810ab40 0x7f23f81a6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:30.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.596+0000 7f23fec00640 1 --2- 172.21.15.72:0/3608750048 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23f810a180 0x7f23f819f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:30.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.596+0000 7f23fec00640 1 --2- 172.21.15.72:0/3608750048 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f23f8108f90 0x7f23f819f1c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:30.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.596+0000 7f23fec00640 1 -- 172.21.15.72:0/3608750048 >> 172.21.15.72:0/3608750048 conn(0x7f23f8077f40 msgr2=0x7f23f810d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:30.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.596+0000 7f23fec00640 1 -- 172.21.15.72:0/3608750048 shutdown_connections 2024-04-26T23:45:30.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.596+0000 7f23fec00640 1 -- 172.21.15.72:0/3608750048 wait complete. 2024-04-26T23:45:30.620 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:45:30.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.809+0000 7f220e200640 1 -- 172.21.15.72:0/1037538704 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f220810a940 msgr2=0x7f22081111d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:30.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.809+0000 7f220e200640 1 --2- 172.21.15.72:0/1037538704 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f220810a940 0x7f22081111d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f2200009ca0 tx=0x7f220002a8c0 comp rx=0 tx=0).stop 2024-04-26T23:45:30.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.809+0000 7f220e200640 1 -- 172.21.15.72:0/1037538704 shutdown_connections 2024-04-26T23:45:30.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.809+0000 7f220e200640 1 --2- 172.21.15.72:0/1037538704 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f220810a940 0x7f22081111d0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:30.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.809+0000 7f220e200640 1 --2- 172.21.15.72:0/1037538704 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2208109f80 0x7f220810a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:30.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.809+0000 7f220e200640 1 --2- 172.21.15.72:0/1037538704 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2208108d90 0x7f22081091b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:30.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.809+0000 7f220e200640 1 -- 172.21.15.72:0/1037538704 >> 172.21.15.72:0/1037538704 conn(0x7f2208077d30 msgr2=0x7f220807a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:30.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.809+0000 7f220e200640 1 -- 172.21.15.72:0/1037538704 shutdown_connections 2024-04-26T23:45:30.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.809+0000 7f220e200640 1 -- 172.21.15.72:0/1037538704 wait complete. 2024-04-26T23:45:30.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.810+0000 7f220e200640 1 Processor -- start 2024-04-26T23:45:30.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.811+0000 7f220e200640 1 -- start start 2024-04-26T23:45:30.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.811+0000 7f220e200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2208108d90 0x7f220819ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:30.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.811+0000 7f220e200640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2208109f80 0x7f220819f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:30.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.811+0000 7f220e200640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f220810a940 0x7f22081a64f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:30.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.811+0000 7f220e200640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f22081148c0 con 0x7f220810a940 2024-04-26T23:45:30.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.811+0000 7f220e200640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2208114740 con 0x7f2208108d90 2024-04-26T23:45:30.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.811+0000 7f220e200640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f2208114a40 con 0x7f2208109f80 2024-04-26T23:45:30.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.811+0000 7f220d800640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f220810a940 0x7f22081a64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:30.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.811+0000 7f220d800640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f220810a940 0x7f22081a64f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54172/0 (socket says 172.21.15.72:54172) 2024-04-26T23:45:30.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.811+0000 7f220d800640 1 -- 172.21.15.72:0/1432176528 learned_addr learned my addr 172.21.15.72:0/1432176528 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:45:30.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.811+0000 7f220ce00640 1 --2- 172.21.15.72:0/1432176528 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2208108d90 0x7f220819ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:30.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.811+0000 7f2207e00640 1 --2- 172.21.15.72:0/1432176528 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2208109f80 0x7f220819f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:30.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.812+0000 7f220d800640 1 -- 172.21.15.72:0/1432176528 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2208109f80 msgr2=0x7f220819f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:30.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.812+0000 7f220d800640 1 --2- 172.21.15.72:0/1432176528 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2208109f80 0x7f220819f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:30.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.812+0000 7f220d800640 1 -- 172.21.15.72:0/1432176528 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2208108d90 msgr2=0x7f220819ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:30.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.812+0000 7f220d800640 1 --2- 172.21.15.72:0/1432176528 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2208108d90 0x7f220819ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:30.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.812+0000 7f220d800640 1 -- 172.21.15.72:0/1432176528 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22081a6b60 con 0x7f220810a940 2024-04-26T23:45:30.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.812+0000 7f220ce00640 1 --2- 172.21.15.72:0/1432176528 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2208108d90 0x7f220819ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:45:30.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.812+0000 7f220d800640 1 --2- 172.21.15.72:0/1432176528 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f220810a940 0x7f22081a64f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f22000097b0 tx=0x7f2200031770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:30.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.812+0000 7f2205600640 1 -- 172.21.15.72:0/1432176528 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2200003b70 con 0x7f220810a940 2024-04-26T23:45:30.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.812+0000 7f2205600640 1 -- 172.21.15.72:0/1432176528 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2200003d10 con 0x7f220810a940 2024-04-26T23:45:30.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.812+0000 7f220e200640 1 -- 172.21.15.72:0/1432176528 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22081a6df0 con 0x7f220810a940 2024-04-26T23:45:30.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.812+0000 7f2205600640 1 -- 172.21.15.72:0/1432176528 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f220002fc10 con 0x7f220810a940 2024-04-26T23:45:30.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.812+0000 7f220e200640 1 -- 172.21.15.72:0/1432176528 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22081a72b0 con 0x7f220810a940 2024-04-26T23:45:30.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.814+0000 7f220e200640 1 -- 172.21.15.72:0/1432176528 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22081100b0 con 0x7f220810a940 2024-04-26T23:45:30.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.816+0000 7f2205600640 1 -- 172.21.15.72:0/1432176528 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f2200004040 con 0x7f220810a940 2024-04-26T23:45:30.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.817+0000 7f2205600640 1 --2- 172.21.15.72:0/1432176528 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f21f00a3c40 0x7f21f00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:30.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.817+0000 7f2205600640 1 -- 172.21.15.72:0/1432176528 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f22000f1db0 con 0x7f220810a940 2024-04-26T23:45:30.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.817+0000 7f220ce00640 1 --2- 172.21.15.72:0/1432176528 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f21f00a3c40 0x7f21f00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:30.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.818+0000 7f220ce00640 1 --2- 172.21.15.72:0/1432176528 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f21f00a3c40 0x7f21f00a6100 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f2208109de0 tx=0x7f21fc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:30.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:30.821+0000 7f2205600640 1 -- 172.21.15.72:0/1432176528 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f22000be410 con 0x7f220810a940 2024-04-26T23:45:30.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:30 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/2263244780' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T23:45:30.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:30 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/2382228455' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:45:30.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:30 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/3608750048' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:45:31.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:30 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/2263244780' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T23:45:31.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:30 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/2382228455' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:45:31.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:30 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/3608750048' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:45:31.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:30 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/2263244780' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T23:45:31.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:30 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/2382228455' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:45:31.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:30 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/3608750048' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:45:31.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.063+0000 7f220e200640 1 -- 172.21.15.72:0/1432176528 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f220802dcd0 con 0x7f220810a940 2024-04-26T23:45:31.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.065+0000 7f2205600640 1 -- 172.21.15.72:0/1432176528 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11874 (secure 0 0 0) 0x7f22000bdd10 con 0x7f220810a940 2024-04-26T23:45:31.068 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:45:31.068 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/2386456166","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2386456166},{"type":"v1","addr":"172.21.15.72:6835","nonce":2386456166}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1528342053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1528342053},{"type":"v1","addr":"172.21.15.136:6833","nonce":1528342053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6833/2427238559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":2427238559},{"type":"v1","addr":"172.21.15.76:6833","nonce":2427238559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24491,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/4050689894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":4050689894},{"type":"v1","addr":"172.21.15.72:6837","nonce":4050689894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3681925367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3681925367},{"type":"v1","addr":"172.21.15.136:6835","nonce":3681925367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4251109085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4251109085},{"type":"v1","addr":"172.21.15.76:6835","nonce":4251109085}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T23:43:02.928649+0000","modified":"2024-04-26T23:43:30.190394+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24590,"mds_1":24571,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24514":{"gid":24514,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6837/3639308555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3639308555},{"type":"v1","addr":"172.21.15.136:6837","nonce":3639308555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.76:6837/2506930945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2506930945},{"type":"v1","addr":"172.21.15.76:6837","nonce":2506930945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6839/3938677630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3938677630},{"type":"v1","addr":"172.21.15.136:6839","nonce":3938677630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.72:6839/1403162102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6838","nonce":1403162102},{"type":"v1","addr":"172.21.15.72:6839","nonce":1403162102}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24571":{"gid":24571,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.72:6841/1356345077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6840","nonce":1356345077},{"type":"v1","addr":"172.21.15.72:6841","nonce":1356345077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.76:6839/3405829058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":3405829058},{"type":"v1","addr":"172.21.15.76:6839","nonce":3405829058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24553,24571,24590]},"id":1}]} 2024-04-26T23:45:31.068 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 23 2024-04-26T23:45:31.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.067+0000 7f220e200640 1 -- 172.21.15.72:0/1432176528 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f21f00a3c40 msgr2=0x7f21f00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:31.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.067+0000 7f220e200640 1 --2- 172.21.15.72:0/1432176528 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f21f00a3c40 0x7f21f00a6100 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f2208109de0 tx=0x7f21fc005f90 comp rx=0 tx=0).stop 2024-04-26T23:45:31.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.067+0000 7f220e200640 1 -- 172.21.15.72:0/1432176528 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f220810a940 msgr2=0x7f22081a64f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:31.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.067+0000 7f220e200640 1 --2- 172.21.15.72:0/1432176528 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f220810a940 0x7f22081a64f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f22000097b0 tx=0x7f2200031770 comp rx=0 tx=0).stop 2024-04-26T23:45:31.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.067+0000 7f220e200640 1 -- 172.21.15.72:0/1432176528 shutdown_connections 2024-04-26T23:45:31.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.067+0000 7f220e200640 1 --2- 172.21.15.72:0/1432176528 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f21f00a3c40 0x7f21f00a6100 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.067+0000 7f220e200640 1 --2- 172.21.15.72:0/1432176528 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f220810a940 0x7f22081a64f0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.067+0000 7f220e200640 1 --2- 172.21.15.72:0/1432176528 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2208109f80 0x7f220819f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.067+0000 7f220e200640 1 --2- 172.21.15.72:0/1432176528 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2208108d90 0x7f220819ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.067+0000 7f220e200640 1 -- 172.21.15.72:0/1432176528 >> 172.21.15.72:0/1432176528 conn(0x7f2208077d30 msgr2=0x7f220810d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:31.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.068+0000 7f220e200640 1 -- 172.21.15.72:0/1432176528 shutdown_connections 2024-04-26T23:45:31.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.068+0000 7f220e200640 1 -- 172.21.15.72:0/1432176528 wait complete. 2024-04-26T23:45:31.090 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-26T23:43:02.928649+0000', 'modified': '2024-04-26T23:43:30.190394+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24590, 'mds_1': 24571, 'mds_2': 24553}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24514': {'gid': 24514, 'name': 'c', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.136:6837/3639308555', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6836', 'nonce': 3639308555}, {'type': 'v1', 'addr': '172.21.15.136:6837', 'nonce': 3639308555}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24533': {'gid': 24533, 'name': 'h', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.76:6837/2506930945', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6836', 'nonce': 2506930945}, {'type': 'v1', 'addr': '172.21.15.76:6837', 'nonce': 2506930945}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24535': {'gid': 24535, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.136:6839/3938677630', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6838', 'nonce': 3938677630}, {'type': 'v1', 'addr': '172.21.15.136:6839', 'nonce': 3938677630}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24553': {'gid': 24553, 'name': 'g', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.72:6839/1403162102', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6838', 'nonce': 1403162102}, {'type': 'v1', 'addr': '172.21.15.72:6839', 'nonce': 1403162102}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24571': {'gid': 24571, 'name': 'a', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.72:6841/1356345077', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6840', 'nonce': 1356345077}, {'type': 'v1', 'addr': '172.21.15.72:6841', 'nonce': 1356345077}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24590': {'gid': 24590, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.76:6839/3405829058', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6838', 'nonce': 3405829058}, {'type': 'v1', 'addr': '172.21.15.76:6839', 'nonce': 3405829058}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24590, 'qdb_cluster': [24553, 24571, 24590]} 2024-04-26T23:45:31.090 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-26T23:45:31.091 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.g status 2024-04-26T23:45:31.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.293+0000 7f3bbb200640 1 -- 172.21.15.72:0/2126440281 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bbc1031e0 msgr2=0x7f3bbc1055d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:31.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.293+0000 7f3bbb200640 1 --2- 172.21.15.72:0/2126440281 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bbc1031e0 0x7f3bbc1055d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3ba4009900 tx=0x7f3ba402a3f0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.293+0000 7f3bbb200640 1 -- 172.21.15.72:0/2126440281 shutdown_connections 2024-04-26T23:45:31.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.293+0000 7f3bbb200640 1 --2- 172.21.15.72:0/2126440281 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3bbc108440 0x7f3bbc10a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.293+0000 7f3bbb200640 1 --2- 172.21.15.72:0/2126440281 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3bbc105b10 0x7f3bbc107f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.293+0000 7f3bbb200640 1 --2- 172.21.15.72:0/2126440281 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bbc1031e0 0x7f3bbc1055d0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.293+0000 7f3bbb200640 1 -- 172.21.15.72:0/2126440281 >> 172.21.15.72:0/2126440281 conn(0x7f3bbc0fcbb0 msgr2=0x7f3bbc0ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:31.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.293+0000 7f3bbb200640 1 -- 172.21.15.72:0/2126440281 shutdown_connections 2024-04-26T23:45:31.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.293+0000 7f3bbb200640 1 -- 172.21.15.72:0/2126440281 wait complete. 2024-04-26T23:45:31.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.294+0000 7f3bbb200640 1 Processor -- start 2024-04-26T23:45:31.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.294+0000 7f3bbb200640 1 -- start start 2024-04-26T23:45:31.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.294+0000 7f3bbb200640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bbc1031e0 0x7f3bbc19ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.294+0000 7f3bbb200640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3bbc105b10 0x7f3bbc19b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.294+0000 7f3bbb200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3bbc108440 0x7f3bbc1a21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.294+0000 7f3bbb200640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f3bbc10e2e0 con 0x7f3bbc1031e0 2024-04-26T23:45:31.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.294+0000 7f3bbb200640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3bbc10e160 con 0x7f3bbc108440 2024-04-26T23:45:31.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.294+0000 7f3bbb200640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f3bbc10e460 con 0x7f3bbc105b10 2024-04-26T23:45:31.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.295+0000 7f3bb9e00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bbc1031e0 0x7f3bbc19ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.295+0000 7f3bb9e00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bbc1031e0 0x7f3bbc19ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54192/0 (socket says 172.21.15.72:54192) 2024-04-26T23:45:31.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.295+0000 7f3bb9e00640 1 -- 172.21.15.72:0/830401928 learned_addr learned my addr 172.21.15.72:0/830401928 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:45:31.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.295+0000 7f3bb9400640 1 --2- 172.21.15.72:0/830401928 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3bbc105b10 0x7f3bbc19b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.295+0000 7f3bba800640 1 --2- 172.21.15.72:0/830401928 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3bbc108440 0x7f3bbc1a21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.295+0000 7f3bb9e00640 1 -- 172.21.15.72:0/830401928 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3bbc105b10 msgr2=0x7f3bbc19b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:31.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.295+0000 7f3bb9e00640 1 --2- 172.21.15.72:0/830401928 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3bbc105b10 0x7f3bbc19b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.295+0000 7f3bb9e00640 1 -- 172.21.15.72:0/830401928 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3bbc108440 msgr2=0x7f3bbc1a21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:31.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.295+0000 7f3bb9e00640 1 --2- 172.21.15.72:0/830401928 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3bbc108440 0x7f3bbc1a21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.295+0000 7f3bb9e00640 1 -- 172.21.15.72:0/830401928 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bbc1a2860 con 0x7f3bbc1031e0 2024-04-26T23:45:31.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.295+0000 7f3bb9400640 1 --2- 172.21.15.72:0/830401928 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3bbc105b10 0x7f3bbc19b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:45:31.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.296+0000 7f3bb9e00640 1 --2- 172.21.15.72:0/830401928 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bbc1031e0 0x7f3bbc19ac50 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3ba4000c00 tx=0x7f3ba4003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:31.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.296+0000 7f3bb2a00640 1 -- 172.21.15.72:0/830401928 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ba403f070 con 0x7f3bbc1031e0 2024-04-26T23:45:31.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.296+0000 7f3bb2a00640 1 -- 172.21.15.72:0/830401928 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ba4031c50 con 0x7f3bbc1031e0 2024-04-26T23:45:31.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.296+0000 7f3bbb200640 1 -- 172.21.15.72:0/830401928 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bbc1a2af0 con 0x7f3bbc1031e0 2024-04-26T23:45:31.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.296+0000 7f3bb2a00640 1 -- 172.21.15.72:0/830401928 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ba4035030 con 0x7f3bbc1031e0 2024-04-26T23:45:31.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.296+0000 7f3bbb200640 1 -- 172.21.15.72:0/830401928 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bbc1a2fe0 con 0x7f3bbc1031e0 2024-04-26T23:45:31.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.299+0000 7f3bc3d9d740 1 Processor -- start 2024-04-26T23:45:31.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.299+0000 7f3bb2a00640 1 -- 172.21.15.72:0/830401928 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3ba4042070 con 0x7f3bbc1031e0 2024-04-26T23:45:31.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.300+0000 7f3bc3d9d740 1 -- start start 2024-04-26T23:45:31.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.300+0000 7f3bc3d9d740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5652111382c0 0x5652111386e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.300+0000 7f3bc3d9d740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x565211138cb0 0x56521113d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.300+0000 7f3bc3d9d740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56521113de20 0x5652111402b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.300+0000 7f3bc3d9d740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x56521109c600 con 0x5652111382c0 2024-04-26T23:45:31.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.300+0000 7f3bc3d9d740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x565211039100 con 0x56521113de20 2024-04-26T23:45:31.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.300+0000 7f3bc3d9d740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x5652110985c0 con 0x565211138cb0 2024-04-26T23:45:31.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.300+0000 7f3bba800640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5652111382c0 0x5652111386e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.300+0000 7f3bba800640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5652111382c0 0x5652111386e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54200/0 (socket says 172.21.15.72:54200) 2024-04-26T23:45:31.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.300+0000 7f3bba800640 1 -- 172.21.15.72:0/2709770615 learned_addr learned my addr 172.21.15.72:0/2709770615 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:45:31.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.300+0000 7f3bb9400640 1 --2- 172.21.15.72:0/2709770615 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56521113de20 0x5652111402b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.300+0000 7f3bb9e00640 1 --2- 172.21.15.72:0/2709770615 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x565211138cb0 0x56521113d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.301+0000 7f3bb9400640 1 -- 172.21.15.72:0/2709770615 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x565211138cb0 msgr2=0x56521113d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:45:31.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.301+0000 7f3bb9400640 1 --2- 172.21.15.72:0/2709770615 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x565211138cb0 0x56521113d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.301+0000 7f3bb9400640 1 -- 172.21.15.72:0/2709770615 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5652111382c0 msgr2=0x5652111386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:45:31.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.301+0000 7f3bb9400640 1 --2- 172.21.15.72:0/2709770615 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5652111382c0 0x5652111386e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.301+0000 7f3bb9400640 1 -- 172.21.15.72:0/2709770615 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565210dae2d0 con 0x56521113de20 2024-04-26T23:45:31.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.301+0000 7f3bb9400640 1 --2- 172.21.15.72:0/2709770615 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56521113de20 0x5652111402b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc06a620 tx=0x7f3bac011c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c59f21310dff065d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:31.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.301+0000 7f3bb2a00640 1 --2- 172.21.15.72:0/830401928 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f3ba00a3c70 0x7f3ba00a6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.301+0000 7f3bab400640 1 -- 172.21.15.72:0/2709770615 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bac003c30 con 0x56521113de20 2024-04-26T23:45:31.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.301+0000 7f3bab400640 1 -- 172.21.15.72:0/2709770615 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3bac00fe20 con 0x56521113de20 2024-04-26T23:45:31.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.301+0000 7f3bab400640 1 -- 172.21.15.72:0/2709770615 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bac0193e0 con 0x56521113de20 2024-04-26T23:45:31.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.301+0000 7f3bb2a00640 1 -- 172.21.15.72:0/830401928 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f3ba40f10b0 con 0x7f3bbc1031e0 2024-04-26T23:45:31.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.301+0000 7f3bba800640 1 --2- 172.21.15.72:0/830401928 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f3ba00a3c70 0x7f3ba00a6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.302+0000 7f3bc3d9d740 1 -- 172.21.15.72:0/2709770615 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56521113de20 msgr2=0x5652111402b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:31.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.302+0000 7f3bc3d9d740 1 --2- 172.21.15.72:0/2709770615 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56521113de20 0x5652111402b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc06a620 tx=0x7f3bac011c70 comp rx=0 tx=0).stop 2024-04-26T23:45:31.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.302+0000 7f3bc3d9d740 1 -- 172.21.15.72:0/2709770615 shutdown_connections 2024-04-26T23:45:31.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.302+0000 7f3bc3d9d740 1 --2- 172.21.15.72:0/2709770615 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56521113de20 0x5652111402b0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.302+0000 7f3bc3d9d740 1 --2- 172.21.15.72:0/2709770615 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x565211138cb0 0x56521113d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.302+0000 7f3bc3d9d740 1 --2- 172.21.15.72:0/2709770615 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5652111382c0 0x5652111386e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.302+0000 7f3bc3d9d740 1 -- 172.21.15.72:0/2709770615 >> 172.21.15.72:0/2709770615 conn(0x5652110ccd10 msgr2=0x565211133d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:31.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.302+0000 7f3bba800640 1 --2- 172.21.15.72:0/830401928 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f3ba00a3c70 0x7f3ba00a6130 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x565211139510 tx=0x7f3bb4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:31.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.302+0000 7f3bc3d9d740 1 -- 172.21.15.72:0/2709770615 shutdown_connections 2024-04-26T23:45:31.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.302+0000 7f3bc3d9d740 1 -- 172.21.15.72:0/2709770615 wait complete. 2024-04-26T23:45:31.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.303+0000 7f3bc3d9d740 1 -- start start 2024-04-26T23:45:31.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.303+0000 7f3bc3d9d740 1 Processor -- start 2024-04-26T23:45:31.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.303+0000 7f3bc3d9d740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5652111382c0 0x5652111426e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.303+0000 7f3bc3d9d740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x565211138cb0 0x565211142c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.303+0000 7f3bc3d9d740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56521113de20 0x565211149c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.303+0000 7f3bc3d9d740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x565211140dd0 con 0x56521113de20 2024-04-26T23:45:31.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.303+0000 7f3bc3d9d740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x565211023f70 con 0x5652111382c0 2024-04-26T23:45:31.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.303+0000 7f3bc3d9d740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x565211140f50 con 0x565211138cb0 2024-04-26T23:45:31.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.304+0000 7f3bc3d9d740 10 client.0 _ll_register_callbacks cb 0x565211092480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:45:31.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.304+0000 7f3bb9e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5652111382c0 0x5652111426e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.304+0000 7f3bb9e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5652111382c0 0x5652111426e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:52812/0 (socket says 172.21.15.72:52812) 2024-04-26T23:45:31.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.304+0000 7f3bb9e00640 1 -- 172.21.15.72:0/617677525 learned_addr learned my addr 172.21.15.72:0/617677525 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:45:31.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.304+0000 7f3bb9400640 1 --2- 172.21.15.72:0/617677525 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x565211138cb0 0x565211142c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.304+0000 7f3bba800640 1 --2- 172.21.15.72:0/617677525 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56521113de20 0x565211149c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.304+0000 7f3bb9e00640 1 -- 172.21.15.72:0/617677525 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x565211138cb0 msgr2=0x565211142c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:45:31.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.304+0000 7f3bb9e00640 1 --2- 172.21.15.72:0/617677525 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x565211138cb0 0x565211142c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.304+0000 7f3bb9e00640 1 -- 172.21.15.72:0/617677525 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56521113de20 msgr2=0x565211149c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:45:31.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.304+0000 7f3bb9e00640 1 --2- 172.21.15.72:0/617677525 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56521113de20 0x565211149c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.304+0000 7f3bb9e00640 1 -- 172.21.15.72:0/617677525 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565210e786b0 con 0x5652111382c0 2024-04-26T23:45:31.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.304+0000 7f3bba800640 1 --2- 172.21.15.72:0/617677525 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56521113de20 0x565211149c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:45:31.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.304+0000 7f3bb9e00640 1 --2- 172.21.15.72:0/617677525 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5652111382c0 0x5652111426e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3ba40bcd70 tx=0x7f3ba4009620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c689c37962dcc81d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:31.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.304+0000 7f3b9fe00640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-04-26T23:45:31.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.305+0000 7f3b9fe00640 1 -- 172.21.15.72:0/617677525 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ba4049070 con 0x5652111382c0 2024-04-26T23:45:31.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.305+0000 7f3b9fe00640 1 -- 172.21.15.72:0/617677525 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f3ba40ee860 con 0x5652111382c0 2024-04-26T23:45:31.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.305+0000 7f3bc3d9d740 1 -- 172.21.15.72:0/617677525 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3bac00fe20 con 0x5652111382c0 2024-04-26T23:45:31.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.305+0000 7f3b9fe00640 1 -- 172.21.15.72:0/617677525 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f3ba40bc740 con 0x5652111382c0 2024-04-26T23:45:31.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.305+0000 7f3bc3d9d740 10 client.25184 fetch_fsmap learned FSMap version 23 2024-04-26T23:45:31.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.305+0000 7f3bc3d9d740 1 -- 172.21.15.72:0/617677525 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x56521114cc40 con 0x5652111382c0 2024-04-26T23:45:31.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.305+0000 7f3b9fe00640 1 -- 172.21.15.72:0/617677525 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3bac00fe20 con 0x5652111382c0 2024-04-26T23:45:31.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.305+0000 7f3b9fe00640 1 -- 172.21.15.72:0/617677525 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7f3ba40f3110 con 0x5652111382c0 2024-04-26T23:45:31.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.306+0000 7f3bc3d9d740 10 client.25184 fetch_fsmap finished waiting for FSMap version 23 2024-04-26T23:45:31.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.306+0000 7f3bc3d9d740 10 client.25184 resolve_mds: resolved name 'g' to daemon mds.g 2024-04-26T23:45:31.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.306+0000 7f3bc3d9d740 20 client.25184 populate_metadata read hostname 'smithi072' 2024-04-26T23:45:31.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.306+0000 7f3bc3d9d740 1 --2- 172.21.15.72:0/617677525 >> [v2:172.21.15.72:6838/1403162102,v1:172.21.15.72:6839/1403162102] conn(0x56521114f280 0x5652111506f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.306+0000 7f3bc3d9d740 4 client.25184 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:45:31.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.306+0000 7f3bc3d9d740 1 -- 172.21.15.72:0/617677525 --> [v2:172.21.15.72:6838/1403162102,v1:172.21.15.72:6839/1403162102] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5652110aa210 con 0x56521114f280 2024-04-26T23:45:31.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.306+0000 7f3bb9400640 1 --2- 172.21.15.72:0/617677525 >> [v2:172.21.15.72:6838/1403162102,v1:172.21.15.72:6839/1403162102] conn(0x56521114f280 0x5652111506f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.307+0000 7f3bb9400640 1 --2- 172.21.15.72:0/617677525 >> [v2:172.21.15.72:6838/1403162102,v1:172.21.15.72:6839/1403162102] conn(0x56521114f280 0x5652111506f0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=86b100a1827f28e3 server_cookie=2aa4768a7c33d293 in_seq=0 out_seq=0 2024-04-26T23:45:31.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.307+0000 7f3b9fe00640 10 client.25184 ms_handle_connect on v2:172.21.15.72:6838/1403162102 2024-04-26T23:45:31.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.307+0000 7f3b9fe00640 1 -- 172.21.15.72:0/617677525 <== mds.2 v2:172.21.15.72:6838/1403162102 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5652110aa210 con 0x56521114f280 2024-04-26T23:45:31.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.307+0000 7f3b9fe00640 10 client.25184 handle_command_reply: tid=0 2024-04-26T23:45:31.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.307+0000 7f3bc3d9d740 1 client.25184 shutdown 2024-04-26T23:45:31.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.308+0000 7f3bc3d9d740 1 -- 172.21.15.72:0/617677525 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5652111382c0 msgr2=0x5652111426e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:31.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.308+0000 7f3bc3d9d740 1 --2- 172.21.15.72:0/617677525 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5652111382c0 0x5652111426e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3ba40bcd70 tx=0x7f3ba4009620 comp rx=0 tx=0).stop 2024-04-26T23:45:31.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.308+0000 7f3bc3d9d740 1 -- 172.21.15.72:0/617677525 shutdown_connections 2024-04-26T23:45:31.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.308+0000 7f3bc3d9d740 1 --2- 172.21.15.72:0/617677525 >> [v2:172.21.15.72:6838/1403162102,v1:172.21.15.72:6839/1403162102] conn(0x56521114f280 0x5652111506f0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.308+0000 7f3bc3d9d740 1 --2- 172.21.15.72:0/617677525 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56521113de20 0x565211149c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.308+0000 7f3b9fe00640 0 client.25184 ms_handle_reset on v2:172.21.15.72:6838/1403162102 2024-04-26T23:45:31.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.308+0000 7f3bc3d9d740 1 --2- 172.21.15.72:0/617677525 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x565211138cb0 0x565211142c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.308+0000 7f3bc3d9d740 1 --2- 172.21.15.72:0/617677525 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x5652111382c0 0x5652111426e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.308+0000 7f3bc3d9d740 1 -- 172.21.15.72:0/617677525 >> 172.21.15.72:0/617677525 conn(0x5652110ccd10 msgr2=0x565211132f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:31.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.308+0000 7f3bc3d9d740 1 -- 172.21.15.72:0/617677525 shutdown_connections 2024-04-26T23:45:31.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.308+0000 7f3bc3d9d740 1 -- 172.21.15.72:0/617677525 wait complete. 2024-04-26T23:45:31.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.308+0000 7f3bc3d9d740 20 client.25184 trim_cache size 0 max 16384 2024-04-26T23:45:31.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.328+0000 7f3bc3d9d740 1 Processor -- start 2024-04-26T23:45:31.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.329+0000 7f3bc3d9d740 1 -- start start 2024-04-26T23:45:31.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.329+0000 7f3bc3d9d740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5652111382c0 0x565211140210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.329+0000 7f3bc3d9d740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x565211138cb0 0x565211149bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.329+0000 7f3bc3d9d740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56521113de20 0x5652111422f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.329+0000 7f3bc3d9d740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x56521114ea30 con 0x565211138cb0 2024-04-26T23:45:31.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.329+0000 7f3bc3d9d740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x56521114f100 con 0x56521113de20 2024-04-26T23:45:31.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.329+0000 7f3bc3d9d740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x56521114f7d0 con 0x5652111382c0 2024-04-26T23:45:31.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.329+0000 7f3bb9400640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x565211138cb0 0x565211149bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.329+0000 7f3bb9e00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5652111382c0 0x565211140210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.329+0000 7f3bb9400640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x565211138cb0 0x565211149bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54222/0 (socket says 172.21.15.72:54222) 2024-04-26T23:45:31.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.329+0000 7f3bb9400640 1 -- 172.21.15.72:0/315883612 learned_addr learned my addr 172.21.15.72:0/315883612 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:45:31.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.329+0000 7f3bba800640 1 --2- 172.21.15.72:0/315883612 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56521113de20 0x5652111422f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.330+0000 7f3bb9e00640 1 -- 172.21.15.72:0/315883612 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56521113de20 msgr2=0x5652111422f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:45:31.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.330+0000 7f3bb9e00640 1 --2- 172.21.15.72:0/315883612 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56521113de20 0x5652111422f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.330+0000 7f3bb9e00640 1 -- 172.21.15.72:0/315883612 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x565211138cb0 msgr2=0x565211149bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:45:31.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.330+0000 7f3bb9e00640 1 --2- 172.21.15.72:0/315883612 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x565211138cb0 0x565211149bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.330+0000 7f3bb9e00640 1 -- 172.21.15.72:0/315883612 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565211141120 con 0x5652111382c0 2024-04-26T23:45:31.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.330+0000 7f3bb9400640 1 --2- 172.21.15.72:0/315883612 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x565211138cb0 0x565211149bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:45:31.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.330+0000 7f3bba800640 1 --2- 172.21.15.72:0/315883612 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56521113de20 0x5652111422f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:45:31.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.330+0000 7f3bb9e00640 1 --2- 172.21.15.72:0/315883612 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5652111382c0 0x565211140210 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3ba40b9720 tx=0x7f3ba40314c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7486cafea8d8a8d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:31.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.330+0000 7f3baaa00640 1 -- 172.21.15.72:0/315883612 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ba4049070 con 0x5652111382c0 2024-04-26T23:45:31.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.330+0000 7f3baaa00640 1 -- 172.21.15.72:0/315883612 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ba40bd4e0 con 0x5652111382c0 2024-04-26T23:45:31.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.330+0000 7f3baaa00640 1 -- 172.21.15.72:0/315883612 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ba4005d50 con 0x5652111382c0 2024-04-26T23:45:31.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.331+0000 7f3bc3d9d740 1 -- 172.21.15.72:0/315883612 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5652111382c0 msgr2=0x565211140210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:31.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.331+0000 7f3bc3d9d740 1 --2- 172.21.15.72:0/315883612 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5652111382c0 0x565211140210 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3ba40b9720 tx=0x7f3ba40314c0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.331+0000 7f3bc3d9d740 1 -- 172.21.15.72:0/315883612 shutdown_connections 2024-04-26T23:45:31.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.331+0000 7f3bc3d9d740 1 --2- 172.21.15.72:0/315883612 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56521113de20 0x5652111422f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.331+0000 7f3bc3d9d740 1 --2- 172.21.15.72:0/315883612 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x565211138cb0 0x565211149bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.331+0000 7f3bc3d9d740 1 --2- 172.21.15.72:0/315883612 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5652111382c0 0x565211140210 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.331+0000 7f3bc3d9d740 1 -- 172.21.15.72:0/315883612 >> 172.21.15.72:0/315883612 conn(0x5652110ccd10 msgr2=0x56521113a010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:31.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.331+0000 7f3bc3d9d740 1 -- 172.21.15.72:0/315883612 shutdown_connections 2024-04-26T23:45:31.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.331+0000 7f3bc3d9d740 1 -- 172.21.15.72:0/315883612 wait complete. 2024-04-26T23:45:31.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.331+0000 7f3bc3d9d740 1 -- start start 2024-04-26T23:45:31.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.332+0000 7f3bc3d9d740 1 Processor -- start 2024-04-26T23:45:31.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.332+0000 7f3bc3d9d740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5652111382c0 0x5652111455a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.332+0000 7f3bc3d9d740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565211138cb0 0x565211145ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.332+0000 7f3bc3d9d740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56521113de20 0x565211148ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.332+0000 7f3bc3d9d740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x565211150570 con 0x56521113de20 2024-04-26T23:45:31.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.332+0000 7f3bc3d9d740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x56521114fea0 con 0x565211138cb0 2024-04-26T23:45:31.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.332+0000 7f3bc3d9d740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x565211150c40 con 0x5652111382c0 2024-04-26T23:45:31.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.332+0000 7f3bc3d9d740 10 client.0 _ll_register_callbacks cb 0x7f3ba40f4850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:45:31.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.332+0000 7f3bba800640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56521113de20 0x565211148ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.332+0000 7f3bb9e00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5652111382c0 0x5652111455a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.332+0000 7f3bba800640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56521113de20 0x565211148ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54232/0 (socket says 172.21.15.72:54232) 2024-04-26T23:45:31.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.332+0000 7f3bba800640 1 -- 172.21.15.72:0/1104047994 learned_addr learned my addr 172.21.15.72:0/1104047994 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:45:31.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.332+0000 7f3bb9400640 1 --2- 172.21.15.72:0/1104047994 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565211138cb0 0x565211145ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.333+0000 7f3bb9e00640 1 -- 172.21.15.72:0/1104047994 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565211138cb0 msgr2=0x565211145ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:45:31.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.333+0000 7f3bb9e00640 1 --2- 172.21.15.72:0/1104047994 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565211138cb0 0x565211145ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.333+0000 7f3bb9e00640 1 -- 172.21.15.72:0/1104047994 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56521113de20 msgr2=0x565211148ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:45:31.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.333+0000 7f3bb9e00640 1 --2- 172.21.15.72:0/1104047994 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56521113de20 0x565211148ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.333+0000 7f3bb9e00640 1 -- 172.21.15.72:0/1104047994 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56521112f460 con 0x5652111382c0 2024-04-26T23:45:31.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.333+0000 7f3bb9e00640 1 --2- 172.21.15.72:0/1104047994 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5652111382c0 0x5652111455a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3ba40bcae0 tx=0x7f3ba40b9660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=89b72ca90acfbca8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:31.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.333+0000 7f3b9fe00640 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2024-04-26T23:45:31.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.333+0000 7f3b9fe00640 1 -- 172.21.15.72:0/1104047994 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ba4049070 con 0x5652111382c0 2024-04-26T23:45:31.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.333+0000 7f3bc3d9d740 1 -- 172.21.15.72:0/1104047994 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ba40bd4e0 con 0x5652111382c0 2024-04-26T23:45:31.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.333+0000 7f3b9fe00640 1 -- 172.21.15.72:0/1104047994 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f3ba40d6140 con 0x5652111382c0 2024-04-26T23:45:31.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.333+0000 7f3b9fe00640 1 -- 172.21.15.72:0/1104047994 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f3ba40baf20 con 0x5652111382c0 2024-04-26T23:45:31.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.334+0000 7f3bc3d9d740 10 client.25162 fetch_fsmap learned FSMap version 23 2024-04-26T23:45:31.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.334+0000 7f3bc3d9d740 1 -- 172.21.15.72:0/1104047994 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x56521112f460 con 0x5652111382c0 2024-04-26T23:45:31.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.334+0000 7f3b9fe00640 1 -- 172.21.15.72:0/1104047994 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ba40bd4e0 con 0x5652111382c0 2024-04-26T23:45:31.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.334+0000 7f3b9fe00640 1 -- 172.21.15.72:0/1104047994 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7f3ba40bd040 con 0x5652111382c0 2024-04-26T23:45:31.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.334+0000 7f3bc3d9d740 10 client.25162 fetch_fsmap finished waiting for FSMap version 23 2024-04-26T23:45:31.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.334+0000 7f3bc3d9d740 10 client.25162 resolve_mds: resolved name 'g' to daemon mds.g 2024-04-26T23:45:31.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.334+0000 7f3bc3d9d740 20 client.25162 populate_metadata read hostname 'smithi072' 2024-04-26T23:45:31.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.334+0000 7f3bc3d9d740 1 --2- 172.21.15.72:0/1104047994 >> [v2:172.21.15.72:6838/1403162102,v1:172.21.15.72:6839/1403162102] conn(0x5652111b96f0 0x5652111b8400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.334+0000 7f3bc3d9d740 4 client.25162 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T23:45:31.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.334+0000 7f3bc3d9d740 1 -- 172.21.15.72:0/1104047994 --> [v2:172.21.15.72:6838/1403162102,v1:172.21.15.72:6839/1403162102] -- command(tid 0: {"prefix": "status"}) -- 0x5652110a4080 con 0x5652111b96f0 2024-04-26T23:45:31.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.334+0000 7f3bb9400640 1 --2- 172.21.15.72:0/1104047994 >> [v2:172.21.15.72:6838/1403162102,v1:172.21.15.72:6839/1403162102] conn(0x5652111b96f0 0x5652111b8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.335+0000 7f3bb9400640 1 --2- 172.21.15.72:0/1104047994 >> [v2:172.21.15.72:6838/1403162102,v1:172.21.15.72:6839/1403162102] conn(0x5652111b96f0 0x5652111b8400 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a84fe330885c375b server_cookie=cc40763f0727cd36 in_seq=0 out_seq=0 2024-04-26T23:45:31.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.335+0000 7f3b9fe00640 10 client.25162 ms_handle_connect on v2:172.21.15.72:6838/1403162102 2024-04-26T23:45:31.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.335+0000 7f3b9fe00640 1 -- 172.21.15.72:0/1104047994 <== mds.2 v2:172.21.15.72:6838/1403162102 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5652110a4080 con 0x5652111b96f0 2024-04-26T23:45:31.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.335+0000 7f3b9fe00640 10 client.25162 handle_command_reply: tid=0 2024-04-26T23:45:31.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.335+0000 7f3bc3d9d740 1 client.25162 shutdown 2024-04-26T23:45:31.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.335+0000 7f3bc3d9d740 1 -- 172.21.15.72:0/1104047994 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5652111382c0 msgr2=0x5652111455a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:31.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.335+0000 7f3bc3d9d740 1 --2- 172.21.15.72:0/1104047994 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5652111382c0 0x5652111455a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3ba40bcae0 tx=0x7f3ba40b9660 comp rx=0 tx=0).stop 2024-04-26T23:45:31.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.335+0000 7f3bc3d9d740 1 -- 172.21.15.72:0/1104047994 shutdown_connections 2024-04-26T23:45:31.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.335+0000 7f3bc3d9d740 1 --2- 172.21.15.72:0/1104047994 >> [v2:172.21.15.72:6838/1403162102,v1:172.21.15.72:6839/1403162102] conn(0x5652111b96f0 0x5652111b8400 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.335+0000 7f3bc3d9d740 1 --2- 172.21.15.72:0/1104047994 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x565211138cb0 0x565211145ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.335+0000 7f3b9fe00640 0 client.25162 ms_handle_reset on v2:172.21.15.72:6838/1403162102 2024-04-26T23:45:31.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.335+0000 7f3bc3d9d740 1 --2- 172.21.15.72:0/1104047994 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56521113de20 0x565211148ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.335+0000 7f3bc3d9d740 1 --2- 172.21.15.72:0/1104047994 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x5652111382c0 0x5652111455a0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.335+0000 7f3bc3d9d740 1 -- 172.21.15.72:0/1104047994 >> 172.21.15.72:0/1104047994 conn(0x5652110ccd10 msgr2=0x5652111422f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:31.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.336+0000 7f3bc3d9d740 1 -- 172.21.15.72:0/1104047994 shutdown_connections 2024-04-26T23:45:31.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.336+0000 7f3bc3d9d740 1 -- 172.21.15.72:0/1104047994 wait complete. 2024-04-26T23:45:31.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.336+0000 7f3bc3d9d740 20 client.25162 trim_cache size 0 max 16384 2024-04-26T23:45:31.344 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-04-26T23:45:31.344 INFO:teuthology.orchestra.run.smithi072.stdout: "cluster_fsid": "3c8453ae-0425-11ef-bc93-c7b262605968", 2024-04-26T23:45:31.344 INFO:teuthology.orchestra.run.smithi072.stdout: "whoami": 2, 2024-04-26T23:45:31.344 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 24553, 2024-04-26T23:45:31.344 INFO:teuthology.orchestra.run.smithi072.stdout: "want_state": "up:active", 2024-04-26T23:45:31.344 INFO:teuthology.orchestra.run.smithi072.stdout: "state": "up:active", 2024-04-26T23:45:31.344 INFO:teuthology.orchestra.run.smithi072.stdout: "fs_name": "cephfs", 2024-04-26T23:45:31.344 INFO:teuthology.orchestra.run.smithi072.stdout: "rank_uptime": 122.152251991, 2024-04-26T23:45:31.344 INFO:teuthology.orchestra.run.smithi072.stdout: "mdsmap_epoch": 23, 2024-04-26T23:45:31.344 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_epoch": 93, 2024-04-26T23:45:31.344 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_epoch_barrier": 89, 2024-04-26T23:45:31.344 INFO:teuthology.orchestra.run.smithi072.stdout: "uptime": 161.48008010999999 2024-04-26T23:45:31.344 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-04-26T23:45:31.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.336+0000 7f3bbb200640 1 -- 172.21.15.72:0/830401928 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f3ba00a3c70 msgr2=0x7f3ba00a6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:31.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.336+0000 7f3bbb200640 1 --2- 172.21.15.72:0/830401928 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f3ba00a3c70 0x7f3ba00a6130 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x565211139510 tx=0x7f3bb4005f90 comp rx=0 tx=0).stop 2024-04-26T23:45:31.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.336+0000 7f3bbb200640 1 -- 172.21.15.72:0/830401928 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bbc1031e0 msgr2=0x7f3bbc19ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:31.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.336+0000 7f3bbb200640 1 --2- 172.21.15.72:0/830401928 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bbc1031e0 0x7f3bbc19ac50 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3ba4000c00 tx=0x7f3ba4003c10 comp rx=0 tx=0).stop 2024-04-26T23:45:31.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.337+0000 7f3bbb200640 1 -- 172.21.15.72:0/830401928 shutdown_connections 2024-04-26T23:45:31.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.337+0000 7f3bbb200640 1 --2- 172.21.15.72:0/830401928 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f3ba00a3c70 0x7f3ba00a6130 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.337+0000 7f3bbb200640 1 --2- 172.21.15.72:0/830401928 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3bbc108440 0x7f3bbc1a21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.337+0000 7f3bbb200640 1 --2- 172.21.15.72:0/830401928 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3bbc105b10 0x7f3bbc19b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.337+0000 7f3bbb200640 1 --2- 172.21.15.72:0/830401928 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bbc1031e0 0x7f3bbc19ac50 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.337+0000 7f3bbb200640 1 -- 172.21.15.72:0/830401928 >> 172.21.15.72:0/830401928 conn(0x7f3bbc0fcbb0 msgr2=0x7f3bbc102910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:31.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.337+0000 7f3bbb200640 1 -- 172.21.15.72:0/830401928 shutdown_connections 2024-04-26T23:45:31.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.337+0000 7f3bbb200640 1 -- 172.21.15.72:0/830401928 wait complete. 2024-04-26T23:45:31.358 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-26T23:45:31.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.559+0000 7faa08000640 1 -- 172.21.15.72:0/1290045335 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa00105da0 msgr2=0x7faa00106220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:31.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.559+0000 7faa08000640 1 --2- 172.21.15.72:0/1290045335 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa00105da0 0x7faa00106220 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa9ec009860 tx=0x7fa9ec02a340 comp rx=0 tx=0).stop 2024-04-26T23:45:31.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.559+0000 7faa08000640 1 -- 172.21.15.72:0/1290045335 shutdown_connections 2024-04-26T23:45:31.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.559+0000 7faa08000640 1 --2- 172.21.15.72:0/1290045335 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa00106760 0x7faa0010cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.559+0000 7faa08000640 1 --2- 172.21.15.72:0/1290045335 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa00105da0 0x7faa00106220 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.559+0000 7faa08000640 1 --2- 172.21.15.72:0/1290045335 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faa00104bb0 0x7faa00104fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.559+0000 7faa08000640 1 -- 172.21.15.72:0/1290045335 >> 172.21.15.72:0/1290045335 conn(0x7faa00100120 msgr2=0x7faa00102560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:31.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.559+0000 7faa08000640 1 -- 172.21.15.72:0/1290045335 shutdown_connections 2024-04-26T23:45:31.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.559+0000 7faa08000640 1 -- 172.21.15.72:0/1290045335 wait complete. 2024-04-26T23:45:31.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.560+0000 7faa08000640 1 Processor -- start 2024-04-26T23:45:31.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.560+0000 7faa08000640 1 -- start start 2024-04-26T23:45:31.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.561+0000 7faa08000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faa00104bb0 0x7faa0019ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.561+0000 7faa08000640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa00105da0 0x7faa0019b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.561+0000 7faa08000640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa00106760 0x7faa001a2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.561+0000 7faa08000640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7faa001106c0 con 0x7faa00105da0 2024-04-26T23:45:31.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.561+0000 7faa08000640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7faa00110540 con 0x7faa00104bb0 2024-04-26T23:45:31.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.561+0000 7faa08000640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7faa00110840 con 0x7faa00106760 2024-04-26T23:45:31.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.561+0000 7faa05600640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faa00104bb0 0x7faa0019ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.561+0000 7faa05600640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faa00104bb0 0x7faa0019ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:52854/0 (socket says 172.21.15.72:52854) 2024-04-26T23:45:31.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.561+0000 7faa05600640 1 -- 172.21.15.72:0/999100967 learned_addr learned my addr 172.21.15.72:0/999100967 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:45:31.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.561+0000 7faa04c00640 1 --2- 172.21.15.72:0/999100967 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa00105da0 0x7faa0019b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.561+0000 7faa06000640 1 --2- 172.21.15.72:0/999100967 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa00106760 0x7faa001a2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.561+0000 7faa05600640 1 -- 172.21.15.72:0/999100967 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa00106760 msgr2=0x7faa001a2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:31.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.561+0000 7faa05600640 1 --2- 172.21.15.72:0/999100967 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa00106760 0x7faa001a2390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.561+0000 7faa05600640 1 -- 172.21.15.72:0/999100967 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa00105da0 msgr2=0x7faa0019b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:31.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.561+0000 7faa05600640 1 --2- 172.21.15.72:0/999100967 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa00105da0 0x7faa0019b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.561+0000 7faa05600640 1 -- 172.21.15.72:0/999100967 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa001a2a00 con 0x7faa00104bb0 2024-04-26T23:45:31.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.561+0000 7faa06000640 1 --2- 172.21.15.72:0/999100967 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa00106760 0x7faa001a2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:45:31.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.562+0000 7faa05600640 1 --2- 172.21.15.72:0/999100967 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faa00104bb0 0x7faa0019ade0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa9f400e9c0 tx=0x7fa9f400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:31.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.562+0000 7fa9fe000640 1 -- 172.21.15.72:0/999100967 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f400cd80 con 0x7faa00104bb0 2024-04-26T23:45:31.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.562+0000 7fa9fe000640 1 -- 172.21.15.72:0/999100967 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9f40035f0 con 0x7faa00104bb0 2024-04-26T23:45:31.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.562+0000 7faa08000640 1 -- 172.21.15.72:0/999100967 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa001a2cf0 con 0x7faa00104bb0 2024-04-26T23:45:31.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.562+0000 7fa9fe000640 1 -- 172.21.15.72:0/999100967 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f4010690 con 0x7faa00104bb0 2024-04-26T23:45:31.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.562+0000 7faa08000640 1 -- 172.21.15.72:0/999100967 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa001a31c0 con 0x7faa00104bb0 2024-04-26T23:45:31.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.565+0000 7faa0adb1740 1 Processor -- start 2024-04-26T23:45:31.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.565+0000 7faa0adb1740 1 -- start start 2024-04-26T23:45:31.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.566+0000 7fa9fe000640 1 -- 172.21.15.72:0/999100967 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fa9f40037a0 con 0x7faa00104bb0 2024-04-26T23:45:31.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.566+0000 7faa0adb1740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56482df7f0f0 0x56482df7f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.566+0000 7faa0adb1740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56482df7fae0 0x56482df845e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.566+0000 7faa0adb1740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56482df84c50 0x56482df870e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.566+0000 7faa0adb1740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x56482dee3600 con 0x56482df7fae0 2024-04-26T23:45:31.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.566+0000 7faa0adb1740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x56482de80100 con 0x56482df84c50 2024-04-26T23:45:31.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.566+0000 7faa0adb1740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x56482dedf5c0 con 0x56482df7f0f0 2024-04-26T23:45:31.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.566+0000 7faa05600640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56482df7fae0 0x56482df845e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.566+0000 7faa05600640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56482df7fae0 0x56482df845e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54266/0 (socket says 172.21.15.72:54266) 2024-04-26T23:45:31.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.566+0000 7faa05600640 1 -- 172.21.15.72:0/3097434276 learned_addr learned my addr 172.21.15.72:0/3097434276 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:45:31.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.566+0000 7faa06000640 1 --2- 172.21.15.72:0/3097434276 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56482df7f0f0 0x56482df7f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.566+0000 7faa04c00640 1 --2- 172.21.15.72:0/3097434276 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56482df84c50 0x56482df870e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.567+0000 7faa05600640 1 -- 172.21.15.72:0/3097434276 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56482df7f0f0 msgr2=0x56482df7f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:45:31.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.567+0000 7faa05600640 1 --2- 172.21.15.72:0/3097434276 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56482df7f0f0 0x56482df7f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.567+0000 7faa05600640 1 -- 172.21.15.72:0/3097434276 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56482df84c50 msgr2=0x56482df870e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:45:31.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.567+0000 7faa05600640 1 --2- 172.21.15.72:0/3097434276 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56482df84c50 0x56482df870e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.567+0000 7faa05600640 1 -- 172.21.15.72:0/3097434276 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56482dbf52d0 con 0x56482df7fae0 2024-04-26T23:45:31.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.567+0000 7faa05600640 1 --2- 172.21.15.72:0/3097434276 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56482df7fae0 0x56482df845e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa9f40d4d50 tx=0x7fa9f40d4e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acddc444f38cc974 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:31.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.567+0000 7fa9f2600640 1 -- 172.21.15.72:0/3097434276 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f40d70d0 con 0x56482df7fae0 2024-04-26T23:45:31.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.567+0000 7fa9f2600640 1 -- 172.21.15.72:0/3097434276 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9f40d7270 con 0x56482df7fae0 2024-04-26T23:45:31.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.567+0000 7fa9f2600640 1 -- 172.21.15.72:0/3097434276 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f40d5e60 con 0x56482df7fae0 2024-04-26T23:45:31.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.567+0000 7fa9fe000640 1 --2- 172.21.15.72:0/999100967 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fa9e80a3c40 0x7fa9e80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.568+0000 7fa9fe000640 1 -- 172.21.15.72:0/999100967 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7fa9f40d0df0 con 0x7faa00104bb0 2024-04-26T23:45:31.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.568+0000 7faa06000640 1 --2- 172.21.15.72:0/999100967 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fa9e80a3c40 0x7fa9e80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.568+0000 7faa0adb1740 1 -- 172.21.15.72:0/3097434276 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56482df7fae0 msgr2=0x56482df845e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:31.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.568+0000 7faa0adb1740 1 --2- 172.21.15.72:0/3097434276 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56482df7fae0 0x56482df845e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa9f40d4d50 tx=0x7fa9f40d4e40 comp rx=0 tx=0).stop 2024-04-26T23:45:31.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.568+0000 7faa0adb1740 1 -- 172.21.15.72:0/3097434276 shutdown_connections 2024-04-26T23:45:31.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.568+0000 7faa0adb1740 1 --2- 172.21.15.72:0/3097434276 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56482df84c50 0x56482df870e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.568+0000 7faa0adb1740 1 --2- 172.21.15.72:0/3097434276 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56482df7fae0 0x56482df845e0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.568+0000 7faa0adb1740 1 --2- 172.21.15.72:0/3097434276 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56482df7f0f0 0x56482df7f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.568+0000 7faa0adb1740 1 -- 172.21.15.72:0/3097434276 >> 172.21.15.72:0/3097434276 conn(0x56482df13d10 msgr2=0x56482df7ab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:31.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.568+0000 7faa0adb1740 1 -- 172.21.15.72:0/3097434276 shutdown_connections 2024-04-26T23:45:31.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.568+0000 7faa0adb1740 1 -- 172.21.15.72:0/3097434276 wait complete. 2024-04-26T23:45:31.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.568+0000 7faa06000640 1 --2- 172.21.15.72:0/999100967 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fa9e80a3c40 0x7fa9e80a6100 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x56482df80340 tx=0x7fa9f8011040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:31.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.569+0000 7faa0adb1740 1 -- start start 2024-04-26T23:45:31.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.569+0000 7faa0adb1740 1 Processor -- start 2024-04-26T23:45:31.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.569+0000 7faa0adb1740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56482df7f0f0 0x56482df89540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.569+0000 7faa0adb1740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56482df7fae0 0x56482df89a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.569+0000 7faa0adb1740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56482df84c50 0x56482df90ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.569+0000 7faa0adb1740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x56482df87c00 con 0x56482df84c50 2024-04-26T23:45:31.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.569+0000 7faa0adb1740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x56482de6af70 con 0x56482df7f0f0 2024-04-26T23:45:31.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.570+0000 7faa0adb1740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x56482df87d80 con 0x56482df7fae0 2024-04-26T23:45:31.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.570+0000 7faa0adb1740 10 client.0 _ll_register_callbacks cb 0x56482ded9480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:45:31.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.570+0000 7faa06000640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56482df84c50 0x56482df90ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.570+0000 7faa06000640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56482df84c50 0x56482df90ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54276/0 (socket says 172.21.15.72:54276) 2024-04-26T23:45:31.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.570+0000 7faa06000640 1 -- 172.21.15.72:0/1205355818 learned_addr learned my addr 172.21.15.72:0/1205355818 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:45:31.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.570+0000 7faa05600640 1 --2- 172.21.15.72:0/1205355818 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56482df7f0f0 0x56482df89540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.570+0000 7faa04c00640 1 --2- 172.21.15.72:0/1205355818 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56482df7fae0 0x56482df89a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.570+0000 7faa06000640 1 -- 172.21.15.72:0/1205355818 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56482df7fae0 msgr2=0x56482df89a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:45:31.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.570+0000 7faa06000640 1 --2- 172.21.15.72:0/1205355818 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56482df7fae0 0x56482df89a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.570+0000 7faa06000640 1 -- 172.21.15.72:0/1205355818 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56482df7f0f0 msgr2=0x56482df89540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:45:31.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.570+0000 7faa06000640 1 --2- 172.21.15.72:0/1205355818 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56482df7f0f0 0x56482df89540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.570+0000 7faa06000640 1 -- 172.21.15.72:0/1205355818 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56482dcbf6b0 con 0x56482df84c50 2024-04-26T23:45:31.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.571+0000 7faa06000640 1 --2- 172.21.15.72:0/1205355818 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56482df84c50 0x56482df90ac0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa9f800bf60 tx=0x7fa9f8004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72fced53aa9012d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:31.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.571+0000 7fa9e6a00640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-04-26T23:45:31.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.571+0000 7fa9e6a00640 1 -- 172.21.15.72:0/1205355818 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f8029020 con 0x56482df84c50 2024-04-26T23:45:31.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.571+0000 7fa9e6a00640 1 -- 172.21.15.72:0/1205355818 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7fa9f80255f0 con 0x56482df84c50 2024-04-26T23:45:31.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.571+0000 7faa0adb1740 1 -- 172.21.15.72:0/1205355818 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9f40d7270 con 0x56482df84c50 2024-04-26T23:45:31.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.571+0000 7fa9e6a00640 1 -- 172.21.15.72:0/1205355818 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fa9f8034050 con 0x56482df84c50 2024-04-26T23:45:31.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.571+0000 7faa0adb1740 10 client.15099 fetch_fsmap learned FSMap version 23 2024-04-26T23:45:31.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.571+0000 7faa0adb1740 1 -- 172.21.15.72:0/1205355818 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x56482df93b70 con 0x56482df84c50 2024-04-26T23:45:31.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.572+0000 7fa9e6a00640 1 -- 172.21.15.72:0/1205355818 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9f40d7270 con 0x56482df84c50 2024-04-26T23:45:31.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.572+0000 7fa9e6a00640 1 -- 172.21.15.72:0/1205355818 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7fa9f8024030 con 0x56482df84c50 2024-04-26T23:45:31.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.572+0000 7faa0adb1740 10 client.15099 fetch_fsmap finished waiting for FSMap version 23 2024-04-26T23:45:31.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.572+0000 7faa0adb1740 10 client.15099 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-26T23:45:31.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.572+0000 7faa0adb1740 20 client.15099 populate_metadata read hostname 'smithi072' 2024-04-26T23:45:31.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.572+0000 7faa0adb1740 1 --2- 172.21.15.72:0/1205355818 >> [v2:172.21.15.72:6840/1356345077,v1:172.21.15.72:6841/1356345077] conn(0x56482df96250 0x56482df976c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.572+0000 7faa0adb1740 4 client.15099 mds_command: new command op to 24571 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:45:31.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.572+0000 7faa0adb1740 1 -- 172.21.15.72:0/1205355818 --> [v2:172.21.15.72:6840/1356345077,v1:172.21.15.72:6841/1356345077] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56482def1210 con 0x56482df96250 2024-04-26T23:45:31.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.572+0000 7faa05600640 1 --2- 172.21.15.72:0/1205355818 >> [v2:172.21.15.72:6840/1356345077,v1:172.21.15.72:6841/1356345077] conn(0x56482df96250 0x56482df976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.573+0000 7fa9e6a00640 10 client.15099 ms_handle_connect on v2:172.21.15.72:6840/1356345077 2024-04-26T23:45:31.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.573+0000 7faa05600640 1 --2- 172.21.15.72:0/1205355818 >> [v2:172.21.15.72:6840/1356345077,v1:172.21.15.72:6841/1356345077] conn(0x56482df96250 0x56482df976c0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e096a388813dd777 server_cookie=f92b3481225fc70 in_seq=0 out_seq=0 2024-04-26T23:45:31.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.574+0000 7fa9e6a00640 1 -- 172.21.15.72:0/1205355818 <== mds.1 v2:172.21.15.72:6840/1356345077 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56482def1210 con 0x56482df96250 2024-04-26T23:45:31.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.574+0000 7fa9e6a00640 10 client.15099 handle_command_reply: tid=0 2024-04-26T23:45:31.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.574+0000 7faa0adb1740 1 client.15099 shutdown 2024-04-26T23:45:31.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.574+0000 7faa0adb1740 1 -- 172.21.15.72:0/1205355818 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56482df84c50 msgr2=0x56482df90ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:31.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.574+0000 7faa0adb1740 1 --2- 172.21.15.72:0/1205355818 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56482df84c50 0x56482df90ac0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa9f800bf60 tx=0x7fa9f8004ab0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.574+0000 7faa0adb1740 1 -- 172.21.15.72:0/1205355818 shutdown_connections 2024-04-26T23:45:31.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.574+0000 7faa0adb1740 1 --2- 172.21.15.72:0/1205355818 >> [v2:172.21.15.72:6840/1356345077,v1:172.21.15.72:6841/1356345077] conn(0x56482df96250 0x56482df976c0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.574+0000 7faa0adb1740 1 --2- 172.21.15.72:0/1205355818 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56482df84c50 0x56482df90ac0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.574+0000 7faa0adb1740 1 --2- 172.21.15.72:0/1205355818 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56482df7fae0 0x56482df89a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.574+0000 7faa0adb1740 1 --2- 172.21.15.72:0/1205355818 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56482df7f0f0 0x56482df89540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.574+0000 7fa9e6a00640 0 client.15099 ms_handle_reset on v2:172.21.15.72:6840/1356345077 2024-04-26T23:45:31.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.574+0000 7faa0adb1740 1 -- 172.21.15.72:0/1205355818 >> 172.21.15.72:0/1205355818 conn(0x56482df13d10 msgr2=0x56482df79eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:31.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.574+0000 7faa0adb1740 1 -- 172.21.15.72:0/1205355818 shutdown_connections 2024-04-26T23:45:31.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.574+0000 7faa0adb1740 1 -- 172.21.15.72:0/1205355818 wait complete. 2024-04-26T23:45:31.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.574+0000 7faa0adb1740 20 client.15099 trim_cache size 0 max 16384 2024-04-26T23:45:31.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.594+0000 7faa0adb1740 1 Processor -- start 2024-04-26T23:45:31.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.594+0000 7faa0adb1740 1 -- start start 2024-04-26T23:45:31.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.595+0000 7faa0adb1740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56482df7f0f0 0x56482df80f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.595+0000 7faa0adb1740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56482df7fae0 0x56482df81480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.595+0000 7faa0adb1740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56482df84c50 0x56482df909e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.595+0000 7faa0adb1740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x56482df95a00 con 0x56482df7f0f0 2024-04-26T23:45:31.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.595+0000 7faa0adb1740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x56482df960d0 con 0x56482df7fae0 2024-04-26T23:45:31.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.595+0000 7faa0adb1740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x56482df967a0 con 0x56482df84c50 2024-04-26T23:45:31.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.595+0000 7faa05600640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56482df7f0f0 0x56482df80f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.595+0000 7faa05600640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56482df7f0f0 0x56482df80f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54278/0 (socket says 172.21.15.72:54278) 2024-04-26T23:45:31.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.595+0000 7faa05600640 1 -- 172.21.15.72:0/996308952 learned_addr learned my addr 172.21.15.72:0/996308952 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:45:31.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.595+0000 7faa06000640 1 --2- 172.21.15.72:0/996308952 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56482df84c50 0x56482df909e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.595+0000 7faa04c00640 1 --2- 172.21.15.72:0/996308952 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56482df7fae0 0x56482df81480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.596+0000 7faa05600640 1 -- 172.21.15.72:0/996308952 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56482df84c50 msgr2=0x56482df909e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:45:31.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.596+0000 7faa05600640 1 --2- 172.21.15.72:0/996308952 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56482df84c50 0x56482df909e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.596+0000 7faa05600640 1 -- 172.21.15.72:0/996308952 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56482df7fae0 msgr2=0x56482df81480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:45:31.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.596+0000 7faa05600640 1 --2- 172.21.15.72:0/996308952 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56482df7fae0 0x56482df81480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.596+0000 7faa05600640 1 -- 172.21.15.72:0/996308952 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56482df841c0 con 0x56482df7f0f0 2024-04-26T23:45:31.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.596+0000 7faa05600640 1 --2- 172.21.15.72:0/996308952 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56482df7f0f0 0x56482df80f40 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa9f409c820 tx=0x7fa9f40046e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbb359cc08ce26b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:31.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.596+0000 7fa9f1c00640 1 -- 172.21.15.72:0/996308952 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f409c8f0 con 0x56482df7f0f0 2024-04-26T23:45:31.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.596+0000 7fa9f1c00640 1 -- 172.21.15.72:0/996308952 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9f409de50 con 0x56482df7f0f0 2024-04-26T23:45:31.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.596+0000 7fa9f1c00640 1 -- 172.21.15.72:0/996308952 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f409c480 con 0x56482df7f0f0 2024-04-26T23:45:31.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.597+0000 7faa0adb1740 1 -- 172.21.15.72:0/996308952 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56482df7f0f0 msgr2=0x56482df80f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:31.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.597+0000 7faa0adb1740 1 --2- 172.21.15.72:0/996308952 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56482df7f0f0 0x56482df80f40 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa9f409c820 tx=0x7fa9f40046e0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.597+0000 7faa0adb1740 1 -- 172.21.15.72:0/996308952 shutdown_connections 2024-04-26T23:45:31.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.597+0000 7faa0adb1740 1 --2- 172.21.15.72:0/996308952 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56482df7fae0 0x56482df81480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.597+0000 7faa0adb1740 1 --2- 172.21.15.72:0/996308952 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56482df84c50 0x56482df909e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.597+0000 7faa0adb1740 1 --2- 172.21.15.72:0/996308952 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56482df7f0f0 0x56482df80f40 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.597+0000 7faa0adb1740 1 -- 172.21.15.72:0/996308952 >> 172.21.15.72:0/996308952 conn(0x56482df13d10 msgr2=0x56482df89540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:31.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.597+0000 7faa0adb1740 1 -- 172.21.15.72:0/996308952 shutdown_connections 2024-04-26T23:45:31.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.597+0000 7faa0adb1740 1 -- 172.21.15.72:0/996308952 wait complete. 2024-04-26T23:45:31.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.597+0000 7faa0adb1740 1 -- start start 2024-04-26T23:45:31.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.598+0000 7faa0adb1740 1 Processor -- start 2024-04-26T23:45:31.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.598+0000 7faa0adb1740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56482df7f0f0 0x56482df8cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.598+0000 7faa0adb1740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56482df7fae0 0x56482df8d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.598+0000 7faa0adb1740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56482df84c50 0x56482df8fb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.598+0000 7faa0adb1740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x56482df97540 con 0x56482df7fae0 2024-04-26T23:45:31.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.598+0000 7faa0adb1740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x56482df96e70 con 0x56482df84c50 2024-04-26T23:45:31.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.598+0000 7faa0adb1740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x56482df97c10 con 0x56482df7f0f0 2024-04-26T23:45:31.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.598+0000 7faa0adb1740 10 client.0 _ll_register_callbacks cb 0x7fa9f800bbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:45:31.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.598+0000 7faa04c00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56482df7fae0 0x56482df8d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.598+0000 7faa06000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56482df84c50 0x56482df8fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.598+0000 7faa04c00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56482df7fae0 0x56482df8d150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54294/0 (socket says 172.21.15.72:54294) 2024-04-26T23:45:31.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.598+0000 7faa04c00640 1 -- 172.21.15.72:0/4136376862 learned_addr learned my addr 172.21.15.72:0/4136376862 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:45:31.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.598+0000 7faa05600640 1 --2- 172.21.15.72:0/4136376862 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56482df7f0f0 0x56482df8cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.599+0000 7faa04c00640 1 -- 172.21.15.72:0/4136376862 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56482df7f0f0 msgr2=0x56482df8cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:45:31.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.599+0000 7faa04c00640 1 --2- 172.21.15.72:0/4136376862 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56482df7f0f0 0x56482df8cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.599+0000 7faa04c00640 1 -- 172.21.15.72:0/4136376862 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56482df84c50 msgr2=0x56482df8fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:45:31.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.599+0000 7faa04c00640 1 --2- 172.21.15.72:0/4136376862 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56482df84c50 0x56482df8fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.599+0000 7faa04c00640 1 -- 172.21.15.72:0/4136376862 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56482df900d0 con 0x56482df7fae0 2024-04-26T23:45:31.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.599+0000 7faa06000640 1 --2- 172.21.15.72:0/4136376862 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56482df84c50 0x56482df8fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:45:31.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.599+0000 7faa05600640 1 --2- 172.21.15.72:0/4136376862 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56482df7f0f0 0x56482df8cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:45:31.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.599+0000 7faa04c00640 1 --2- 172.21.15.72:0/4136376862 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56482df7fae0 0x56482df8d150 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa9ec02a860 tx=0x7fa9ec030e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=250dda02cc90f587 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:31.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.599+0000 7fa9e6a00640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-04-26T23:45:31.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.599+0000 7fa9e6a00640 1 -- 172.21.15.72:0/4136376862 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9ec037040 con 0x56482df7fae0 2024-04-26T23:45:31.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.599+0000 7fa9e6a00640 1 -- 172.21.15.72:0/4136376862 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7fa9ec004040 con 0x56482df7fae0 2024-04-26T23:45:31.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.599+0000 7faa0adb1740 1 -- 172.21.15.72:0/4136376862 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9f409de50 con 0x56482df7fae0 2024-04-26T23:45:31.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.600+0000 7fa9e6a00640 1 -- 172.21.15.72:0/4136376862 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fa9ec052050 con 0x56482df7fae0 2024-04-26T23:45:31.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.600+0000 7faa0adb1740 10 client.15105 fetch_fsmap learned FSMap version 23 2024-04-26T23:45:31.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.600+0000 7faa0adb1740 1 -- 172.21.15.72:0/4136376862 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x56482dffc590 con 0x56482df7fae0 2024-04-26T23:45:31.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.600+0000 7fa9e6a00640 1 -- 172.21.15.72:0/4136376862 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9f409de50 con 0x56482df7fae0 2024-04-26T23:45:31.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.600+0000 7fa9e6a00640 1 -- 172.21.15.72:0/4136376862 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7fa9ec032440 con 0x56482df7fae0 2024-04-26T23:45:31.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.600+0000 7faa0adb1740 10 client.15105 fetch_fsmap finished waiting for FSMap version 23 2024-04-26T23:45:31.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.600+0000 7faa0adb1740 10 client.15105 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-26T23:45:31.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.600+0000 7faa0adb1740 20 client.15105 populate_metadata read hostname 'smithi072' 2024-04-26T23:45:31.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.600+0000 7faa0adb1740 1 --2- 172.21.15.72:0/4136376862 >> [v2:172.21.15.72:6840/1356345077,v1:172.21.15.72:6841/1356345077] conn(0x56482e001170 0x56482dfffe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.600+0000 7faa0adb1740 4 client.15105 mds_command: new command op to 24571 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T23:45:31.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.600+0000 7faa0adb1740 1 -- 172.21.15.72:0/4136376862 --> [v2:172.21.15.72:6840/1356345077,v1:172.21.15.72:6841/1356345077] -- command(tid 0: {"prefix": "status"}) -- 0x56482deeb080 con 0x56482e001170 2024-04-26T23:45:31.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.601+0000 7faa05600640 1 --2- 172.21.15.72:0/4136376862 >> [v2:172.21.15.72:6840/1356345077,v1:172.21.15.72:6841/1356345077] conn(0x56482e001170 0x56482dfffe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.601+0000 7faa05600640 1 --2- 172.21.15.72:0/4136376862 >> [v2:172.21.15.72:6840/1356345077,v1:172.21.15.72:6841/1356345077] conn(0x56482e001170 0x56482dfffe80 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a292935b7fc91ee6 server_cookie=68668daba8256968 in_seq=0 out_seq=0 2024-04-26T23:45:31.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.601+0000 7fa9e6a00640 10 client.15105 ms_handle_connect on v2:172.21.15.72:6840/1356345077 2024-04-26T23:45:31.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.601+0000 7fa9e6a00640 1 -- 172.21.15.72:0/4136376862 <== mds.1 v2:172.21.15.72:6840/1356345077 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x56482deeb080 con 0x56482e001170 2024-04-26T23:45:31.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.601+0000 7fa9e6a00640 10 client.15105 handle_command_reply: tid=0 2024-04-26T23:45:31.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.601+0000 7faa0adb1740 1 client.15105 shutdown 2024-04-26T23:45:31.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.602+0000 7faa0adb1740 1 -- 172.21.15.72:0/4136376862 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56482df7fae0 msgr2=0x56482df8d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:31.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.602+0000 7faa0adb1740 1 --2- 172.21.15.72:0/4136376862 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56482df7fae0 0x56482df8d150 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa9ec02a860 tx=0x7fa9ec030e50 comp rx=0 tx=0).stop 2024-04-26T23:45:31.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.602+0000 7faa0adb1740 1 -- 172.21.15.72:0/4136376862 shutdown_connections 2024-04-26T23:45:31.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.602+0000 7faa0adb1740 1 --2- 172.21.15.72:0/4136376862 >> [v2:172.21.15.72:6840/1356345077,v1:172.21.15.72:6841/1356345077] conn(0x56482e001170 0x56482dfffe80 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.602+0000 7faa0adb1740 1 --2- 172.21.15.72:0/4136376862 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x56482df84c50 0x56482df8fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.602+0000 7faa0adb1740 1 --2- 172.21.15.72:0/4136376862 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56482df7fae0 0x56482df8d150 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.602+0000 7faa0adb1740 1 --2- 172.21.15.72:0/4136376862 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x56482df7f0f0 0x56482df8cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.602+0000 7fa9e6a00640 0 client.15105 ms_handle_reset on v2:172.21.15.72:6840/1356345077 2024-04-26T23:45:31.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.602+0000 7faa0adb1740 1 -- 172.21.15.72:0/4136376862 >> 172.21.15.72:0/4136376862 conn(0x56482df13d10 msgr2=0x56482df89540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:31.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.602+0000 7faa0adb1740 1 -- 172.21.15.72:0/4136376862 shutdown_connections 2024-04-26T23:45:31.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.602+0000 7faa0adb1740 1 -- 172.21.15.72:0/4136376862 wait complete. 2024-04-26T23:45:31.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.602+0000 7faa0adb1740 20 client.15105 trim_cache size 0 max 16384 2024-04-26T23:45:31.610 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-04-26T23:45:31.610 INFO:teuthology.orchestra.run.smithi072.stdout: "cluster_fsid": "3c8453ae-0425-11ef-bc93-c7b262605968", 2024-04-26T23:45:31.610 INFO:teuthology.orchestra.run.smithi072.stdout: "whoami": 1, 2024-04-26T23:45:31.610 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 24571, 2024-04-26T23:45:31.610 INFO:teuthology.orchestra.run.smithi072.stdout: "want_state": "up:active", 2024-04-26T23:45:31.610 INFO:teuthology.orchestra.run.smithi072.stdout: "state": "up:active", 2024-04-26T23:45:31.610 INFO:teuthology.orchestra.run.smithi072.stdout: "fs_name": "cephfs", 2024-04-26T23:45:31.610 INFO:teuthology.orchestra.run.smithi072.stdout: "rank_uptime": 123.431495856, 2024-04-26T23:45:31.610 INFO:teuthology.orchestra.run.smithi072.stdout: "mdsmap_epoch": 23, 2024-04-26T23:45:31.611 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_epoch": 93, 2024-04-26T23:45:31.611 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_epoch_barrier": 89, 2024-04-26T23:45:31.611 INFO:teuthology.orchestra.run.smithi072.stdout: "uptime": 157.453469168 2024-04-26T23:45:31.611 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-04-26T23:45:31.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.602+0000 7faa08000640 1 -- 172.21.15.72:0/999100967 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fa9e80a3c40 msgr2=0x7fa9e80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:31.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.602+0000 7faa08000640 1 --2- 172.21.15.72:0/999100967 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fa9e80a3c40 0x7fa9e80a6100 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x56482df80340 tx=0x7fa9f8011040 comp rx=0 tx=0).stop 2024-04-26T23:45:31.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.603+0000 7faa08000640 1 -- 172.21.15.72:0/999100967 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faa00104bb0 msgr2=0x7faa0019ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:31.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.603+0000 7faa08000640 1 --2- 172.21.15.72:0/999100967 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faa00104bb0 0x7faa0019ade0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa9f400e9c0 tx=0x7fa9f400ee80 comp rx=0 tx=0).stop 2024-04-26T23:45:31.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.603+0000 7faa08000640 1 -- 172.21.15.72:0/999100967 shutdown_connections 2024-04-26T23:45:31.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.603+0000 7faa08000640 1 --2- 172.21.15.72:0/999100967 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fa9e80a3c40 0x7fa9e80a6100 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.603+0000 7faa08000640 1 --2- 172.21.15.72:0/999100967 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa00106760 0x7faa001a2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.603+0000 7faa08000640 1 --2- 172.21.15.72:0/999100967 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa00105da0 0x7faa0019b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.603+0000 7faa08000640 1 --2- 172.21.15.72:0/999100967 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faa00104bb0 0x7faa0019ade0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.603+0000 7faa08000640 1 -- 172.21.15.72:0/999100967 >> 172.21.15.72:0/999100967 conn(0x7faa00100120 msgr2=0x7faa001091c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:31.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.603+0000 7faa08000640 1 -- 172.21.15.72:0/999100967 shutdown_connections 2024-04-26T23:45:31.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.603+0000 7faa08000640 1 -- 172.21.15.72:0/999100967 wait complete. 2024-04-26T23:45:31.613 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:31 smithi072 ceph-mon[28184]: pgmap v399: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:45:31.613 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:31 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/1432176528' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:45:31.625 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-26T23:45:31.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.810+0000 7fa076600640 1 -- 172.21.15.72:0/777500479 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa070104bb0 msgr2=0x7fa070104fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:31.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.810+0000 7fa076600640 1 --2- 172.21.15.72:0/777500479 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa070104bb0 0x7fa070104fd0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa060009900 tx=0x7fa06002a3f0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.811+0000 7fa076600640 1 -- 172.21.15.72:0/777500479 shutdown_connections 2024-04-26T23:45:31.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.811+0000 7fa076600640 1 --2- 172.21.15.72:0/777500479 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa070106760 0x7fa07010cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.811+0000 7fa076600640 1 --2- 172.21.15.72:0/777500479 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa070105da0 0x7fa070106220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.811+0000 7fa076600640 1 --2- 172.21.15.72:0/777500479 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa070104bb0 0x7fa070104fd0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.811+0000 7fa076600640 1 -- 172.21.15.72:0/777500479 >> 172.21.15.72:0/777500479 conn(0x7fa070100120 msgr2=0x7fa070102560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:31.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.811+0000 7fa076600640 1 -- 172.21.15.72:0/777500479 shutdown_connections 2024-04-26T23:45:31.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.811+0000 7fa076600640 1 -- 172.21.15.72:0/777500479 wait complete. 2024-04-26T23:45:31.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.812+0000 7fa076600640 1 Processor -- start 2024-04-26T23:45:31.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.812+0000 7fa076600640 1 -- start start 2024-04-26T23:45:31.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.812+0000 7fa076600640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa070104bb0 0x7fa07019adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.812+0000 7fa076600640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa070105da0 0x7fa07019b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.813+0000 7fa076600640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa070106760 0x7fa0701a2360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.813+0000 7fa076600640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fa070110730 con 0x7fa070104bb0 2024-04-26T23:45:31.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.813+0000 7fa076600640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa0701105b0 con 0x7fa070105da0 2024-04-26T23:45:31.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.813+0000 7fa076600640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fa0701108b0 con 0x7fa070106760 2024-04-26T23:45:31.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.813+0000 7fa06f400640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa070104bb0 0x7fa07019adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.813+0000 7fa06f400640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa070104bb0 0x7fa07019adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54312/0 (socket says 172.21.15.72:54312) 2024-04-26T23:45:31.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.813+0000 7fa06f400640 1 -- 172.21.15.72:0/1400911840 learned_addr learned my addr 172.21.15.72:0/1400911840 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:45:31.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.813+0000 7fa067e00640 1 --2- 172.21.15.72:0/1400911840 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa070105da0 0x7fa07019b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.813+0000 7fa06fe00640 1 --2- 172.21.15.72:0/1400911840 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa070106760 0x7fa0701a2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.813+0000 7fa06f400640 1 -- 172.21.15.72:0/1400911840 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa070106760 msgr2=0x7fa0701a2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:31.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.813+0000 7fa06f400640 1 --2- 172.21.15.72:0/1400911840 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa070106760 0x7fa0701a2360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.813+0000 7fa06f400640 1 -- 172.21.15.72:0/1400911840 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa070105da0 msgr2=0x7fa07019b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:31.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.813+0000 7fa06f400640 1 --2- 172.21.15.72:0/1400911840 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa070105da0 0x7fa07019b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.813+0000 7fa06f400640 1 -- 172.21.15.72:0/1400911840 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0701a29d0 con 0x7fa070104bb0 2024-04-26T23:45:31.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.813+0000 7fa067e00640 1 --2- 172.21.15.72:0/1400911840 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa070105da0 0x7fa07019b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:45:31.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.814+0000 7fa06f400640 1 --2- 172.21.15.72:0/1400911840 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa070104bb0 0x7fa07019adb0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa060031500 tx=0x7fa060003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:31.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.814+0000 7fa06cc00640 1 -- 172.21.15.72:0/1400911840 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa06003f070 con 0x7fa070104bb0 2024-04-26T23:45:31.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.814+0000 7fa06cc00640 1 -- 172.21.15.72:0/1400911840 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa060031c50 con 0x7fa070104bb0 2024-04-26T23:45:31.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.814+0000 7fa076600640 1 -- 172.21.15.72:0/1400911840 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0701a2c60 con 0x7fa070104bb0 2024-04-26T23:45:31.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.814+0000 7fa06cc00640 1 -- 172.21.15.72:0/1400911840 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa060035030 con 0x7fa070104bb0 2024-04-26T23:45:31.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.814+0000 7fa076600640 1 -- 172.21.15.72:0/1400911840 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0701a31a0 con 0x7fa070104bb0 2024-04-26T23:45:31.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.817+0000 7fa0795d1740 1 Processor -- start 2024-04-26T23:45:31.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.817+0000 7fa06cc00640 1 -- 172.21.15.72:0/1400911840 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fa060042070 con 0x7fa070104bb0 2024-04-26T23:45:31.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.817+0000 7fa0795d1740 1 -- start start 2024-04-26T23:45:31.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.818+0000 7fa0795d1740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564c77f5e0f0 0x564c77f5e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.818+0000 7fa0795d1740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564c77f5eae0 0x564c77f635e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.818+0000 7fa0795d1740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564c77f63c50 0x564c77f660e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.818+0000 7fa0795d1740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x564c77ec2600 con 0x564c77f63c50 2024-04-26T23:45:31.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.818+0000 7fa0795d1740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x564c77e5f100 con 0x564c77f5e0f0 2024-04-26T23:45:31.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.818+0000 7fa0795d1740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x564c77ebe5c0 con 0x564c77f5eae0 2024-04-26T23:45:31.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.818+0000 7fa067e00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564c77f63c50 0x564c77f660e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.818+0000 7fa06fe00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564c77f5e0f0 0x564c77f5e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.818+0000 7fa06fe00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564c77f5e0f0 0x564c77f5e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:52906/0 (socket says 172.21.15.72:52906) 2024-04-26T23:45:31.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.818+0000 7fa06fe00640 1 -- 172.21.15.72:0/1027586271 learned_addr learned my addr 172.21.15.72:0/1027586271 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:45:31.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.818+0000 7fa06f400640 1 --2- 172.21.15.72:0/1027586271 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564c77f5eae0 0x564c77f635e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.818+0000 7fa06fe00640 1 -- 172.21.15.72:0/1027586271 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564c77f5eae0 msgr2=0x564c77f635e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:45:31.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.818+0000 7fa06fe00640 1 --2- 172.21.15.72:0/1027586271 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564c77f5eae0 0x564c77f635e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.819+0000 7fa06fe00640 1 -- 172.21.15.72:0/1027586271 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564c77f63c50 msgr2=0x564c77f660e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:45:31.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.819+0000 7fa06fe00640 1 --2- 172.21.15.72:0/1027586271 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564c77f63c50 0x564c77f660e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.819+0000 7fa06fe00640 1 -- 172.21.15.72:0/1027586271 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c77bd42d0 con 0x564c77f5e0f0 2024-04-26T23:45:31.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.819+0000 7fa06fe00640 1 --2- 172.21.15.72:0/1027586271 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564c77f5e0f0 0x564c77f5e510 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa068009fd0 tx=0x7fa0680118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70ca568fa4ca2feb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:31.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.819+0000 7fa06cc00640 1 --2- 172.21.15.72:0/1400911840 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fa0540a3cc0 0x7fa0540a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.819+0000 7fa06cc00640 1 -- 172.21.15.72:0/1400911840 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7fa0600f0fe0 con 0x7fa070104bb0 2024-04-26T23:45:31.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.819+0000 7fa064c00640 1 -- 172.21.15.72:0/1027586271 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa068004d20 con 0x564c77f5e0f0 2024-04-26T23:45:31.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.819+0000 7fa06f400640 1 --2- 172.21.15.72:0/1400911840 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fa0540a3cc0 0x7fa0540a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.819+0000 7fa064c00640 1 -- 172.21.15.72:0/1027586271 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa068004050 con 0x564c77f5e0f0 2024-04-26T23:45:31.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.819+0000 7fa064c00640 1 -- 172.21.15.72:0/1027586271 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa068005280 con 0x564c77f5e0f0 2024-04-26T23:45:31.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.820+0000 7fa0795d1740 1 -- 172.21.15.72:0/1027586271 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564c77f5e0f0 msgr2=0x564c77f5e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:31.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.820+0000 7fa06f400640 1 --2- 172.21.15.72:0/1400911840 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fa0540a3cc0 0x7fa0540a6180 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x564c77f5f550 tx=0x7fa0600f4b30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:31.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.820+0000 7fa0795d1740 1 --2- 172.21.15.72:0/1027586271 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564c77f5e0f0 0x564c77f5e510 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa068009fd0 tx=0x7fa0680118d0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.820+0000 7fa0795d1740 1 -- 172.21.15.72:0/1027586271 shutdown_connections 2024-04-26T23:45:31.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.820+0000 7fa0795d1740 1 --2- 172.21.15.72:0/1027586271 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564c77f63c50 0x564c77f660e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.820+0000 7fa0795d1740 1 --2- 172.21.15.72:0/1027586271 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564c77f5eae0 0x564c77f635e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.820+0000 7fa0795d1740 1 --2- 172.21.15.72:0/1027586271 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564c77f5e0f0 0x564c77f5e510 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.820+0000 7fa0795d1740 1 -- 172.21.15.72:0/1027586271 >> 172.21.15.72:0/1027586271 conn(0x564c77ef2d10 msgr2=0x564c77f59b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:31.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.820+0000 7fa0795d1740 1 -- 172.21.15.72:0/1027586271 shutdown_connections 2024-04-26T23:45:31.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.820+0000 7fa0795d1740 1 -- 172.21.15.72:0/1027586271 wait complete. 2024-04-26T23:45:31.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.821+0000 7fa0795d1740 1 -- start start 2024-04-26T23:45:31.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.821+0000 7fa0795d1740 1 Processor -- start 2024-04-26T23:45:31.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.821+0000 7fa0795d1740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564c77f5e0f0 0x564c77f68580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.821+0000 7fa0795d1740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564c77f5eae0 0x564c77f68ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.821+0000 7fa0795d1740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564c77f63c50 0x564c77f6fb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.821+0000 7fa0795d1740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x564c77f66c00 con 0x564c77f5e0f0 2024-04-26T23:45:31.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.821+0000 7fa0795d1740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x564c77e49f70 con 0x564c77f5eae0 2024-04-26T23:45:31.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.821+0000 7fa0795d1740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x564c77f66d80 con 0x564c77f63c50 2024-04-26T23:45:31.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.822+0000 7fa0795d1740 10 client.0 _ll_register_callbacks cb 0x564c77eb8480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:45:31.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.822+0000 7fa06fe00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564c77f5e0f0 0x564c77f68580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.822+0000 7fa067e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564c77f5eae0 0x564c77f68ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.822+0000 7fa06fe00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564c77f5e0f0 0x564c77f68580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54328/0 (socket says 172.21.15.72:54328) 2024-04-26T23:45:31.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.822+0000 7fa06fe00640 1 -- 172.21.15.72:0/1870323155 learned_addr learned my addr 172.21.15.72:0/1870323155 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:45:31.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.822+0000 7fa06fe00640 1 --2- 172.21.15.72:0/1870323155 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564c77f63c50 0x564c77f6fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.822+0000 7fa067e00640 1 -- 172.21.15.72:0/1870323155 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564c77f63c50 msgr2=0x564c77f6fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:45:31.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.822+0000 7fa067e00640 1 --2- 172.21.15.72:0/1870323155 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564c77f63c50 0x564c77f6fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.822+0000 7fa067e00640 1 -- 172.21.15.72:0/1870323155 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564c77f5e0f0 msgr2=0x564c77f68580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:45:31.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.822+0000 7fa067e00640 1 --2- 172.21.15.72:0/1870323155 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564c77f5e0f0 0x564c77f68580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.822+0000 7fa067e00640 1 -- 172.21.15.72:0/1870323155 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c77c9e6b0 con 0x564c77f5eae0 2024-04-26T23:45:31.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.822+0000 7fa06fe00640 1 --2- 172.21.15.72:0/1870323155 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564c77f5e0f0 0x564c77f68580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:45:31.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.823+0000 7fa067e00640 1 --2- 172.21.15.72:0/1870323155 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564c77f5eae0 0x564c77f68ac0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa07019c2c0 tx=0x7fa05800f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74de20ac72ef777c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:31.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.823+0000 7fa05d600640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-04-26T23:45:31.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.823+0000 7fa05d600640 1 -- 172.21.15.72:0/1870323155 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa058012110 con 0x564c77f5eae0 2024-04-26T23:45:31.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.823+0000 7fa05d600640 1 -- 172.21.15.72:0/1870323155 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7fa058004050 con 0x564c77f5eae0 2024-04-26T23:45:31.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.823+0000 7fa0795d1740 1 -- 172.21.15.72:0/1870323155 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa068004050 con 0x564c77f5eae0 2024-04-26T23:45:31.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.823+0000 7fa05d600640 1 -- 172.21.15.72:0/1870323155 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fa058025050 con 0x564c77f5eae0 2024-04-26T23:45:31.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.823+0000 7fa0795d1740 10 client.25220 fetch_fsmap learned FSMap version 23 2024-04-26T23:45:31.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.823+0000 7fa0795d1740 1 -- 172.21.15.72:0/1870323155 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x564c77f72b50 con 0x564c77f5eae0 2024-04-26T23:45:31.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.823+0000 7fa05d600640 1 -- 172.21.15.72:0/1870323155 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa068004050 con 0x564c77f5eae0 2024-04-26T23:45:31.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.824+0000 7fa05d600640 1 -- 172.21.15.72:0/1870323155 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7fa058010430 con 0x564c77f5eae0 2024-04-26T23:45:31.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.824+0000 7fa0795d1740 10 client.25220 fetch_fsmap finished waiting for FSMap version 23 2024-04-26T23:45:31.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.824+0000 7fa0795d1740 10 client.25220 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T23:45:31.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.824+0000 7fa0795d1740 20 client.25220 populate_metadata read hostname 'smithi072' 2024-04-26T23:45:31.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.824+0000 7fa0795d1740 1 --2- 172.21.15.72:0/1870323155 >> [v2:172.21.15.76:6838/3405829058,v1:172.21.15.76:6839/3405829058] conn(0x564c77f75200 0x564c77f76670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.824+0000 7fa0795d1740 4 client.25220 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:45:31.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.824+0000 7fa0795d1740 1 -- 172.21.15.72:0/1870323155 --> [v2:172.21.15.76:6838/3405829058,v1:172.21.15.76:6839/3405829058] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564c77ed0210 con 0x564c77f75200 2024-04-26T23:45:31.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.824+0000 7fa06fe00640 1 --2- 172.21.15.72:0/1870323155 >> [v2:172.21.15.76:6838/3405829058,v1:172.21.15.76:6839/3405829058] conn(0x564c77f75200 0x564c77f76670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.825+0000 7fa06fe00640 1 --2- 172.21.15.72:0/1870323155 >> [v2:172.21.15.76:6838/3405829058,v1:172.21.15.76:6839/3405829058] conn(0x564c77f75200 0x564c77f76670 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ea43448223c4f8 server_cookie=41e283b4f4e0b9da in_seq=0 out_seq=0 2024-04-26T23:45:31.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.825+0000 7fa05d600640 10 client.25220 ms_handle_connect on v2:172.21.15.76:6838/3405829058 2024-04-26T23:45:31.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.826+0000 7fa05d600640 1 -- 172.21.15.72:0/1870323155 <== mds.0 v2:172.21.15.76:6838/3405829058 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564c77ed0210 con 0x564c77f75200 2024-04-26T23:45:31.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.826+0000 7fa05d600640 10 client.25220 handle_command_reply: tid=0 2024-04-26T23:45:31.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.826+0000 7fa0795d1740 1 client.25220 shutdown 2024-04-26T23:45:31.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.826+0000 7fa0795d1740 1 -- 172.21.15.72:0/1870323155 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564c77f5eae0 msgr2=0x564c77f68ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:31.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.826+0000 7fa0795d1740 1 --2- 172.21.15.72:0/1870323155 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564c77f5eae0 0x564c77f68ac0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa07019c2c0 tx=0x7fa05800f790 comp rx=0 tx=0).stop 2024-04-26T23:45:31.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.826+0000 7fa0795d1740 1 -- 172.21.15.72:0/1870323155 shutdown_connections 2024-04-26T23:45:31.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.826+0000 7fa0795d1740 1 --2- 172.21.15.72:0/1870323155 >> [v2:172.21.15.76:6838/3405829058,v1:172.21.15.76:6839/3405829058] conn(0x564c77f75200 0x564c77f76670 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.826+0000 7fa0795d1740 1 --2- 172.21.15.72:0/1870323155 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564c77f5eae0 0x564c77f68ac0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.826+0000 7fa0795d1740 1 --2- 172.21.15.72:0/1870323155 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564c77f63c50 0x564c77f6fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.826+0000 7fa0795d1740 1 --2- 172.21.15.72:0/1870323155 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564c77f5e0f0 0x564c77f68580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.826+0000 7fa05d600640 0 client.25220 ms_handle_reset on v2:172.21.15.76:6838/3405829058 2024-04-26T23:45:31.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.826+0000 7fa0795d1740 1 -- 172.21.15.72:0/1870323155 >> 172.21.15.72:0/1870323155 conn(0x564c77ef2d10 msgr2=0x564c77f58eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:31.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.826+0000 7fa0795d1740 1 -- 172.21.15.72:0/1870323155 shutdown_connections 2024-04-26T23:45:31.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.826+0000 7fa0795d1740 1 -- 172.21.15.72:0/1870323155 wait complete. 2024-04-26T23:45:31.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.826+0000 7fa0795d1740 20 client.25220 trim_cache size 0 max 16384 2024-04-26T23:45:31.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.846+0000 7fa0795d1740 1 Processor -- start 2024-04-26T23:45:31.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.846+0000 7fa0795d1740 1 -- start start 2024-04-26T23:45:31.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.846+0000 7fa0795d1740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564c77f5e0f0 0x564c77f5ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.846+0000 7fa0795d1740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564c77f5eae0 0x564c77f6fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.846+0000 7fa0795d1740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564c77f63c50 0x564c77f60480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.846+0000 7fa0795d1740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x564c77f749b0 con 0x564c77f63c50 2024-04-26T23:45:31.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.846+0000 7fa0795d1740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x564c77f75080 con 0x564c77f5e0f0 2024-04-26T23:45:31.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.846+0000 7fa0795d1740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x564c77f75750 con 0x564c77f5eae0 2024-04-26T23:45:31.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.846+0000 7fa06fe00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564c77f63c50 0x564c77f60480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.847+0000 7fa067e00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564c77f5eae0 0x564c77f6fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.847+0000 7fa06fe00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564c77f63c50 0x564c77f60480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54340/0 (socket says 172.21.15.72:54340) 2024-04-26T23:45:31.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.847+0000 7fa06fe00640 1 -- 172.21.15.72:0/734779090 learned_addr learned my addr 172.21.15.72:0/734779090 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:45:31.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.847+0000 7fa06fe00640 1 --2- 172.21.15.72:0/734779090 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564c77f5e0f0 0x564c77f5ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.847+0000 7fa06fe00640 1 -- 172.21.15.72:0/734779090 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564c77f5eae0 msgr2=0x564c77f6fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:45:31.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.847+0000 7fa06fe00640 1 --2- 172.21.15.72:0/734779090 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564c77f5eae0 0x564c77f6fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.847+0000 7fa06fe00640 1 -- 172.21.15.72:0/734779090 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564c77f5e0f0 msgr2=0x564c77f5ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:45:31.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.847+0000 7fa06fe00640 1 --2- 172.21.15.72:0/734779090 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564c77f5e0f0 0x564c77f5ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.847+0000 7fa06fe00640 1 -- 172.21.15.72:0/734779090 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c77f63230 con 0x564c77f63c50 2024-04-26T23:45:31.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.847+0000 7fa06fe00640 1 --2- 172.21.15.72:0/734779090 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564c77f63c50 0x564c77f60480 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa068011dd0 tx=0x7fa068003f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=297747a58625142c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:31.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.847+0000 7fa05fe00640 1 -- 172.21.15.72:0/734779090 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0680045a0 con 0x564c77f63c50 2024-04-26T23:45:31.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.848+0000 7fa05fe00640 1 -- 172.21.15.72:0/734779090 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa068004d30 con 0x564c77f63c50 2024-04-26T23:45:31.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.848+0000 7fa05fe00640 1 -- 172.21.15.72:0/734779090 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa06802c5c0 con 0x564c77f63c50 2024-04-26T23:45:31.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.848+0000 7fa0795d1740 1 -- 172.21.15.72:0/734779090 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564c77f63c50 msgr2=0x564c77f60480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:31.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.848+0000 7fa0795d1740 1 --2- 172.21.15.72:0/734779090 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564c77f63c50 0x564c77f60480 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa068011dd0 tx=0x7fa068003f50 comp rx=0 tx=0).stop 2024-04-26T23:45:31.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.848+0000 7fa0795d1740 1 -- 172.21.15.72:0/734779090 shutdown_connections 2024-04-26T23:45:31.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.848+0000 7fa0795d1740 1 --2- 172.21.15.72:0/734779090 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564c77f63c50 0x564c77f60480 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.848+0000 7fa0795d1740 1 --2- 172.21.15.72:0/734779090 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564c77f5eae0 0x564c77f6fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.848+0000 7fa0795d1740 1 --2- 172.21.15.72:0/734779090 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564c77f5e0f0 0x564c77f5ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.848+0000 7fa0795d1740 1 -- 172.21.15.72:0/734779090 >> 172.21.15.72:0/734779090 conn(0x564c77ef2d10 msgr2=0x564c77f595f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:31.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.848+0000 7fa0795d1740 1 -- 172.21.15.72:0/734779090 shutdown_connections 2024-04-26T23:45:31.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.848+0000 7fa0795d1740 1 -- 172.21.15.72:0/734779090 wait complete. 2024-04-26T23:45:31.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.849+0000 7fa0795d1740 1 -- start start 2024-04-26T23:45:31.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.849+0000 7fa0795d1740 1 Processor -- start 2024-04-26T23:45:31.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.849+0000 7fa0795d1740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564c77f5e0f0 0x564c77f6b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.849+0000 7fa0795d1740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564c77f5eae0 0x564c77f6bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.849+0000 7fa0795d1740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564c77f63c50 0x564c77f6ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.849+0000 7fa0795d1740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x564c77f764f0 con 0x564c77f63c50 2024-04-26T23:45:31.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.849+0000 7fa0795d1740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x564c77f75e20 con 0x564c77f5e0f0 2024-04-26T23:45:31.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.849+0000 7fa0795d1740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x564c77f76bc0 con 0x564c77f5eae0 2024-04-26T23:45:31.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.849+0000 7fa0795d1740 10 client.0 _ll_register_callbacks cb 0x7fa058009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:45:31.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.850+0000 7fa06fe00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564c77f63c50 0x564c77f6ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.850+0000 7fa06fe00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564c77f63c50 0x564c77f6ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54352/0 (socket says 172.21.15.72:54352) 2024-04-26T23:45:31.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.850+0000 7fa06fe00640 1 -- 172.21.15.72:0/469770924 learned_addr learned my addr 172.21.15.72:0/469770924 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:45:31.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.850+0000 7fa06fe00640 1 --2- 172.21.15.72:0/469770924 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564c77f5e0f0 0x564c77f6b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.850+0000 7fa067e00640 1 --2- 172.21.15.72:0/469770924 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564c77f5eae0 0x564c77f6bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.850+0000 7fa06fe00640 1 -- 172.21.15.72:0/469770924 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564c77f5eae0 msgr2=0x564c77f6bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:45:31.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.850+0000 7fa06fe00640 1 --2- 172.21.15.72:0/469770924 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564c77f5eae0 0x564c77f6bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.850+0000 7fa06fe00640 1 -- 172.21.15.72:0/469770924 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564c77f5e0f0 msgr2=0x564c77f6b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:45:31.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.850+0000 7fa06fe00640 1 --2- 172.21.15.72:0/469770924 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564c77f5e0f0 0x564c77f6b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.850+0000 7fa06fe00640 1 -- 172.21.15.72:0/469770924 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c77f6f320 con 0x564c77f63c50 2024-04-26T23:45:31.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.850+0000 7fa06fe00640 1 --2- 172.21.15.72:0/469770924 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564c77f63c50 0x564c77f6ed30 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa068005d70 tx=0x7fa0680096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8136238c0dd97c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:31.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.850+0000 7fa05d600640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-04-26T23:45:31.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.851+0000 7fa05d600640 1 -- 172.21.15.72:0/469770924 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa06802c5c0 con 0x564c77f63c50 2024-04-26T23:45:31.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.851+0000 7fa05d600640 1 -- 172.21.15.72:0/469770924 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7fa0680045a0 con 0x564c77f63c50 2024-04-26T23:45:31.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.851+0000 7fa0795d1740 1 -- 172.21.15.72:0/469770924 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa068004d30 con 0x564c77f63c50 2024-04-26T23:45:31.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.851+0000 7fa05d600640 1 -- 172.21.15.72:0/469770924 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fa0680047a0 con 0x564c77f63c50 2024-04-26T23:45:31.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.851+0000 7fa0795d1740 10 client.15123 fetch_fsmap learned FSMap version 23 2024-04-26T23:45:31.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.851+0000 7fa0795d1740 1 -- 172.21.15.72:0/469770924 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x564c77fe3f90 con 0x564c77f63c50 2024-04-26T23:45:31.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.851+0000 7fa05d600640 1 -- 172.21.15.72:0/469770924 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa068004d30 con 0x564c77f63c50 2024-04-26T23:45:31.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.851+0000 7fa05d600640 1 -- 172.21.15.72:0/469770924 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7fa068038040 con 0x564c77f63c50 2024-04-26T23:45:31.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.851+0000 7fa0795d1740 10 client.15123 fetch_fsmap finished waiting for FSMap version 23 2024-04-26T23:45:31.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.851+0000 7fa0795d1740 10 client.15123 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T23:45:31.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.851+0000 7fa0795d1740 20 client.15123 populate_metadata read hostname 'smithi072' 2024-04-26T23:45:31.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.851+0000 7fa0795d1740 1 --2- 172.21.15.72:0/469770924 >> [v2:172.21.15.76:6838/3405829058,v1:172.21.15.76:6839/3405829058] conn(0x564c77fe8380 0x564c77fe7090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:31.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.851+0000 7fa0795d1740 4 client.15123 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T23:45:31.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.851+0000 7fa0795d1740 1 -- 172.21.15.72:0/469770924 --> [v2:172.21.15.76:6838/3405829058,v1:172.21.15.76:6839/3405829058] -- command(tid 0: {"prefix": "status"}) -- 0x564c77eca080 con 0x564c77fe8380 2024-04-26T23:45:31.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.852+0000 7fa067e00640 1 --2- 172.21.15.72:0/469770924 >> [v2:172.21.15.76:6838/3405829058,v1:172.21.15.76:6839/3405829058] conn(0x564c77fe8380 0x564c77fe7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:31.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.852+0000 7fa067e00640 1 --2- 172.21.15.72:0/469770924 >> [v2:172.21.15.76:6838/3405829058,v1:172.21.15.76:6839/3405829058] conn(0x564c77fe8380 0x564c77fe7090 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7da2f59834c9d0b3 server_cookie=92be31473e34373c in_seq=0 out_seq=0 2024-04-26T23:45:31.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.852+0000 7fa05d600640 10 client.15123 ms_handle_connect on v2:172.21.15.76:6838/3405829058 2024-04-26T23:45:31.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.853+0000 7fa05d600640 1 -- 172.21.15.72:0/469770924 <== mds.0 v2:172.21.15.76:6838/3405829058 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x564c77eca080 con 0x564c77fe8380 2024-04-26T23:45:31.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.853+0000 7fa05d600640 10 client.15123 handle_command_reply: tid=0 2024-04-26T23:45:31.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.853+0000 7fa0795d1740 1 client.15123 shutdown 2024-04-26T23:45:31.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.853+0000 7fa0795d1740 1 -- 172.21.15.72:0/469770924 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564c77f63c50 msgr2=0x564c77f6ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:31.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.853+0000 7fa0795d1740 1 --2- 172.21.15.72:0/469770924 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564c77f63c50 0x564c77f6ed30 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa068005d70 tx=0x7fa0680096c0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.853+0000 7fa0795d1740 1 -- 172.21.15.72:0/469770924 shutdown_connections 2024-04-26T23:45:31.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.853+0000 7fa0795d1740 1 --2- 172.21.15.72:0/469770924 >> [v2:172.21.15.76:6838/3405829058,v1:172.21.15.76:6839/3405829058] conn(0x564c77fe8380 0x564c77fe7090 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.853+0000 7fa0795d1740 1 --2- 172.21.15.72:0/469770924 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x564c77f63c50 0x564c77f6ed30 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.853+0000 7fa0795d1740 1 --2- 172.21.15.72:0/469770924 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x564c77f5eae0 0x564c77f6bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.853+0000 7fa05d600640 0 client.15123 ms_handle_reset on v2:172.21.15.76:6838/3405829058 2024-04-26T23:45:31.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.853+0000 7fa0795d1740 1 --2- 172.21.15.72:0/469770924 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x564c77f5e0f0 0x564c77f6b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.853+0000 7fa0795d1740 1 -- 172.21.15.72:0/469770924 >> 172.21.15.72:0/469770924 conn(0x564c77ef2d10 msgr2=0x564c77f595f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:31.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.853+0000 7fa0795d1740 1 -- 172.21.15.72:0/469770924 shutdown_connections 2024-04-26T23:45:31.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.853+0000 7fa0795d1740 1 -- 172.21.15.72:0/469770924 wait complete. 2024-04-26T23:45:31.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.853+0000 7fa0795d1740 20 client.15123 trim_cache size 0 max 16384 2024-04-26T23:45:31.865 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-04-26T23:45:31.865 INFO:teuthology.orchestra.run.smithi072.stdout: "cluster_fsid": "3c8453ae-0425-11ef-bc93-c7b262605968", 2024-04-26T23:45:31.866 INFO:teuthology.orchestra.run.smithi072.stdout: "whoami": 0, 2024-04-26T23:45:31.866 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 24590, 2024-04-26T23:45:31.866 INFO:teuthology.orchestra.run.smithi072.stdout: "want_state": "up:active", 2024-04-26T23:45:31.866 INFO:teuthology.orchestra.run.smithi072.stdout: "state": "up:active", 2024-04-26T23:45:31.866 INFO:teuthology.orchestra.run.smithi072.stdout: "fs_name": "cephfs", 2024-04-26T23:45:31.866 INFO:teuthology.orchestra.run.smithi072.stdout: "rank_uptime": 148.39224010999999, 2024-04-26T23:45:31.866 INFO:teuthology.orchestra.run.smithi072.stdout: "mdsmap_epoch": 23, 2024-04-26T23:45:31.866 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_epoch": 93, 2024-04-26T23:45:31.866 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_epoch_barrier": 89, 2024-04-26T23:45:31.866 INFO:teuthology.orchestra.run.smithi072.stdout: "uptime": 154.89163654399999 2024-04-26T23:45:31.866 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-04-26T23:45:31.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.854+0000 7fa076600640 1 -- 172.21.15.72:0/1400911840 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fa0540a3cc0 msgr2=0x7fa0540a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:31.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.854+0000 7fa076600640 1 --2- 172.21.15.72:0/1400911840 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fa0540a3cc0 0x7fa0540a6180 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x564c77f5f550 tx=0x7fa0600f4b30 comp rx=0 tx=0).stop 2024-04-26T23:45:31.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.854+0000 7fa076600640 1 -- 172.21.15.72:0/1400911840 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa070104bb0 msgr2=0x7fa07019adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:31.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.854+0000 7fa076600640 1 --2- 172.21.15.72:0/1400911840 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa070104bb0 0x7fa07019adb0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa060031500 tx=0x7fa060003c10 comp rx=0 tx=0).stop 2024-04-26T23:45:31.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.854+0000 7fa076600640 1 -- 172.21.15.72:0/1400911840 shutdown_connections 2024-04-26T23:45:31.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.854+0000 7fa076600640 1 --2- 172.21.15.72:0/1400911840 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fa0540a3cc0 0x7fa0540a6180 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.854+0000 7fa076600640 1 --2- 172.21.15.72:0/1400911840 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa070105da0 0x7fa07019b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.854+0000 7fa076600640 1 --2- 172.21.15.72:0/1400911840 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa070106760 0x7fa0701a2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.854+0000 7fa076600640 1 --2- 172.21.15.72:0/1400911840 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa070104bb0 0x7fa07019adb0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:31.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.854+0000 7fa076600640 1 -- 172.21.15.72:0/1400911840 >> 172.21.15.72:0/1400911840 conn(0x7fa070100120 msgr2=0x7fa0701090a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:31.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.854+0000 7fa076600640 1 -- 172.21.15.72:0/1400911840 shutdown_connections 2024-04-26T23:45:31.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:31.854+0000 7fa076600640 1 -- 172.21.15.72:0/1400911840 wait complete. 2024-04-26T23:45:31.876 INFO:tasks.cephadm:Setup complete, yielding 2024-04-26T23:45:31.876 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-26T23:45:31.886 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi072.front.sepia.ceph.com 2024-04-26T23:45:31.886 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-26T23:45:31.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:31 smithi076 ceph-mon[32774]: pgmap v399: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:45:31.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:31 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/1432176528' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:45:32.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:31 smithi136 ceph-mon[33050]: pgmap v399: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:45:32.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:31 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/1432176528' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:45:32.343 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:45:33.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:33 smithi072 ceph-mon[28184]: pgmap v400: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:45:33.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:33 smithi076 ceph-mon[32774]: pgmap v400: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:45:34.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:33 smithi136 ceph-mon[33050]: pgmap v400: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:45:34.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.462+0000 7f1adce00700 1 -- 172.21.15.72:0/3098061672 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1ad81067d0 msgr2=0x7f1ad8108bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:34.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.462+0000 7f1adce00700 1 --2- 172.21.15.72:0/3098061672 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1ad81067d0 0x7f1ad8108bc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0009990 tx=0x7f1ac0009ca0 comp rx=0 tx=0).stop 2024-04-26T23:45:34.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.462+0000 7f1adce00700 1 -- 172.21.15.72:0/3098061672 shutdown_connections 2024-04-26T23:45:34.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.462+0000 7f1adce00700 1 --2- 172.21.15.72:0/3098061672 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ad8109100 0x7f1ad810b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:34.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.462+0000 7f1adce00700 1 --2- 172.21.15.72:0/3098061672 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1ad8103ea0 0x7f1ad8106290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:34.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.462+0000 7f1adce00700 1 --2- 172.21.15.72:0/3098061672 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1ad81067d0 0x7f1ad8108bc0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:34.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.462+0000 7f1adce00700 1 -- 172.21.15.72:0/3098061672 >> 172.21.15.72:0/3098061672 conn(0x7f1ad80fd830 msgr2=0x7f1ad80ffc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:34.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.462+0000 7f1adce00700 1 -- 172.21.15.72:0/3098061672 shutdown_connections 2024-04-26T23:45:34.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.463+0000 7f1adce00700 1 -- 172.21.15.72:0/3098061672 wait complete. 2024-04-26T23:45:34.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.464+0000 7f1adce00700 1 Processor -- start 2024-04-26T23:45:34.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.464+0000 7f1adce00700 1 -- start start 2024-04-26T23:45:34.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.464+0000 7f1adce00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1ad8103ea0 0x7f1ad819b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:34.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.464+0000 7f1adce00700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1ad81067d0 0x7f1ad819bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:34.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.464+0000 7f1adce00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ad8109100 0x7f1ad81a2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:34.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.464+0000 7f1adce00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad810ee80 con 0x7f1ad8109100 2024-04-26T23:45:34.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.464+0000 7f1adce00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad810ed00 con 0x7f1ad8103ea0 2024-04-26T23:45:34.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.464+0000 7f1adce00700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad810f000 con 0x7f1ad81067d0 2024-04-26T23:45:34.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.465+0000 7f1ad7400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1ad8103ea0 0x7f1ad819b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:34.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.465+0000 7f1ad7400700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1ad8103ea0 0x7f1ad819b6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:52958/0 (socket says 172.21.15.72:52958) 2024-04-26T23:45:34.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.465+0000 7f1ad7e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ad8109100 0x7f1ad81a2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:34.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.465+0000 7f1ad7400700 1 -- 172.21.15.72:0/586625621 learned_addr learned my addr 172.21.15.72:0/586625621 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:45:34.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.465+0000 7f1acea00700 1 --2- 172.21.15.72:0/586625621 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1ad81067d0 0x7f1ad819bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:34.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.465+0000 7f1ad7400700 1 -- 172.21.15.72:0/586625621 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1ad81067d0 msgr2=0x7f1ad819bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:34.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.465+0000 7f1ad7400700 1 --2- 172.21.15.72:0/586625621 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1ad81067d0 0x7f1ad819bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:34.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.465+0000 7f1ad7400700 1 -- 172.21.15.72:0/586625621 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ad8109100 msgr2=0x7f1ad81a2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:34.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.465+0000 7f1ad7400700 1 --2- 172.21.15.72:0/586625621 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ad8109100 0x7f1ad81a2ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:34.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.465+0000 7f1ad7400700 1 -- 172.21.15.72:0/586625621 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ad81a3340 con 0x7f1ad8103ea0 2024-04-26T23:45:34.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.465+0000 7f1ad7400700 1 --2- 172.21.15.72:0/586625621 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1ad8103ea0 0x7f1ad819b6e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1ac800c370 tx=0x7f1ac800c680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:34.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.465+0000 7f1ad4c00700 1 -- 172.21.15.72:0/586625621 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac800cea0 con 0x7f1ad8103ea0 2024-04-26T23:45:34.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.466+0000 7f1ad4c00700 1 -- 172.21.15.72:0/586625621 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ac80090d0 con 0x7f1ad8103ea0 2024-04-26T23:45:34.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.466+0000 7f1ad4c00700 1 -- 172.21.15.72:0/586625621 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac8004770 con 0x7f1ad8103ea0 2024-04-26T23:45:34.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.466+0000 7f1adce00700 1 -- 172.21.15.72:0/586625621 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ad81a35d0 con 0x7f1ad8103ea0 2024-04-26T23:45:34.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.466+0000 7f1adce00700 1 -- 172.21.15.72:0/586625621 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ad81a3ac0 con 0x7f1ad8103ea0 2024-04-26T23:45:34.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.467+0000 7f1adce00700 1 -- 172.21.15.72:0/586625621 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ad806a110 con 0x7f1ad8103ea0 2024-04-26T23:45:34.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.469+0000 7f1ad4c00700 1 -- 172.21.15.72:0/586625621 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1ac801a020 con 0x7f1ad8103ea0 2024-04-26T23:45:34.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.470+0000 7f1ad4c00700 1 --2- 172.21.15.72:0/586625621 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f1abc0a3d50 0x7f1abc0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:34.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.471+0000 7f1ad4c00700 1 -- 172.21.15.72:0/586625621 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f1ac80d0350 con 0x7f1ad8103ea0 2024-04-26T23:45:34.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.471+0000 7f1acea00700 1 --2- 172.21.15.72:0/586625621 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f1abc0a3d50 0x7f1abc0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:34.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.471+0000 7f1acea00700 1 --2- 172.21.15.72:0/586625621 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f1abc0a3d50 0x7f1abc0a6210 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1ac000b2e0 tx=0x7f1ac001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:34.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.475+0000 7f1ad4c00700 1 -- 172.21.15.72:0/586625621 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1ac8010040 con 0x7f1ad8103ea0 2024-04-26T23:45:34.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.724+0000 7f1adce00700 1 -- 172.21.15.72:0/586625621 --> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f1ad8063860 con 0x7f1abc0a3d50 2024-04-26T23:45:34.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.726+0000 7f1ad4c00700 1 -- 172.21.15.72:0/586625621 <== mgr.14152 v2:172.21.15.72:6800/4281177709 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f1ad8063860 con 0x7f1abc0a3d50 2024-04-26T23:45:34.730 INFO:teuthology.orchestra.run.smithi072.stdout:Backend: cephadm 2024-04-26T23:45:34.730 INFO:teuthology.orchestra.run.smithi072.stdout:Available: Yes 2024-04-26T23:45:34.730 INFO:teuthology.orchestra.run.smithi072.stdout:Paused: No 2024-04-26T23:45:34.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.729+0000 7f1adce00700 1 -- 172.21.15.72:0/586625621 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f1abc0a3d50 msgr2=0x7f1abc0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:34.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.729+0000 7f1adce00700 1 --2- 172.21.15.72:0/586625621 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f1abc0a3d50 0x7f1abc0a6210 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1ac000b2e0 tx=0x7f1ac001a040 comp rx=0 tx=0).stop 2024-04-26T23:45:34.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.729+0000 7f1adce00700 1 -- 172.21.15.72:0/586625621 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1ad8103ea0 msgr2=0x7f1ad819b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:34.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.729+0000 7f1adce00700 1 --2- 172.21.15.72:0/586625621 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1ad8103ea0 0x7f1ad819b6e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1ac800c370 tx=0x7f1ac800c680 comp rx=0 tx=0).stop 2024-04-26T23:45:34.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.729+0000 7f1adce00700 1 -- 172.21.15.72:0/586625621 shutdown_connections 2024-04-26T23:45:34.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.729+0000 7f1adce00700 1 --2- 172.21.15.72:0/586625621 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f1abc0a3d50 0x7f1abc0a6210 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:34.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.729+0000 7f1adce00700 1 --2- 172.21.15.72:0/586625621 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ad8109100 0x7f1ad81a2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:34.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.729+0000 7f1adce00700 1 --2- 172.21.15.72:0/586625621 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1ad8103ea0 0x7f1ad819b6e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:34.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.729+0000 7f1adce00700 1 --2- 172.21.15.72:0/586625621 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1ad81067d0 0x7f1ad819bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:34.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.729+0000 7f1adce00700 1 -- 172.21.15.72:0/586625621 >> 172.21.15.72:0/586625621 conn(0x7f1ad80fd830 msgr2=0x7f1ad8103650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:34.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.730+0000 7f1adce00700 1 -- 172.21.15.72:0/586625621 shutdown_connections 2024-04-26T23:45:34.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:34.730+0000 7f1adce00700 1 -- 172.21.15.72:0/586625621 wait complete. 2024-04-26T23:45:35.686 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-26T23:45:35.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:35 smithi076 ceph-mon[32774]: pgmap v401: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:45:36.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:35 smithi072 ceph-mon[28184]: pgmap v401: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:45:36.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:35 smithi136 ceph-mon[33050]: pgmap v401: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:45:36.146 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:45:36.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:36 smithi076 ceph-mon[32774]: from='client.25232 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:45:37.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:36 smithi072 ceph-mon[28184]: from='client.25232 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:45:37.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:36 smithi136 ceph-mon[33050]: from='client.25232 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:45:37.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:37 smithi076 ceph-mon[32774]: pgmap v402: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:45:37.985 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:37 smithi072 ceph-mon[28184]: pgmap v402: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:45:38.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:37 smithi136 ceph-mon[33050]: pgmap v402: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:45:38.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.557+0000 7f6fcca00700 1 -- 172.21.15.72:0/2587528271 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6fc810acc0 msgr2=0x7f6fc810b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:38.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.557+0000 7f6fcca00700 1 --2- 172.21.15.72:0/2587528271 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6fc810acc0 0x7f6fc810b120 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f6fb40099c0 tx=0x7f6fb4009cd0 comp rx=0 tx=0).stop 2024-04-26T23:45:38.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.557+0000 7f6fcca00700 1 -- 172.21.15.72:0/2587528271 shutdown_connections 2024-04-26T23:45:38.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.557+0000 7f6fcca00700 1 --2- 172.21.15.72:0/2587528271 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6fc810b660 0x7f6fc8111f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:38.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.557+0000 7f6fcca00700 1 --2- 172.21.15.72:0/2587528271 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6fc8109ac0 0x7f6fc8109ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:38.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.557+0000 7f6fcca00700 1 --2- 172.21.15.72:0/2587528271 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6fc810acc0 0x7f6fc810b120 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:38.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.557+0000 7f6fcca00700 1 -- 172.21.15.72:0/2587528271 >> 172.21.15.72:0/2587528271 conn(0x7f6fc8078a90 msgr2=0x7f6fc807aed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:38.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.557+0000 7f6fcca00700 1 -- 172.21.15.72:0/2587528271 shutdown_connections 2024-04-26T23:45:38.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.557+0000 7f6fcca00700 1 -- 172.21.15.72:0/2587528271 wait complete. 2024-04-26T23:45:38.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.559+0000 7f6fcca00700 1 Processor -- start 2024-04-26T23:45:38.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.559+0000 7f6fcca00700 1 -- start start 2024-04-26T23:45:38.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.559+0000 7f6fcca00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6fc8109ac0 0x7f6fc819fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:38.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.559+0000 7f6fcca00700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6fc810acc0 0x7f6fc81a0100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:38.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.559+0000 7f6fcca00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6fc810b660 0x7f6fc81a7180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:38.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.559+0000 7f6fcca00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f6fc8115580 con 0x7f6fc8109ac0 2024-04-26T23:45:38.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.559+0000 7f6fcca00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f6fc8115400 con 0x7f6fc810b660 2024-04-26T23:45:38.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.559+0000 7f6fcca00700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f6fc8115700 con 0x7f6fc810acc0 2024-04-26T23:45:38.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.559+0000 7f6fc5e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6fc8109ac0 0x7f6fc819fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:38.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.560+0000 7f6fc5400700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6fc810acc0 0x7f6fc81a0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:38.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.560+0000 7f6fc5400700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6fc810acc0 0x7f6fc81a0100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.72:60782/0 (socket says 172.21.15.72:60782) 2024-04-26T23:45:38.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.560+0000 7f6fc5e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6fc8109ac0 0x7f6fc819fbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37896/0 (socket says 172.21.15.72:37896) 2024-04-26T23:45:38.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.560+0000 7f6fc5400700 1 -- 172.21.15.72:0/3882708626 learned_addr learned my addr 172.21.15.72:0/3882708626 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:45:38.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.560+0000 7f6fc6800700 1 --2- 172.21.15.72:0/3882708626 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6fc810b660 0x7f6fc81a7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:38.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.560+0000 7f6fc5400700 1 -- 172.21.15.72:0/3882708626 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6fc810b660 msgr2=0x7f6fc81a7180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:38.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.560+0000 7f6fc5400700 1 --2- 172.21.15.72:0/3882708626 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6fc810b660 0x7f6fc81a7180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:38.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.560+0000 7f6fc5400700 1 -- 172.21.15.72:0/3882708626 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6fc8109ac0 msgr2=0x7f6fc819fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:38.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.560+0000 7f6fc5400700 1 --2- 172.21.15.72:0/3882708626 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6fc8109ac0 0x7f6fc819fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:38.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.560+0000 7f6fc5400700 1 -- 172.21.15.72:0/3882708626 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fc81a7820 con 0x7f6fc810acc0 2024-04-26T23:45:38.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.560+0000 7f6fc5e00700 1 --2- 172.21.15.72:0/3882708626 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6fc8109ac0 0x7f6fc819fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:45:38.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.560+0000 7f6fc5400700 1 --2- 172.21.15.72:0/3882708626 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6fc810acc0 0x7f6fc81a0100 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6fb4005910 tx=0x7f6fb4003e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:38.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.561+0000 7f6fbea00700 1 -- 172.21.15.72:0/3882708626 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fb4009e60 con 0x7f6fc810acc0 2024-04-26T23:45:38.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.561+0000 7f6fcca00700 1 -- 172.21.15.72:0/3882708626 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fc81a7ab0 con 0x7f6fc810acc0 2024-04-26T23:45:38.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.561+0000 7f6fbea00700 1 -- 172.21.15.72:0/3882708626 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6fb400be50 con 0x7f6fc810acc0 2024-04-26T23:45:38.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.561+0000 7f6fbea00700 1 -- 172.21.15.72:0/3882708626 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fb400fd90 con 0x7f6fc810acc0 2024-04-26T23:45:38.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.561+0000 7f6fcca00700 1 -- 172.21.15.72:0/3882708626 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fc81a7f10 con 0x7f6fc810acc0 2024-04-26T23:45:38.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.562+0000 7f6fcca00700 1 -- 172.21.15.72:0/3882708626 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fc806a110 con 0x7f6fc810acc0 2024-04-26T23:45:38.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.564+0000 7f6fbea00700 1 -- 172.21.15.72:0/3882708626 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f6fb400f4f0 con 0x7f6fc810acc0 2024-04-26T23:45:38.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.565+0000 7f6fbea00700 1 --2- 172.21.15.72:0/3882708626 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6fac0a3cd0 0x7f6fac0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:38.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.566+0000 7f6fbea00700 1 -- 172.21.15.72:0/3882708626 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f6fb40d1870 con 0x7f6fc810acc0 2024-04-26T23:45:38.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.566+0000 7f6fc5e00700 1 --2- 172.21.15.72:0/3882708626 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6fac0a3cd0 0x7f6fac0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:38.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.566+0000 7f6fc5e00700 1 --2- 172.21.15.72:0/3882708626 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6fac0a3cd0 0x7f6fac0a6190 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6fc810ab20 tx=0x7f6fb8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:38.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.568+0000 7f6fbea00700 1 -- 172.21.15.72:0/3882708626 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6fb4013070 con 0x7f6fc810acc0 2024-04-26T23:45:38.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.810+0000 7f6fcca00700 1 -- 172.21.15.72:0/3882708626 --> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f6fc8063860 con 0x7f6fac0a3cd0 2024-04-26T23:45:38.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.851+0000 7f6fbea00700 1 -- 172.21.15.72:0/3882708626 <== mgr.14152 v2:172.21.15.72:6800/4281177709 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7f6fc8063860 con 0x7f6fac0a3cd0 2024-04-26T23:45:38.854 INFO:teuthology.orchestra.run.smithi072.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-26T23:45:38.855 INFO:teuthology.orchestra.run.smithi072.stdout:mds.a smithi072 running (2m) 52s ago 2m 16.1M - 19.0.0-3306-gbc612ffb 55dc54019ad8 97c724a655b4 2024-04-26T23:45:38.855 INFO:teuthology.orchestra.run.smithi072.stdout:mds.b smithi076 running (2m) 93s ago 2m 26.0M - 19.0.0-3306-gbc612ffb 55dc54019ad8 66c08b6d66f7 2024-04-26T23:45:38.855 INFO:teuthology.orchestra.run.smithi072.stdout:mds.c smithi136 running (2m) 93s ago 2m 18.5M - 19.0.0-3306-gbc612ffb 55dc54019ad8 2a2b7325f4c0 2024-04-26T23:45:38.855 INFO:teuthology.orchestra.run.smithi072.stdout:mds.d smithi072 running (3m) 52s ago 3m 12.5M - 19.0.0-3306-gbc612ffb 55dc54019ad8 7714218b798c 2024-04-26T23:45:38.855 INFO:teuthology.orchestra.run.smithi072.stdout:mds.e smithi076 running (3m) 93s ago 3m 13.0M - 19.0.0-3306-gbc612ffb 55dc54019ad8 6131b82b68ba 2024-04-26T23:45:38.855 INFO:teuthology.orchestra.run.smithi072.stdout:mds.f smithi136 running (3m) 93s ago 3m 18.4M - 19.0.0-3306-gbc612ffb 55dc54019ad8 760eb78752d2 2024-04-26T23:45:38.855 INFO:teuthology.orchestra.run.smithi072.stdout:mds.g smithi072 running (2m) 52s ago 2m 15.9M - 19.0.0-3306-gbc612ffb 55dc54019ad8 e329aec6b4c2 2024-04-26T23:45:38.855 INFO:teuthology.orchestra.run.smithi072.stdout:mds.h smithi076 running (2m) 93s ago 2m 18.4M - 19.0.0-3306-gbc612ffb 55dc54019ad8 9efa73adf794 2024-04-26T23:45:38.855 INFO:teuthology.orchestra.run.smithi072.stdout:mds.i smithi136 running (2m) 93s ago 2m 17.0M - 19.0.0-3306-gbc612ffb 55dc54019ad8 fb4207c7c94b 2024-04-26T23:45:38.856 INFO:teuthology.orchestra.run.smithi072.stdout:mds.j smithi072 running (3m) 52s ago 3m 12.5M - 19.0.0-3306-gbc612ffb 55dc54019ad8 16627108e879 2024-04-26T23:45:38.856 INFO:teuthology.orchestra.run.smithi072.stdout:mds.k smithi076 running (3m) 93s ago 3m 14.1M - 19.0.0-3306-gbc612ffb 55dc54019ad8 78176133a811 2024-04-26T23:45:38.856 INFO:teuthology.orchestra.run.smithi072.stdout:mds.l smithi136 running (3m) 93s ago 3m 15.7M - 19.0.0-3306-gbc612ffb 55dc54019ad8 f0dc138430b0 2024-04-26T23:45:38.856 INFO:teuthology.orchestra.run.smithi072.stdout:mgr.x smithi072 *:9283,8765 running (11m) 52s ago 11m 516M - 19.0.0-3306-gbc612ffb 55dc54019ad8 bf3f10c9dfc7 2024-04-26T23:45:38.856 INFO:teuthology.orchestra.run.smithi072.stdout:mgr.y smithi076 *:8443,8765 running (9m) 93s ago 9m 453M - 19.0.0-3306-gbc612ffb 55dc54019ad8 d9487d6301b7 2024-04-26T23:45:38.856 INFO:teuthology.orchestra.run.smithi072.stdout:mgr.z smithi136 *:8443,8765 running (9m) 93s ago 9m 454M - 19.0.0-3306-gbc612ffb 55dc54019ad8 b4eaa4a59213 2024-04-26T23:45:38.856 INFO:teuthology.orchestra.run.smithi072.stdout:mon.a smithi072 running (11m) 52s ago 11m 48.8M 2048M 19.0.0-3306-gbc612ffb 55dc54019ad8 a100b14c4c02 2024-04-26T23:45:38.856 INFO:teuthology.orchestra.run.smithi072.stdout:mon.b smithi076 running (9m) 93s ago 9m 40.5M 2048M 19.0.0-3306-gbc612ffb 55dc54019ad8 470c048cdec7 2024-04-26T23:45:38.856 INFO:teuthology.orchestra.run.smithi072.stdout:mon.c smithi136 running (9m) 93s ago 9m 43.5M 2048M 19.0.0-3306-gbc612ffb 55dc54019ad8 dcf9bcf7ae9c 2024-04-26T23:45:38.856 INFO:teuthology.orchestra.run.smithi072.stdout:osd.0 smithi072 running (8m) 52s ago 8m 44.6M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 4d99bcfa9a7b 2024-04-26T23:45:38.857 INFO:teuthology.orchestra.run.smithi072.stdout:osd.1 smithi076 running (8m) 93s ago 8m 46.9M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 41515053043a 2024-04-26T23:45:38.857 INFO:teuthology.orchestra.run.smithi072.stdout:osd.2 smithi136 running (8m) 93s ago 8m 46.2M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 8f0a51f4f780 2024-04-26T23:45:38.857 INFO:teuthology.orchestra.run.smithi072.stdout:osd.3 smithi072 running (7m) 52s ago 7m 53.8M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 c0270c8d18d0 2024-04-26T23:45:38.857 INFO:teuthology.orchestra.run.smithi072.stdout:osd.4 smithi076 running (7m) 93s ago 7m 44.9M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 4149ad404ea1 2024-04-26T23:45:38.857 INFO:teuthology.orchestra.run.smithi072.stdout:osd.5 smithi136 running (6m) 93s ago 6m 55.5M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 544c2273c27d 2024-04-26T23:45:38.857 INFO:teuthology.orchestra.run.smithi072.stdout:osd.6 smithi072 running (6m) 52s ago 6m 53.6M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 680f476a0f1d 2024-04-26T23:45:38.857 INFO:teuthology.orchestra.run.smithi072.stdout:osd.7 smithi076 running (5m) 93s ago 5m 55.1M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 b046f7b13818 2024-04-26T23:45:38.858 INFO:teuthology.orchestra.run.smithi072.stdout:osd.8 smithi136 running (5m) 93s ago 5m 45.4M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 e41def0d4aa1 2024-04-26T23:45:38.858 INFO:teuthology.orchestra.run.smithi072.stdout:osd.9 smithi072 running (4m) 52s ago 4m 41.9M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 f07e5c61c056 2024-04-26T23:45:38.858 INFO:teuthology.orchestra.run.smithi072.stdout:osd.10 smithi076 running (4m) 93s ago 4m 56.5M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 9564dae304f8 2024-04-26T23:45:38.858 INFO:teuthology.orchestra.run.smithi072.stdout:osd.11 smithi136 running (3m) 93s ago 3m 42.2M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 573f0f625925 2024-04-26T23:45:38.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.854+0000 7f6fcca00700 1 -- 172.21.15.72:0/3882708626 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6fac0a3cd0 msgr2=0x7f6fac0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:38.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.854+0000 7f6fcca00700 1 --2- 172.21.15.72:0/3882708626 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6fac0a3cd0 0x7f6fac0a6190 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6fc810ab20 tx=0x7f6fb8009260 comp rx=0 tx=0).stop 2024-04-26T23:45:38.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.854+0000 7f6fcca00700 1 -- 172.21.15.72:0/3882708626 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6fc810acc0 msgr2=0x7f6fc81a0100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:38.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.854+0000 7f6fcca00700 1 --2- 172.21.15.72:0/3882708626 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6fc810acc0 0x7f6fc81a0100 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6fb4005910 tx=0x7f6fb4003e60 comp rx=0 tx=0).stop 2024-04-26T23:45:38.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.854+0000 7f6fcca00700 1 -- 172.21.15.72:0/3882708626 shutdown_connections 2024-04-26T23:45:38.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.854+0000 7f6fcca00700 1 --2- 172.21.15.72:0/3882708626 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6fac0a3cd0 0x7f6fac0a6190 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:38.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.854+0000 7f6fcca00700 1 --2- 172.21.15.72:0/3882708626 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6fc810b660 0x7f6fc81a7180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:38.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.854+0000 7f6fcca00700 1 --2- 172.21.15.72:0/3882708626 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6fc8109ac0 0x7f6fc819fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:38.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.854+0000 7f6fcca00700 1 --2- 172.21.15.72:0/3882708626 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6fc810acc0 0x7f6fc81a0100 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:38.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.854+0000 7f6fcca00700 1 -- 172.21.15.72:0/3882708626 >> 172.21.15.72:0/3882708626 conn(0x7f6fc8078a90 msgr2=0x7f6fc810dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:38.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.854+0000 7f6fcca00700 1 -- 172.21.15.72:0/3882708626 shutdown_connections 2024-04-26T23:45:38.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:38.854+0000 7f6fcca00700 1 -- 172.21.15.72:0/3882708626 wait complete. 2024-04-26T23:45:39.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:38 smithi072 ceph-mon[28184]: pgmap v403: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:45:39.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:38 smithi136 ceph-mon[33050]: pgmap v403: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:45:39.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:38 smithi076 ceph-mon[32774]: pgmap v403: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:45:39.826 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-26T23:45:40.045 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:39 smithi072 ceph-mon[28184]: from='client.25210 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:45:40.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:39 smithi136 ceph-mon[33050]: from='client.25210 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:45:40.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:39 smithi076 ceph-mon[32774]: from='client.25210 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:45:40.288 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:45:41.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:40 smithi072 ceph-mon[28184]: pgmap v404: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:45:41.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:40 smithi136 ceph-mon[33050]: pgmap v404: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:45:41.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:40 smithi076 ceph-mon[32774]: pgmap v404: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:45:42.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.589+0000 7f730ca00700 1 -- 172.21.15.72:0/1986304693 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f730810b660 msgr2=0x7f7308111f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:42.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.589+0000 7f730ca00700 1 --2- 172.21.15.72:0/1986304693 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f730810b660 0x7f7308111f10 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f730000b0d0 tx=0x7f730000b3e0 comp rx=0 tx=0).stop 2024-04-26T23:45:42.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.589+0000 7f730ca00700 1 -- 172.21.15.72:0/1986304693 shutdown_connections 2024-04-26T23:45:42.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.589+0000 7f730ca00700 1 --2- 172.21.15.72:0/1986304693 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f730810b660 0x7f7308111f10 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:42.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.589+0000 7f730ca00700 1 --2- 172.21.15.72:0/1986304693 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7308109ac0 0x7f7308109ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:42.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.589+0000 7f730ca00700 1 --2- 172.21.15.72:0/1986304693 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f730810acc0 0x7f730810b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:42.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.589+0000 7f730ca00700 1 -- 172.21.15.72:0/1986304693 >> 172.21.15.72:0/1986304693 conn(0x7f7308078a90 msgr2=0x7f730807aed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:42.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.589+0000 7f730ca00700 1 -- 172.21.15.72:0/1986304693 shutdown_connections 2024-04-26T23:45:42.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.589+0000 7f730ca00700 1 -- 172.21.15.72:0/1986304693 wait complete. 2024-04-26T23:45:42.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.590+0000 7f730ca00700 1 Processor -- start 2024-04-26T23:45:42.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.591+0000 7f730ca00700 1 -- start start 2024-04-26T23:45:42.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.591+0000 7f730ca00700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7308109ac0 0x7f730819fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:42.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.591+0000 7f730ca00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f730810acc0 0x7f73081a0090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:42.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.591+0000 7f730ca00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f730810b660 0x7f73081a7110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:42.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.591+0000 7f730ca00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f7308115560 con 0x7f730810b660 2024-04-26T23:45:42.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.591+0000 7f730ca00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f73081153e0 con 0x7f730810acc0 2024-04-26T23:45:42.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.591+0000 7f730ca00700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f73081156e0 con 0x7f7308109ac0 2024-04-26T23:45:42.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.591+0000 7f7306800700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f730810b660 0x7f73081a7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:42.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.591+0000 7f7305e00700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7308109ac0 0x7f730819fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:42.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.591+0000 7f7305e00700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7308109ac0 0x7f730819fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.72:60786/0 (socket says 172.21.15.72:60786) 2024-04-26T23:45:42.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.591+0000 7f7305e00700 1 -- 172.21.15.72:0/2143488542 learned_addr learned my addr 172.21.15.72:0/2143488542 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:45:42.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.591+0000 7f7305400700 1 --2- 172.21.15.72:0/2143488542 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f730810acc0 0x7f73081a0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:42.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.592+0000 7f7305e00700 1 -- 172.21.15.72:0/2143488542 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f730810acc0 msgr2=0x7f73081a0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:42.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.592+0000 7f7305e00700 1 --2- 172.21.15.72:0/2143488542 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f730810acc0 0x7f73081a0090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:42.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.592+0000 7f7305e00700 1 -- 172.21.15.72:0/2143488542 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f730810b660 msgr2=0x7f73081a7110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:42.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.592+0000 7f7305e00700 1 --2- 172.21.15.72:0/2143488542 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f730810b660 0x7f73081a7110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:42.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.592+0000 7f7305e00700 1 -- 172.21.15.72:0/2143488542 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73081a77b0 con 0x7f7308109ac0 2024-04-26T23:45:42.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.592+0000 7f7305400700 1 --2- 172.21.15.72:0/2143488542 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f730810acc0 0x7f73081a0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:45:42.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.592+0000 7f7306800700 1 --2- 172.21.15.72:0/2143488542 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f730810b660 0x7f73081a7110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:45:42.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.592+0000 7f7305e00700 1 --2- 172.21.15.72:0/2143488542 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7308109ac0 0x7f730819fb50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f72f800e9a0 tx=0x7f72f800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:42.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.592+0000 7f72fea00700 1 -- 172.21.15.72:0/2143488542 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72f8004220 con 0x7f7308109ac0 2024-04-26T23:45:42.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.592+0000 7f730ca00700 1 -- 172.21.15.72:0/2143488542 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73081a7aa0 con 0x7f7308109ac0 2024-04-26T23:45:42.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.593+0000 7f72fea00700 1 -- 172.21.15.72:0/2143488542 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72f80043c0 con 0x7f7308109ac0 2024-04-26T23:45:42.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.593+0000 7f72fea00700 1 -- 172.21.15.72:0/2143488542 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72f8003cc0 con 0x7f7308109ac0 2024-04-26T23:45:42.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.593+0000 7f730ca00700 1 -- 172.21.15.72:0/2143488542 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73081a8030 con 0x7f7308109ac0 2024-04-26T23:45:42.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.594+0000 7f730ca00700 1 -- 172.21.15.72:0/2143488542 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f730806a110 con 0x7f7308109ac0 2024-04-26T23:45:42.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.596+0000 7f72fea00700 1 -- 172.21.15.72:0/2143488542 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f72f800c810 con 0x7f7308109ac0 2024-04-26T23:45:42.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.597+0000 7f72fea00700 1 --2- 172.21.15.72:0/2143488542 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f72ec0a3cd0 0x7f72ec0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:42.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.597+0000 7f72fea00700 1 -- 172.21.15.72:0/2143488542 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f72f80d0d10 con 0x7f7308109ac0 2024-04-26T23:45:42.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.597+0000 7f7305400700 1 --2- 172.21.15.72:0/2143488542 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f72ec0a3cd0 0x7f72ec0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:42.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.598+0000 7f7305400700 1 --2- 172.21.15.72:0/2143488542 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f72ec0a3cd0 0x7f72ec0a6190 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f72f4011490 tx=0x7f72f4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:42.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.599+0000 7f72fea00700 1 -- 172.21.15.72:0/2143488542 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f72f809d140 con 0x7f7308109ac0 2024-04-26T23:45:42.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.838+0000 7f730ca00700 1 -- 172.21.15.72:0/2143488542 --> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f7308063860 con 0x7f72ec0a3cd0 2024-04-26T23:45:42.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.854+0000 7f72fea00700 1 -- 172.21.15.72:0/2143488542 <== mgr.14152 v2:172.21.15.72:6800/4281177709 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f7308063860 con 0x7f72ec0a3cd0 2024-04-26T23:45:42.859 INFO:teuthology.orchestra.run.smithi072.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-26T23:45:42.859 INFO:teuthology.orchestra.run.smithi072.stdout:mds.all 12/12 97s ago 3m smithi072=a;smithi072=d;smithi072=g;smithi072=j;smithi076=b;smithi076=e;smithi076=h;smithi076=k;smithi136=c;smithi136=f;smithi136=i;smithi136=l;count:12 2024-04-26T23:45:42.859 INFO:teuthology.orchestra.run.smithi072.stdout:mgr 3/3 97s ago 9m smithi072=x;smithi076=y;smithi136=z;count:3 2024-04-26T23:45:42.859 INFO:teuthology.orchestra.run.smithi072.stdout:mon 3/3 97s ago 9m smithi072:172.21.15.72=a;smithi076:172.21.15.76=b;smithi136:172.21.15.136=c;count:3 2024-04-26T23:45:42.859 INFO:teuthology.orchestra.run.smithi072.stdout:osd 12 97s ago - 2024-04-26T23:45:42.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.858+0000 7f730ca00700 1 -- 172.21.15.72:0/2143488542 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f72ec0a3cd0 msgr2=0x7f72ec0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:42.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.858+0000 7f730ca00700 1 --2- 172.21.15.72:0/2143488542 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f72ec0a3cd0 0x7f72ec0a6190 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f72f4011490 tx=0x7f72f4009260 comp rx=0 tx=0).stop 2024-04-26T23:45:42.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.858+0000 7f730ca00700 1 -- 172.21.15.72:0/2143488542 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7308109ac0 msgr2=0x7f730819fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:42.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.858+0000 7f730ca00700 1 --2- 172.21.15.72:0/2143488542 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7308109ac0 0x7f730819fb50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f72f800e9a0 tx=0x7f72f800ecb0 comp rx=0 tx=0).stop 2024-04-26T23:45:42.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.858+0000 7f730ca00700 1 -- 172.21.15.72:0/2143488542 shutdown_connections 2024-04-26T23:45:42.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.858+0000 7f730ca00700 1 --2- 172.21.15.72:0/2143488542 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f72ec0a3cd0 0x7f72ec0a6190 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:42.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.858+0000 7f730ca00700 1 --2- 172.21.15.72:0/2143488542 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f730810b660 0x7f73081a7110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:42.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.858+0000 7f730ca00700 1 --2- 172.21.15.72:0/2143488542 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7308109ac0 0x7f730819fb50 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:42.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.858+0000 7f730ca00700 1 --2- 172.21.15.72:0/2143488542 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f730810acc0 0x7f73081a0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:42.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.858+0000 7f730ca00700 1 -- 172.21.15.72:0/2143488542 >> 172.21.15.72:0/2143488542 conn(0x7f7308078a90 msgr2=0x7f730810dfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:42.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.858+0000 7f730ca00700 1 -- 172.21.15.72:0/2143488542 shutdown_connections 2024-04-26T23:45:42.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:42.859+0000 7f730ca00700 1 -- 172.21.15.72:0/2143488542 wait complete. 2024-04-26T23:45:43.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:43 smithi072 ceph-mon[28184]: pgmap v405: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:45:43.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:43 smithi136 ceph-mon[33050]: pgmap v405: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:45:43.888 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-26T23:45:43.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:43 smithi076 ceph-mon[32774]: pgmap v405: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:45:44.324 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:45:44.726 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:44 smithi072 ceph-mon[28184]: from='client.25216 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:45:44.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:44 smithi136 ceph-mon[33050]: from='client.25216 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:45:44.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:44 smithi076 ceph-mon[32774]: from='client.25216 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:45:45.739 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:45 smithi072 ceph-mon[28184]: pgmap v406: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:45:45.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:45 smithi076 ceph-mon[32774]: pgmap v406: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:45:46.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:45 smithi136 ceph-mon[33050]: pgmap v406: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:45:46.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.546+0000 7f3a70000700 1 -- 172.21.15.72:0/3401084868 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a681056d0 msgr2=0x7f3a68105af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:46.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.546+0000 7f3a70000700 1 --2- 172.21.15.72:0/3401084868 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a681056d0 0x7f3a68105af0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c0099c0 tx=0x7f3a5c009cd0 comp rx=0 tx=0).stop 2024-04-26T23:45:46.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.546+0000 7f3a70000700 1 -- 172.21.15.72:0/3401084868 shutdown_connections 2024-04-26T23:45:46.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.546+0000 7f3a70000700 1 --2- 172.21.15.72:0/3401084868 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a68107270 0x7f3a6810db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:46.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.546+0000 7f3a70000700 1 --2- 172.21.15.72:0/3401084868 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a681056d0 0x7f3a68105af0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:46.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.546+0000 7f3a70000700 1 --2- 172.21.15.72:0/3401084868 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a681068d0 0x7f3a68106d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:46.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.546+0000 7f3a70000700 1 -- 172.21.15.72:0/3401084868 >> 172.21.15.72:0/3401084868 conn(0x7f3a68100c70 msgr2=0x7f3a681030b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:46.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.547+0000 7f3a70000700 1 -- 172.21.15.72:0/3401084868 shutdown_connections 2024-04-26T23:45:46.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.547+0000 7f3a70000700 1 -- 172.21.15.72:0/3401084868 wait complete. 2024-04-26T23:45:46.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.548+0000 7f3a70000700 1 Processor -- start 2024-04-26T23:45:46.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.548+0000 7f3a70000700 1 -- start start 2024-04-26T23:45:46.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.548+0000 7f3a70000700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a681056d0 0x7f3a6819b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:46.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.548+0000 7f3a70000700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a681068d0 0x7f3a6819bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:46.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.548+0000 7f3a70000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a68107270 0x7f3a681a2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:46.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.548+0000 7f3a70000700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f3a68111170 con 0x7f3a681068d0 2024-04-26T23:45:46.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.548+0000 7f3a70000700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3a68110ff0 con 0x7f3a68107270 2024-04-26T23:45:46.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.548+0000 7f3a70000700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f3a681112f0 con 0x7f3a681056d0 2024-04-26T23:45:46.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.549+0000 7f3a6cc00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a681068d0 0x7f3a6819bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:46.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.549+0000 7f3a6d600700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a681056d0 0x7f3a6819b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:46.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.549+0000 7f3a6cc00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a681068d0 0x7f3a6819bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37940/0 (socket says 172.21.15.72:37940) 2024-04-26T23:45:46.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.549+0000 7f3a6cc00700 1 -- 172.21.15.72:0/876402905 learned_addr learned my addr 172.21.15.72:0/876402905 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:45:46.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.549+0000 7f3a6e000700 1 --2- 172.21.15.72:0/876402905 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a68107270 0x7f3a681a2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:46.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.549+0000 7f3a6d600700 1 -- 172.21.15.72:0/876402905 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a68107270 msgr2=0x7f3a681a2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:46.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.549+0000 7f3a6d600700 1 --2- 172.21.15.72:0/876402905 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a68107270 0x7f3a681a2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:46.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.549+0000 7f3a6d600700 1 -- 172.21.15.72:0/876402905 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a681068d0 msgr2=0x7f3a6819bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:46.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.549+0000 7f3a6d600700 1 --2- 172.21.15.72:0/876402905 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a681068d0 0x7f3a6819bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:46.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.549+0000 7f3a6d600700 1 -- 172.21.15.72:0/876402905 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a681a3440 con 0x7f3a681056d0 2024-04-26T23:45:46.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.550+0000 7f3a6d600700 1 --2- 172.21.15.72:0/876402905 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a681056d0 0x7f3a6819b830 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c0099c0 tx=0x7f3a5c003b70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:46.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.550+0000 7f3a66000700 1 -- 172.21.15.72:0/876402905 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a5c009e60 con 0x7f3a681056d0 2024-04-26T23:45:46.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.550+0000 7f3a70000700 1 -- 172.21.15.72:0/876402905 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a681a36d0 con 0x7f3a681056d0 2024-04-26T23:45:46.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.550+0000 7f3a66000700 1 -- 172.21.15.72:0/876402905 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a5c005570 con 0x7f3a681056d0 2024-04-26T23:45:46.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.550+0000 7f3a66000700 1 -- 172.21.15.72:0/876402905 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a5c00fbd0 con 0x7f3a681056d0 2024-04-26T23:45:46.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.550+0000 7f3a70000700 1 -- 172.21.15.72:0/876402905 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a681a3c00 con 0x7f3a681056d0 2024-04-26T23:45:46.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.551+0000 7f3a70000700 1 -- 172.21.15.72:0/876402905 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a6806a110 con 0x7f3a681056d0 2024-04-26T23:45:46.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.553+0000 7f3a66000700 1 -- 172.21.15.72:0/876402905 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3a5c00f460 con 0x7f3a681056d0 2024-04-26T23:45:46.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.555+0000 7f3a66000700 1 --2- 172.21.15.72:0/876402905 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f3a500a3cd0 0x7f3a500a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:46.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.555+0000 7f3a66000700 1 -- 172.21.15.72:0/876402905 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f3a5c0d1710 con 0x7f3a681056d0 2024-04-26T23:45:46.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.555+0000 7f3a6cc00700 1 --2- 172.21.15.72:0/876402905 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f3a500a3cd0 0x7f3a500a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:46.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.556+0000 7f3a6cc00700 1 --2- 172.21.15.72:0/876402905 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f3a500a3cd0 0x7f3a500a6190 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f3a54009d20 tx=0x7f3a54009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:46.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.557+0000 7f3a66000700 1 -- 172.21.15.72:0/876402905 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f3a5c013070 con 0x7f3a681056d0 2024-04-26T23:45:46.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.791+0000 7f3a70000700 1 -- 172.21.15.72:0/876402905 --> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f3a68063860 con 0x7f3a500a3cd0 2024-04-26T23:45:46.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.793+0000 7f3a66000700 1 -- 172.21.15.72:0/876402905 <== mgr.14152 v2:172.21.15.72:6800/4281177709 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f3a68063860 con 0x7f3a500a3cd0 2024-04-26T23:45:46.797 INFO:teuthology.orchestra.run.smithi072.stdout:HOST ADDR LABELS STATUS 2024-04-26T23:45:46.798 INFO:teuthology.orchestra.run.smithi072.stdout:smithi072 172.21.15.72 2024-04-26T23:45:46.798 INFO:teuthology.orchestra.run.smithi072.stdout:smithi076 172.21.15.76 2024-04-26T23:45:46.798 INFO:teuthology.orchestra.run.smithi072.stdout:smithi136 172.21.15.136 2024-04-26T23:45:46.798 INFO:teuthology.orchestra.run.smithi072.stdout:3 hosts in cluster 2024-04-26T23:45:46.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.796+0000 7f3a70000700 1 -- 172.21.15.72:0/876402905 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f3a500a3cd0 msgr2=0x7f3a500a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:46.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.796+0000 7f3a70000700 1 --2- 172.21.15.72:0/876402905 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f3a500a3cd0 0x7f3a500a6190 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f3a54009d20 tx=0x7f3a54009260 comp rx=0 tx=0).stop 2024-04-26T23:45:46.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.796+0000 7f3a70000700 1 -- 172.21.15.72:0/876402905 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a681056d0 msgr2=0x7f3a6819b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:46.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.796+0000 7f3a70000700 1 --2- 172.21.15.72:0/876402905 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a681056d0 0x7f3a6819b830 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c0099c0 tx=0x7f3a5c003b70 comp rx=0 tx=0).stop 2024-04-26T23:45:46.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.796+0000 7f3a70000700 1 -- 172.21.15.72:0/876402905 shutdown_connections 2024-04-26T23:45:46.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.796+0000 7f3a70000700 1 --2- 172.21.15.72:0/876402905 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f3a500a3cd0 0x7f3a500a6190 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:46.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.796+0000 7f3a70000700 1 --2- 172.21.15.72:0/876402905 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a68107270 0x7f3a681a2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:46.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.796+0000 7f3a70000700 1 --2- 172.21.15.72:0/876402905 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a681056d0 0x7f3a6819b830 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:46.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.796+0000 7f3a70000700 1 --2- 172.21.15.72:0/876402905 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a681068d0 0x7f3a6819bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:46.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.796+0000 7f3a70000700 1 -- 172.21.15.72:0/876402905 >> 172.21.15.72:0/876402905 conn(0x7f3a68100c70 msgr2=0x7f3a68109bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:46.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.796+0000 7f3a70000700 1 -- 172.21.15.72:0/876402905 shutdown_connections 2024-04-26T23:45:46.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:46.796+0000 7f3a70000700 1 -- 172.21.15.72:0/876402905 wait complete. 2024-04-26T23:45:47.594 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-26T23:45:47.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:47 smithi072 ceph-mon[28184]: pgmap v407: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:45:47.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:47 smithi076 ceph-mon[32774]: pgmap v407: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:45:48.037 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:45:48.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:47 smithi136 ceph-mon[33050]: pgmap v407: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:45:48.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:48 smithi072 ceph-mon[28184]: from='client.25222 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:45:48.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:48 smithi076 ceph-mon[32774]: from='client.25222 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:45:49.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:48 smithi136 ceph-mon[33050]: from='client.25222 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:45:49.783 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:49 smithi072 ceph-mon[28184]: pgmap v408: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:45:49.783 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:49 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:45:49.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:49 smithi076 ceph-mon[32774]: pgmap v408: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:45:49.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:49 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:45:50.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:49 smithi136 ceph-mon[33050]: pgmap v408: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:45:50.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:49 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:45:50.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.291+0000 7fc47d600700 1 -- 172.21.15.72:0/2895397239 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc478104690 msgr2=0x7fc478104af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:50.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.291+0000 7fc47d600700 1 --2- 172.21.15.72:0/2895397239 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc478104690 0x7fc478104af0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc4640099c0 tx=0x7fc464009cd0 comp rx=0 tx=0).stop 2024-04-26T23:45:50.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.292+0000 7fc47d600700 1 -- 172.21.15.72:0/2895397239 shutdown_connections 2024-04-26T23:45:50.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.292+0000 7fc47d600700 1 --2- 172.21.15.72:0/2895397239 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc478105030 0x7fc47810b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:50.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.292+0000 7fc47d600700 1 --2- 172.21.15.72:0/2895397239 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc478103490 0x7fc4781038b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:50.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.292+0000 7fc47d600700 1 --2- 172.21.15.72:0/2895397239 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc478104690 0x7fc478104af0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:50.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.292+0000 7fc47d600700 1 -- 172.21.15.72:0/2895397239 >> 172.21.15.72:0/2895397239 conn(0x7fc4780fe9f0 msgr2=0x7fc478100e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:50.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.292+0000 7fc47d600700 1 -- 172.21.15.72:0/2895397239 shutdown_connections 2024-04-26T23:45:50.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.292+0000 7fc47d600700 1 -- 172.21.15.72:0/2895397239 wait complete. 2024-04-26T23:45:50.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.293+0000 7fc47d600700 1 Processor -- start 2024-04-26T23:45:50.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.293+0000 7fc47d600700 1 -- start start 2024-04-26T23:45:50.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.294+0000 7fc47d600700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc478103490 0x7fc4780771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:50.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.294+0000 7fc47d600700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc478104690 0x7fc478075810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:50.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.294+0000 7fc47d600700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc478105030 0x7fc478075d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:50.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.294+0000 7fc47d600700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fc47810ef80 con 0x7fc478104690 2024-04-26T23:45:50.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.294+0000 7fc47d600700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fc47810ee00 con 0x7fc478103490 2024-04-26T23:45:50.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.294+0000 7fc47d600700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fc47810f100 con 0x7fc478105030 2024-04-26T23:45:50.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.294+0000 7fc476a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc478103490 0x7fc4780771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:50.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.294+0000 7fc476a00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc478103490 0x7fc4780771c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:38234/0 (socket says 172.21.15.72:38234) 2024-04-26T23:45:50.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.294+0000 7fc476a00700 1 -- 172.21.15.72:0/2985836923 learned_addr learned my addr 172.21.15.72:0/2985836923 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:45:50.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.294+0000 7fc476000700 1 --2- 172.21.15.72:0/2985836923 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc478104690 0x7fc478075810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:50.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.294+0000 7fc477400700 1 --2- 172.21.15.72:0/2985836923 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc478105030 0x7fc478075d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:50.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.295+0000 7fc476a00700 1 -- 172.21.15.72:0/2985836923 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc478105030 msgr2=0x7fc478075d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:50.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.295+0000 7fc476a00700 1 --2- 172.21.15.72:0/2985836923 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc478105030 0x7fc478075d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:50.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.295+0000 7fc476a00700 1 -- 172.21.15.72:0/2985836923 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc478104690 msgr2=0x7fc478075810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:50.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.295+0000 7fc476a00700 1 --2- 172.21.15.72:0/2985836923 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc478104690 0x7fc478075810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:50.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.295+0000 7fc476a00700 1 -- 172.21.15.72:0/2985836923 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc478076600 con 0x7fc478103490 2024-04-26T23:45:50.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.295+0000 7fc476a00700 1 --2- 172.21.15.72:0/2985836923 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc478103490 0x7fc4780771c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc46800e970 tx=0x7fc46800ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:50.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.295+0000 7fc46f400700 1 -- 172.21.15.72:0/2985836923 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc46800cbd0 con 0x7fc478103490 2024-04-26T23:45:50.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.295+0000 7fc46f400700 1 -- 172.21.15.72:0/2985836923 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4680044d0 con 0x7fc478103490 2024-04-26T23:45:50.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.295+0000 7fc47d600700 1 -- 172.21.15.72:0/2985836923 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4781a3790 con 0x7fc478103490 2024-04-26T23:45:50.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.295+0000 7fc46f400700 1 -- 172.21.15.72:0/2985836923 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc468003e40 con 0x7fc478103490 2024-04-26T23:45:50.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.295+0000 7fc47d600700 1 -- 172.21.15.72:0/2985836923 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4781a3ca0 con 0x7fc478103490 2024-04-26T23:45:50.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.297+0000 7fc47d600700 1 -- 172.21.15.72:0/2985836923 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc47806a110 con 0x7fc478103490 2024-04-26T23:45:50.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.298+0000 7fc46f400700 1 -- 172.21.15.72:0/2985836923 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc468004020 con 0x7fc478103490 2024-04-26T23:45:50.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.299+0000 7fc46f400700 1 --2- 172.21.15.72:0/2985836923 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fc45c0a3cd0 0x7fc45c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:50.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.300+0000 7fc46f400700 1 -- 172.21.15.72:0/2985836923 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7fc4680d0d60 con 0x7fc478103490 2024-04-26T23:45:50.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.300+0000 7fc476000700 1 --2- 172.21.15.72:0/2985836923 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fc45c0a3cd0 0x7fc45c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:50.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.301+0000 7fc476000700 1 --2- 172.21.15.72:0/2985836923 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fc45c0a3cd0 0x7fc45c0a6190 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fc464000c00 tx=0x7fc464005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:50.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.304+0000 7fc46f400700 1 -- 172.21.15.72:0/2985836923 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc468016330 con 0x7fc478103490 2024-04-26T23:45:50.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.544+0000 7fc47d600700 1 -- 172.21.15.72:0/2985836923 --> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7fc478063860 con 0x7fc45c0a3cd0 2024-04-26T23:45:50.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.548+0000 7fc46f400700 1 -- 172.21.15.72:0/2985836923 <== mgr.14152 v2:172.21.15.72:6800/4281177709 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7fc478063860 con 0x7fc45c0a3cd0 2024-04-26T23:45:50.552 INFO:teuthology.orchestra.run.smithi072.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-26T23:45:50.552 INFO:teuthology.orchestra.run.smithi072.stdout:smithi072 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400V7400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-04-26T23:45:50.553 INFO:teuthology.orchestra.run.smithi072.stdout:smithi076 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400U4400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-04-26T23:45:50.553 INFO:teuthology.orchestra.run.smithi072.stdout:smithi136 /dev/nvme0n1 ssd INTEL_SSDPEDMX020T4_CVPD514100122P0SGN 1863G No 3m ago Has a FileSystem, LVM detected 2024-04-26T23:45:50.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.551+0000 7fc47d600700 1 -- 172.21.15.72:0/2985836923 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fc45c0a3cd0 msgr2=0x7fc45c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:50.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.551+0000 7fc47d600700 1 --2- 172.21.15.72:0/2985836923 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fc45c0a3cd0 0x7fc45c0a6190 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fc464000c00 tx=0x7fc464005fb0 comp rx=0 tx=0).stop 2024-04-26T23:45:50.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.551+0000 7fc47d600700 1 -- 172.21.15.72:0/2985836923 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc478103490 msgr2=0x7fc4780771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:50.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.551+0000 7fc47d600700 1 --2- 172.21.15.72:0/2985836923 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc478103490 0x7fc4780771c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc46800e970 tx=0x7fc46800ed30 comp rx=0 tx=0).stop 2024-04-26T23:45:50.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.552+0000 7fc47d600700 1 -- 172.21.15.72:0/2985836923 shutdown_connections 2024-04-26T23:45:50.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.552+0000 7fc47d600700 1 --2- 172.21.15.72:0/2985836923 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fc45c0a3cd0 0x7fc45c0a6190 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:50.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.552+0000 7fc47d600700 1 --2- 172.21.15.72:0/2985836923 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc478105030 0x7fc478075d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:50.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.552+0000 7fc47d600700 1 --2- 172.21.15.72:0/2985836923 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc478103490 0x7fc4780771c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:50.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.552+0000 7fc47d600700 1 --2- 172.21.15.72:0/2985836923 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc478104690 0x7fc478075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:50.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.552+0000 7fc47d600700 1 -- 172.21.15.72:0/2985836923 >> 172.21.15.72:0/2985836923 conn(0x7fc4780fe9f0 msgr2=0x7fc478107a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:50.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.552+0000 7fc47d600700 1 -- 172.21.15.72:0/2985836923 shutdown_connections 2024-04-26T23:45:50.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:50.552+0000 7fc47d600700 1 -- 172.21.15.72:0/2985836923 wait complete. 2024-04-26T23:45:51.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:51 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:45:51.294 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:51 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:45:51.295 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:51 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:45:51.295 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:51 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:45:51.295 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:51 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:45:51.295 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:51 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:45:51.295 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:51 smithi072 ceph-mon[28184]: pgmap v409: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:45:51.295 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:51 smithi072 ceph-mon[28184]: from='client.25256 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:45:51.295 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:51 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:45:51.295 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:51 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:45:51.295 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:51 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:45:51.295 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:51 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:45:51.295 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:51 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:45:51.295 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:51 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:45:51.295 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:51 smithi076 ceph-mon[32774]: pgmap v409: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:45:51.295 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:51 smithi076 ceph-mon[32774]: from='client.25256 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:45:51.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:51 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:45:51.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:51 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:45:51.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:51 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:45:51.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:51 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:45:51.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:51 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:45:51.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:51 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:45:51.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:51 smithi136 ceph-mon[33050]: pgmap v409: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:45:51.300 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:51 smithi136 ceph-mon[33050]: from='client.25256 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:45:51.396 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-26T23:45:51.406 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi072.front.sepia.ceph.com 2024-04-26T23:45:51.406 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-26T23:45:52.460 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:45:52.783 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:52 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:45:52.783 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:52 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:45:52.783 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:52 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:45:52.783 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:52 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:45:52.784 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:52 smithi072 ceph-mon[28184]: pgmap v410: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:45:53.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:52 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:45:53.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:52 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:45:53.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:52 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:45:53.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:52 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:45:53.051 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:52 smithi136 ceph-mon[33050]: pgmap v410: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:45:53.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:52 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:45:53.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:52 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:45:53.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:52 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:45:53.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:52 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:45:53.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:52 smithi076 ceph-mon[32774]: pgmap v410: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:45:54.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:54.838+0000 7fde00c00700 1 -- 172.21.15.72:0/4116743655 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fddfc10fa70 msgr2=0x7fddfc111e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:54.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:54.838+0000 7fde00c00700 1 --2- 172.21.15.72:0/4116743655 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fddfc10fa70 0x7fddfc111e60 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fddf400b0a0 tx=0x7fddf400b3b0 comp rx=0 tx=0).stop 2024-04-26T23:45:54.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:54.839+0000 7fde00c00700 1 -- 172.21.15.72:0/4116743655 shutdown_connections 2024-04-26T23:45:54.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:54.839+0000 7fde00c00700 1 --2- 172.21.15.72:0/4116743655 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fddfc10fa70 0x7fddfc111e60 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:54.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:54.839+0000 7fde00c00700 1 --2- 172.21.15.72:0/4116743655 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fddfc077450 0x7fddfc0758b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:54.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:54.839+0000 7fde00c00700 1 --2- 172.21.15.72:0/4116743655 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fddfc075e80 0x7fddfc076300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:54.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:54.839+0000 7fde00c00700 1 -- 172.21.15.72:0/4116743655 >> 172.21.15.72:0/4116743655 conn(0x7fddfc0fe9f0 msgr2=0x7fddfc100e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:54.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:54.839+0000 7fde00c00700 1 -- 172.21.15.72:0/4116743655 shutdown_connections 2024-04-26T23:45:54.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:54.839+0000 7fde00c00700 1 -- 172.21.15.72:0/4116743655 wait complete. 2024-04-26T23:45:54.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:54.840+0000 7fde00c00700 1 Processor -- start 2024-04-26T23:45:54.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:54.840+0000 7fde00c00700 1 -- start start 2024-04-26T23:45:54.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:54.841+0000 7fde00c00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fddfc075e80 0x7fddfc19fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:54.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:54.841+0000 7fde00c00700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fddfc077450 0x7fddfc1a0110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:54.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:54.841+0000 7fde00c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fddfc10fa70 0x7fddfc1a7190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:54.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:54.841+0000 7fde00c00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fddfc115570 con 0x7fddfc075e80 2024-04-26T23:45:54.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:54.841+0000 7fde00c00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fddfc1153f0 con 0x7fddfc10fa70 2024-04-26T23:45:54.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:54.841+0000 7fde00c00700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fddfc1156f0 con 0x7fddfc077450 2024-04-26T23:45:54.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:54.841+0000 7fddf9400700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fddfc077450 0x7fddfc1a0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:54.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:54.841+0000 7fddf9e00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fddfc075e80 0x7fddfc19fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:54.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:54.841+0000 7fddf9400700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fddfc077450 0x7fddfc1a0110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.72:50346/0 (socket says 172.21.15.72:50346) 2024-04-26T23:45:54.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:54.841+0000 7fddf9400700 1 -- 172.21.15.72:0/1210789390 learned_addr learned my addr 172.21.15.72:0/1210789390 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:45:54.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:54.841+0000 7fddfa800700 1 --2- 172.21.15.72:0/1210789390 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fddfc10fa70 0x7fddfc1a7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:54.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:54.842+0000 7fddf9400700 1 -- 172.21.15.72:0/1210789390 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fddfc10fa70 msgr2=0x7fddfc1a7190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:54.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:54.842+0000 7fddf9400700 1 --2- 172.21.15.72:0/1210789390 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fddfc10fa70 0x7fddfc1a7190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:54.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:54.842+0000 7fddf9400700 1 -- 172.21.15.72:0/1210789390 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fddfc075e80 msgr2=0x7fddfc19fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:54.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:54.842+0000 7fddf9400700 1 --2- 172.21.15.72:0/1210789390 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fddfc075e80 0x7fddfc19fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:54.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:54.842+0000 7fddf9400700 1 -- 172.21.15.72:0/1210789390 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddfc1a7830 con 0x7fddfc077450 2024-04-26T23:45:54.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:54.842+0000 7fddf9400700 1 --2- 172.21.15.72:0/1210789390 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fddfc077450 0x7fddfc1a0110 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fdde800e9a0 tx=0x7fdde800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:54.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:54.842+0000 7fddf2a00700 1 -- 172.21.15.72:0/1210789390 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdde8004220 con 0x7fddfc077450 2024-04-26T23:45:54.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:54.842+0000 7fddf2a00700 1 -- 172.21.15.72:0/1210789390 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdde80043c0 con 0x7fddfc077450 2024-04-26T23:45:54.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:54.842+0000 7fde00c00700 1 -- 172.21.15.72:0/1210789390 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddfc1a7b20 con 0x7fddfc077450 2024-04-26T23:45:54.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:54.843+0000 7fddf2a00700 1 -- 172.21.15.72:0/1210789390 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdde8003cc0 con 0x7fddfc077450 2024-04-26T23:45:54.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:54.843+0000 7fde00c00700 1 -- 172.21.15.72:0/1210789390 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddfc1a8060 con 0x7fddfc077450 2024-04-26T23:45:54.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:54.844+0000 7fde00c00700 1 -- 172.21.15.72:0/1210789390 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddfc06a110 con 0x7fddfc077450 2024-04-26T23:45:54.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:54.847+0000 7fddf2a00700 1 -- 172.21.15.72:0/1210789390 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fdde800c810 con 0x7fddfc077450 2024-04-26T23:45:54.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:54.848+0000 7fddf2a00700 1 --2- 172.21.15.72:0/1210789390 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fdde00a3cd0 0x7fdde00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:54.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:54.848+0000 7fddf2a00700 1 -- 172.21.15.72:0/1210789390 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7fdde80d0b60 con 0x7fddfc077450 2024-04-26T23:45:54.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:54.848+0000 7fddf9e00700 1 --2- 172.21.15.72:0/1210789390 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fdde00a3cd0 0x7fdde00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:54.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:54.849+0000 7fddf9e00700 1 --2- 172.21.15.72:0/1210789390 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fdde00a3cd0 0x7fdde00a6190 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fddfc077830 tx=0x7fddec009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:54.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:54.853+0000 7fddf2a00700 1 -- 172.21.15.72:0/1210789390 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fdde809cf90 con 0x7fddfc077450 2024-04-26T23:45:55.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:55.139+0000 7fde00c00700 1 -- 172.21.15.72:0/1210789390 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7fddfc02e490 con 0x7fddfc077450 2024-04-26T23:45:55.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:55.140+0000 7fddf2a00700 1 -- 172.21.15.72:0/1210789390 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 23 v23) ==== 76+0+3054 (secure 0 0 0) 0x7fddfc02e490 con 0x7fddfc077450 2024-04-26T23:45:55.145 INFO:teuthology.orchestra.run.smithi072.stdout:e23 2024-04-26T23:45:55.146 INFO:teuthology.orchestra.run.smithi072.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-26T23:45:55.146 INFO:teuthology.orchestra.run.smithi072.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-26T23:45:55.146 INFO:teuthology.orchestra.run.smithi072.stdout:legacy client fscid: 1 2024-04-26T23:45:55.146 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:45:55.146 INFO:teuthology.orchestra.run.smithi072.stdout:Filesystem 'cephfs' (1) 2024-04-26T23:45:55.146 INFO:teuthology.orchestra.run.smithi072.stdout:fs_name cephfs 2024-04-26T23:45:55.146 INFO:teuthology.orchestra.run.smithi072.stdout:epoch 23 2024-04-26T23:45:55.146 INFO:teuthology.orchestra.run.smithi072.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-26T23:45:55.146 INFO:teuthology.orchestra.run.smithi072.stdout:created 2024-04-26T23:43:02.928649+0000 2024-04-26T23:45:55.146 INFO:teuthology.orchestra.run.smithi072.stdout:modified 2024-04-26T23:43:30.190394+0000 2024-04-26T23:45:55.146 INFO:teuthology.orchestra.run.smithi072.stdout:tableserver 0 2024-04-26T23:45:55.146 INFO:teuthology.orchestra.run.smithi072.stdout:root 0 2024-04-26T23:45:55.146 INFO:teuthology.orchestra.run.smithi072.stdout:session_timeout 300 2024-04-26T23:45:55.147 INFO:teuthology.orchestra.run.smithi072.stdout:session_autoclose 300 2024-04-26T23:45:55.147 INFO:teuthology.orchestra.run.smithi072.stdout:max_file_size 1099511627776 2024-04-26T23:45:55.147 INFO:teuthology.orchestra.run.smithi072.stdout:max_xattr_size 65536 2024-04-26T23:45:55.147 INFO:teuthology.orchestra.run.smithi072.stdout:required_client_features {} 2024-04-26T23:45:55.147 INFO:teuthology.orchestra.run.smithi072.stdout:last_failure 0 2024-04-26T23:45:55.147 INFO:teuthology.orchestra.run.smithi072.stdout:last_failure_osd_epoch 0 2024-04-26T23:45:55.147 INFO:teuthology.orchestra.run.smithi072.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-26T23:45:55.147 INFO:teuthology.orchestra.run.smithi072.stdout:max_mds 3 2024-04-26T23:45:55.147 INFO:teuthology.orchestra.run.smithi072.stdout:in 0,1,2 2024-04-26T23:45:55.147 INFO:teuthology.orchestra.run.smithi072.stdout:up {0=24590,1=24571,2=24553} 2024-04-26T23:45:55.147 INFO:teuthology.orchestra.run.smithi072.stdout:failed 2024-04-26T23:45:55.147 INFO:teuthology.orchestra.run.smithi072.stdout:damaged 2024-04-26T23:45:55.147 INFO:teuthology.orchestra.run.smithi072.stdout:stopped 2024-04-26T23:45:55.147 INFO:teuthology.orchestra.run.smithi072.stdout:data_pools [3,4] 2024-04-26T23:45:55.147 INFO:teuthology.orchestra.run.smithi072.stdout:metadata_pool 2 2024-04-26T23:45:55.148 INFO:teuthology.orchestra.run.smithi072.stdout:inline_data disabled 2024-04-26T23:45:55.148 INFO:teuthology.orchestra.run.smithi072.stdout:balancer 2024-04-26T23:45:55.148 INFO:teuthology.orchestra.run.smithi072.stdout:bal_rank_mask -1 2024-04-26T23:45:55.148 INFO:teuthology.orchestra.run.smithi072.stdout:standby_count_wanted 0 2024-04-26T23:45:55.148 INFO:teuthology.orchestra.run.smithi072.stdout:qdb_cluster leader: 24590 members: 24553,24571,24590 2024-04-26T23:45:55.148 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.b{0:24590} state up:active seq 3 addr [v2:172.21.15.76:6838/3405829058,v1:172.21.15.76:6839/3405829058] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T23:45:55.148 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.i{0:24535} state up:standby-replay seq 1 addr [v2:172.21.15.136:6838/3938677630,v1:172.21.15.136:6839/3938677630] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T23:45:55.148 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.a{1:24571} state up:active seq 10 addr [v2:172.21.15.72:6840/1356345077,v1:172.21.15.72:6841/1356345077] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T23:45:55.148 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.h{1:24533} state up:standby-replay seq 1 addr [v2:172.21.15.76:6836/2506930945,v1:172.21.15.76:6837/2506930945] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T23:45:55.148 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.g{2:24553} state up:active seq 11 addr [v2:172.21.15.72:6838/1403162102,v1:172.21.15.72:6839/1403162102] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T23:45:55.148 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.c{2:24514} state up:standby-replay seq 1 addr [v2:172.21.15.136:6836/3639308555,v1:172.21.15.136:6837/3639308555] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T23:45:55.148 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:45:55.148 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:45:55.148 INFO:teuthology.orchestra.run.smithi072.stdout:Standby daemons: 2024-04-26T23:45:55.148 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:45:55.149 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.j{-1:14397} state up:standby seq 1 addr [v2:172.21.15.72:6834/2386456166,v1:172.21.15.72:6835/2386456166] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T23:45:55.149 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.l{-1:24445} state up:standby seq 1 addr [v2:172.21.15.136:6832/1528342053,v1:172.21.15.136:6833/1528342053] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T23:45:55.149 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.k{-1:24455} state up:standby seq 1 addr [v2:172.21.15.76:6832/2427238559,v1:172.21.15.76:6833/2427238559] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T23:45:55.149 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.d{-1:24491} state up:standby seq 1 addr [v2:172.21.15.72:6836/4050689894,v1:172.21.15.72:6837/4050689894] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T23:45:55.149 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.f{-1:24502} state up:standby seq 1 addr [v2:172.21.15.136:6834/3681925367,v1:172.21.15.136:6835/3681925367] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T23:45:55.149 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.e{-1:24503} state up:standby seq 1 addr [v2:172.21.15.76:6834/4251109085,v1:172.21.15.76:6835/4251109085] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T23:45:55.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:55.143+0000 7fde00c00700 1 -- 172.21.15.72:0/1210789390 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fdde00a3cd0 msgr2=0x7fdde00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:55.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:55.143+0000 7fde00c00700 1 --2- 172.21.15.72:0/1210789390 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fdde00a3cd0 0x7fdde00a6190 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fddfc077830 tx=0x7fddec009260 comp rx=0 tx=0).stop 2024-04-26T23:45:55.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:55.143+0000 7fde00c00700 1 -- 172.21.15.72:0/1210789390 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fddfc077450 msgr2=0x7fddfc1a0110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:55.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:55.143+0000 7fde00c00700 1 --2- 172.21.15.72:0/1210789390 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fddfc077450 0x7fddfc1a0110 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fdde800e9a0 tx=0x7fdde800ecb0 comp rx=0 tx=0).stop 2024-04-26T23:45:55.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:55.143+0000 7fde00c00700 1 -- 172.21.15.72:0/1210789390 shutdown_connections 2024-04-26T23:45:55.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:55.143+0000 7fde00c00700 1 --2- 172.21.15.72:0/1210789390 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fdde00a3cd0 0x7fdde00a6190 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:55.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:55.143+0000 7fde00c00700 1 --2- 172.21.15.72:0/1210789390 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fddfc10fa70 0x7fddfc1a7190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:55.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:55.143+0000 7fde00c00700 1 --2- 172.21.15.72:0/1210789390 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fddfc075e80 0x7fddfc19fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:55.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:55.143+0000 7fde00c00700 1 --2- 172.21.15.72:0/1210789390 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fddfc077450 0x7fddfc1a0110 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:55.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:55.143+0000 7fde00c00700 1 -- 172.21.15.72:0/1210789390 >> 172.21.15.72:0/1210789390 conn(0x7fddfc0fe9f0 msgr2=0x7fddfc102e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:55.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:55.144+0000 7fde00c00700 1 -- 172.21.15.72:0/1210789390 shutdown_connections 2024-04-26T23:45:55.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:55.144+0000 7fde00c00700 1 -- 172.21.15.72:0/1210789390 wait complete. 2024-04-26T23:45:55.152 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 23 2024-04-26T23:45:55.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:55 smithi136 ceph-mon[33050]: pgmap v411: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:45:55.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:55 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:45:55.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:55 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:45:55.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:55 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/1210789390' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-26T23:45:55.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:55 smithi076 ceph-mon[32774]: pgmap v411: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:45:55.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:55 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:45:55.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:55 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:45:55.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:55 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/1210789390' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-26T23:45:56.040 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:55 smithi072 ceph-mon[28184]: pgmap v411: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:45:56.040 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:55 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:45:56.040 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:55 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:45:56.040 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:55 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/1210789390' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-26T23:45:56.102 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-26T23:45:56.520 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:45:56.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:56 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:45:56.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:56 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:45:56.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:56 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:45:56.854 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:56 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:45:56.854 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:56 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:45:56.854 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:56 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:45:56.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:56 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:45:56.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:56 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:45:56.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:56 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:45:57.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:57 smithi072 ceph-mon[28184]: pgmap v412: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:45:57.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:57 smithi076 ceph-mon[32774]: pgmap v412: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:45:58.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:57 smithi136 ceph-mon[33050]: pgmap v412: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:45:58.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:58.856+0000 7ff341c00700 1 -- 172.21.15.72:0/3756363093 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff33c077450 msgr2=0x7ff33c0758b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:58.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:58.856+0000 7ff341c00700 1 --2- 172.21.15.72:0/3756363093 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff33c077450 0x7ff33c0758b0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff32c009990 tx=0x7ff32c009ca0 comp rx=0 tx=0).stop 2024-04-26T23:45:58.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:58.856+0000 7ff341c00700 1 -- 172.21.15.72:0/3756363093 shutdown_connections 2024-04-26T23:45:58.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:58.856+0000 7ff341c00700 1 --2- 172.21.15.72:0/3756363093 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff33c10b360 0x7ff33c10d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:58.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:58.856+0000 7ff341c00700 1 --2- 172.21.15.72:0/3756363093 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff33c077450 0x7ff33c0758b0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:58.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:58.856+0000 7ff341c00700 1 --2- 172.21.15.72:0/3756363093 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff33c075e80 0x7ff33c076300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:58.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:58.856+0000 7ff341c00700 1 -- 172.21.15.72:0/3756363093 >> 172.21.15.72:0/3756363093 conn(0x7ff33c0fe9f0 msgr2=0x7ff33c100e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:58.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:58.856+0000 7ff341c00700 1 -- 172.21.15.72:0/3756363093 shutdown_connections 2024-04-26T23:45:58.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:58.856+0000 7ff341c00700 1 -- 172.21.15.72:0/3756363093 wait complete. 2024-04-26T23:45:58.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:58.857+0000 7ff341c00700 1 Processor -- start 2024-04-26T23:45:58.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:58.857+0000 7ff341c00700 1 -- start start 2024-04-26T23:45:58.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:58.858+0000 7ff341c00700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff33c075e80 0x7ff33c19b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:58.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:58.858+0000 7ff341c00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff33c077450 0x7ff33c19bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:58.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:58.858+0000 7ff341c00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff33c10b360 0x7ff33c1a2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:58.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:58.858+0000 7ff341c00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7ff33c111160 con 0x7ff33c10b360 2024-04-26T23:45:58.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:58.858+0000 7ff341c00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff33c110fe0 con 0x7ff33c077450 2024-04-26T23:45:58.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:58.858+0000 7ff341c00700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff33c1112e0 con 0x7ff33c075e80 2024-04-26T23:45:58.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:58.858+0000 7ff33b400700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff33c10b360 0x7ff33c1a2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:58.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:58.858+0000 7ff33aa00700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff33c075e80 0x7ff33c19b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:58.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:58.858+0000 7ff332000700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff33c077450 0x7ff33c19bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:58.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:58.858+0000 7ff33b400700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff33c10b360 0x7ff33c1a2e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57878/0 (socket says 172.21.15.72:57878) 2024-04-26T23:45:58.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:58.858+0000 7ff33b400700 1 -- 172.21.15.72:0/3937876694 learned_addr learned my addr 172.21.15.72:0/3937876694 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:45:58.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:58.859+0000 7ff332000700 1 -- 172.21.15.72:0/3937876694 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff33c075e80 msgr2=0x7ff33c19b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:58.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:58.859+0000 7ff332000700 1 --2- 172.21.15.72:0/3937876694 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff33c075e80 0x7ff33c19b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:58.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:58.859+0000 7ff332000700 1 -- 172.21.15.72:0/3937876694 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff33c10b360 msgr2=0x7ff33c1a2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:58.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:58.859+0000 7ff332000700 1 --2- 172.21.15.72:0/3937876694 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff33c10b360 0x7ff33c1a2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:58.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:58.859+0000 7ff332000700 1 -- 172.21.15.72:0/3937876694 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff33c1a34b0 con 0x7ff33c077450 2024-04-26T23:45:58.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:58.859+0000 7ff33aa00700 1 --2- 172.21.15.72:0/3937876694 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff33c075e80 0x7ff33c19b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:45:58.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:58.859+0000 7ff332000700 1 --2- 172.21.15.72:0/3937876694 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff33c077450 0x7ff33c19bd90 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff32400e9c0 tx=0x7ff32400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:58.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:58.859+0000 7ff333e00700 1 -- 172.21.15.72:0/3937876694 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff324004220 con 0x7ff33c077450 2024-04-26T23:45:58.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:58.859+0000 7ff333e00700 1 -- 172.21.15.72:0/3937876694 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3240043c0 con 0x7ff33c077450 2024-04-26T23:45:58.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:58.859+0000 7ff341c00700 1 -- 172.21.15.72:0/3937876694 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff33c1a37a0 con 0x7ff33c077450 2024-04-26T23:45:58.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:58.859+0000 7ff333e00700 1 -- 172.21.15.72:0/3937876694 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff324003c30 con 0x7ff33c077450 2024-04-26T23:45:58.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:58.859+0000 7ff341c00700 1 -- 172.21.15.72:0/3937876694 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff33c1a3ce0 con 0x7ff33c077450 2024-04-26T23:45:58.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:58.861+0000 7ff341c00700 1 -- 172.21.15.72:0/3937876694 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff33c06a110 con 0x7ff33c077450 2024-04-26T23:45:58.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:58.862+0000 7ff333e00700 1 -- 172.21.15.72:0/3937876694 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff32400c7c0 con 0x7ff33c077450 2024-04-26T23:45:58.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:58.864+0000 7ff333e00700 1 --2- 172.21.15.72:0/3937876694 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff3200a3cd0 0x7ff3200a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:45:58.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:58.864+0000 7ff333e00700 1 -- 172.21.15.72:0/3937876694 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7ff3240d0bd0 con 0x7ff33c077450 2024-04-26T23:45:58.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:58.864+0000 7ff33aa00700 1 --2- 172.21.15.72:0/3937876694 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff3200a3cd0 0x7ff3200a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:45:58.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:58.865+0000 7ff33aa00700 1 --2- 172.21.15.72:0/3937876694 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff3200a3cd0 0x7ff3200a6190 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ff32c00b2e0 tx=0x7ff32c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:45:58.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:58.868+0000 7ff333e00700 1 -- 172.21.15.72:0/3937876694 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff32409d000 con 0x7ff33c077450 2024-04-26T23:45:59.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:59.095+0000 7ff341c00700 1 -- 172.21.15.72:0/3937876694 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7ff33c02e490 con 0x7ff33c077450 2024-04-26T23:45:59.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:59.097+0000 7ff333e00700 1 -- 172.21.15.72:0/3937876694 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v93) ==== 56+0+5715 (secure 0 0 0) 0x7ff32409c900 con 0x7ff33c077450 2024-04-26T23:45:59.103 INFO:teuthology.orchestra.run.smithi072.stdout:epoch 93 2024-04-26T23:45:59.103 INFO:teuthology.orchestra.run.smithi072.stdout:fsid 3c8453ae-0425-11ef-bc93-c7b262605968 2024-04-26T23:45:59.103 INFO:teuthology.orchestra.run.smithi072.stdout:created 2024-04-26T23:33:48.088300+0000 2024-04-26T23:45:59.103 INFO:teuthology.orchestra.run.smithi072.stdout:modified 2024-04-26T23:45:00.532006+0000 2024-04-26T23:45:59.103 INFO:teuthology.orchestra.run.smithi072.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-26T23:45:59.103 INFO:teuthology.orchestra.run.smithi072.stdout:crush_version 27 2024-04-26T23:45:59.103 INFO:teuthology.orchestra.run.smithi072.stdout:full_ratio 0.95 2024-04-26T23:45:59.103 INFO:teuthology.orchestra.run.smithi072.stdout:backfillfull_ratio 0.9 2024-04-26T23:45:59.103 INFO:teuthology.orchestra.run.smithi072.stdout:nearfull_ratio 0.85 2024-04-26T23:45:59.103 INFO:teuthology.orchestra.run.smithi072.stdout:require_min_compat_client luminous 2024-04-26T23:45:59.103 INFO:teuthology.orchestra.run.smithi072.stdout:min_compat_client luminous 2024-04-26T23:45:59.103 INFO:teuthology.orchestra.run.smithi072.stdout:require_osd_release squid 2024-04-26T23:45:59.103 INFO:teuthology.orchestra.run.smithi072.stdout:stretch_mode_enabled false 2024-04-26T23:45:59.104 INFO:teuthology.orchestra.run.smithi072.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-26T23:45:59.104 INFO:teuthology.orchestra.run.smithi072.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-04-26T23:45:59.104 INFO:teuthology.orchestra.run.smithi072.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-04-26T23:45:59.104 INFO:teuthology.orchestra.run.smithi072.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 89 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-04-26T23:45:59.104 INFO:teuthology.orchestra.run.smithi072.stdout:max_osd 12 2024-04-26T23:45:59.104 INFO:teuthology.orchestra.run.smithi072.stdout:osd.0 up in weight 1 up_from 8 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.72:6802/1639298875,v1:172.21.15.72:6803/1639298875] [v2:172.21.15.72:6804/1639298875,v1:172.21.15.72:6805/1639298875] exists,up e20c9ebf-56f6-47c1-a11b-e1e615af042b 2024-04-26T23:45:59.104 INFO:teuthology.orchestra.run.smithi072.stdout:osd.1 up in weight 1 up_from 13 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.76:6800/4185342193,v1:172.21.15.76:6801/4185342193] [v2:172.21.15.76:6802/4185342193,v1:172.21.15.76:6803/4185342193] exists,up e6cac469-780c-4d66-b560-b686072371fb 2024-04-26T23:45:59.104 INFO:teuthology.orchestra.run.smithi072.stdout:osd.2 up in weight 1 up_from 18 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.136:6800/3033860799,v1:172.21.15.136:6801/3033860799] [v2:172.21.15.136:6802/3033860799,v1:172.21.15.136:6803/3033860799] exists,up 838a49fc-18de-446c-8336-67b3e05da231 2024-04-26T23:45:59.104 INFO:teuthology.orchestra.run.smithi072.stdout:osd.3 up in weight 1 up_from 25 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.72:6810/2071107815,v1:172.21.15.72:6811/2071107815] [v2:172.21.15.72:6812/2071107815,v1:172.21.15.72:6813/2071107815] exists,up e6da89be-5ba7-48a4-9306-f75498660e0e 2024-04-26T23:45:59.104 INFO:teuthology.orchestra.run.smithi072.stdout:osd.4 up in weight 1 up_from 33 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.76:6808/147449238,v1:172.21.15.76:6809/147449238] [v2:172.21.15.76:6810/147449238,v1:172.21.15.76:6811/147449238] exists,up 0b17c491-d798-488f-892d-521ef2fbc51e 2024-04-26T23:45:59.104 INFO:teuthology.orchestra.run.smithi072.stdout:osd.5 up in weight 1 up_from 38 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.136:6808/2320149212,v1:172.21.15.136:6809/2320149212] [v2:172.21.15.136:6810/2320149212,v1:172.21.15.136:6811/2320149212] exists,up c7e887b3-ab38-4608-9bfd-1ed49956e8af 2024-04-26T23:45:59.104 INFO:teuthology.orchestra.run.smithi072.stdout:osd.6 up in weight 1 up_from 46 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.72:6818/389499207,v1:172.21.15.72:6819/389499207] [v2:172.21.15.72:6820/389499207,v1:172.21.15.72:6821/389499207] exists,up f0bd604b-0f57-4a38-9295-eb200264f1ff 2024-04-26T23:45:59.104 INFO:teuthology.orchestra.run.smithi072.stdout:osd.7 up in weight 1 up_from 51 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.76:6816/375888575,v1:172.21.15.76:6817/375888575] [v2:172.21.15.76:6818/375888575,v1:172.21.15.76:6819/375888575] exists,up 1879d4e4-038e-4cd6-a3ec-3a940bcc3797 2024-04-26T23:45:59.104 INFO:teuthology.orchestra.run.smithi072.stdout:osd.8 up in weight 1 up_from 59 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.136:6816/3446774052,v1:172.21.15.136:6817/3446774052] [v2:172.21.15.136:6818/3446774052,v1:172.21.15.136:6819/3446774052] exists,up 6f404062-97e7-4ccb-a582-76dc1ffae548 2024-04-26T23:45:59.105 INFO:teuthology.orchestra.run.smithi072.stdout:osd.9 up in weight 1 up_from 67 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.72:6826/3423789152,v1:172.21.15.72:6827/3423789152] [v2:172.21.15.72:6828/3423789152,v1:172.21.15.72:6829/3423789152] exists,up 2989ca68-3164-4667-9bcf-230dcbb7b8f7 2024-04-26T23:45:59.105 INFO:teuthology.orchestra.run.smithi072.stdout:osd.10 up in weight 1 up_from 72 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.76:6824/1120991574,v1:172.21.15.76:6825/1120991574] [v2:172.21.15.76:6826/1120991574,v1:172.21.15.76:6827/1120991574] exists,up 9ec6d46a-0913-4208-97b5-167188ee8cb3 2024-04-26T23:45:59.105 INFO:teuthology.orchestra.run.smithi072.stdout:osd.11 up in weight 1 up_from 80 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.136:6824/1135995193,v1:172.21.15.136:6825/1135995193] [v2:172.21.15.136:6826/1135995193,v1:172.21.15.136:6827/1135995193] exists,up bdd0774f-6e58-4918-b4cc-afd4f1118d3f 2024-04-26T23:45:59.105 INFO:teuthology.orchestra.run.smithi072.stdout:pg_upmap_items 2.a [2,11] 2024-04-26T23:45:59.105 INFO:teuthology.orchestra.run.smithi072.stdout:pg_upmap_items 2.f [8,11] 2024-04-26T23:45:59.105 INFO:teuthology.orchestra.run.smithi072.stdout:pg_upmap_items 2.10 [2,4] 2024-04-26T23:45:59.105 INFO:teuthology.orchestra.run.smithi072.stdout:pg_upmap_items 3.e [4,8] 2024-04-26T23:45:59.105 INFO:teuthology.orchestra.run.smithi072.stdout:pg_upmap_items 3.10 [5,9] 2024-04-26T23:45:59.105 INFO:teuthology.orchestra.run.smithi072.stdout:pg_upmap_items 3.18 [3,9] 2024-04-26T23:45:59.105 INFO:teuthology.orchestra.run.smithi072.stdout:pg_upmap_items 3.1b [4,6] 2024-04-26T23:45:59.105 INFO:teuthology.orchestra.run.smithi072.stdout:pg_upmap_items 3.21 [4,8] 2024-04-26T23:45:59.105 INFO:teuthology.orchestra.run.smithi072.stdout:pg_upmap_items 3.22 [3,9] 2024-04-26T23:45:59.105 INFO:teuthology.orchestra.run.smithi072.stdout:pg_upmap_items 3.27 [4,6] 2024-04-26T23:45:59.105 INFO:teuthology.orchestra.run.smithi072.stdout:pg_upmap_items 3.37 [5,7] 2024-04-26T23:45:59.105 INFO:teuthology.orchestra.run.smithi072.stdout:pg_upmap_items 4.2 [9,1] 2024-04-26T23:45:59.106 INFO:teuthology.orchestra.run.smithi072.stdout:pg_upmap_items 4.6 [9,11] 2024-04-26T23:45:59.106 INFO:teuthology.orchestra.run.smithi072.stdout:pg_upmap_items 4.e [5,8] 2024-04-26T23:45:59.106 INFO:teuthology.orchestra.run.smithi072.stdout:pg_upmap_items 4.16 [9,8] 2024-04-26T23:45:59.106 INFO:teuthology.orchestra.run.smithi072.stdout:blocklist 172.21.15.72:0/3625071330 expires 2024-04-27T23:34:58.266218+0000 2024-04-26T23:45:59.106 INFO:teuthology.orchestra.run.smithi072.stdout:blocklist 172.21.15.72:6801/2903979134 expires 2024-04-27T23:34:58.266218+0000 2024-04-26T23:45:59.106 INFO:teuthology.orchestra.run.smithi072.stdout:blocklist 172.21.15.72:6800/2903979134 expires 2024-04-27T23:34:58.266218+0000 2024-04-26T23:45:59.106 INFO:teuthology.orchestra.run.smithi072.stdout:blocklist 172.21.15.72:0/1519142576 expires 2024-04-27T23:34:58.266218+0000 2024-04-26T23:45:59.106 INFO:teuthology.orchestra.run.smithi072.stdout:blocklist 172.21.15.72:0/3669626460 expires 2024-04-27T23:34:24.233319+0000 2024-04-26T23:45:59.106 INFO:teuthology.orchestra.run.smithi072.stdout:blocklist 172.21.15.72:0/945623297 expires 2024-04-27T23:34:24.233319+0000 2024-04-26T23:45:59.106 INFO:teuthology.orchestra.run.smithi072.stdout:blocklist 172.21.15.72:0/3479150073 expires 2024-04-27T23:34:24.233319+0000 2024-04-26T23:45:59.106 INFO:teuthology.orchestra.run.smithi072.stdout:blocklist 172.21.15.72:6800/4175896290 expires 2024-04-27T23:34:24.233319+0000 2024-04-26T23:45:59.106 INFO:teuthology.orchestra.run.smithi072.stdout:blocklist 172.21.15.72:6801/4175896290 expires 2024-04-27T23:34:24.233319+0000 2024-04-26T23:45:59.106 INFO:teuthology.orchestra.run.smithi072.stdout:blocklist 172.21.15.72:0/3783175012 expires 2024-04-27T23:34:58.266218+0000 2024-04-26T23:45:59.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:59.100+0000 7ff341c00700 1 -- 172.21.15.72:0/3937876694 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff3200a3cd0 msgr2=0x7ff3200a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:59.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:59.100+0000 7ff341c00700 1 --2- 172.21.15.72:0/3937876694 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff3200a3cd0 0x7ff3200a6190 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ff32c00b2e0 tx=0x7ff32c005fb0 comp rx=0 tx=0).stop 2024-04-26T23:45:59.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:59.100+0000 7ff341c00700 1 -- 172.21.15.72:0/3937876694 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff33c077450 msgr2=0x7ff33c19bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:45:59.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:59.100+0000 7ff341c00700 1 --2- 172.21.15.72:0/3937876694 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff33c077450 0x7ff33c19bd90 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff32400e9c0 tx=0x7ff32400ecd0 comp rx=0 tx=0).stop 2024-04-26T23:45:59.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:59.100+0000 7ff341c00700 1 -- 172.21.15.72:0/3937876694 shutdown_connections 2024-04-26T23:45:59.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:59.100+0000 7ff341c00700 1 --2- 172.21.15.72:0/3937876694 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff3200a3cd0 0x7ff3200a6190 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:59.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:59.100+0000 7ff341c00700 1 --2- 172.21.15.72:0/3937876694 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff33c10b360 0x7ff33c1a2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:59.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:59.100+0000 7ff341c00700 1 --2- 172.21.15.72:0/3937876694 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff33c075e80 0x7ff33c19b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:59.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:59.100+0000 7ff341c00700 1 --2- 172.21.15.72:0/3937876694 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff33c077450 0x7ff33c19bd90 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:45:59.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:59.100+0000 7ff341c00700 1 -- 172.21.15.72:0/3937876694 >> 172.21.15.72:0/3937876694 conn(0x7ff33c0fe9f0 msgr2=0x7ff33c107260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:45:59.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:59.100+0000 7ff341c00700 1 -- 172.21.15.72:0/3937876694 shutdown_connections 2024-04-26T23:45:59.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:45:59.100+0000 7ff341c00700 1 -- 172.21.15.72:0/3937876694 wait complete. 2024-04-26T23:45:59.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:59 smithi072 ceph-mon[28184]: pgmap v413: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:45:59.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:45:59 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/3937876694' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-26T23:45:59.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:59 smithi076 ceph-mon[32774]: pgmap v413: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:45:59.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:45:59 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/3937876694' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-26T23:46:00.024 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-26T23:46:00.035 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:46:00.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:59 smithi136 ceph-mon[33050]: pgmap v413: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:46:00.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:45:59 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/3937876694' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-26T23:46:00.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.243+0000 7f249e800640 1 -- 172.21.15.72:0/177608385 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2498105b70 msgr2=0x7f2498105ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:00.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.243+0000 7f249e800640 1 --2- 172.21.15.72:0/177608385 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2498105b70 0x7f2498105ff0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2484009900 tx=0x7f248402a3f0 comp rx=0 tx=0).stop 2024-04-26T23:46:00.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.243+0000 7f249e800640 1 -- 172.21.15.72:0/177608385 shutdown_connections 2024-04-26T23:46:00.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.243+0000 7f249e800640 1 --2- 172.21.15.72:0/177608385 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2498106530 0x7f249810cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:00.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.243+0000 7f249e800640 1 --2- 172.21.15.72:0/177608385 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2498105b70 0x7f2498105ff0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:00.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.243+0000 7f249e800640 1 --2- 172.21.15.72:0/177608385 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2498104980 0x7f2498104da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:00.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.243+0000 7f249e800640 1 -- 172.21.15.72:0/177608385 >> 172.21.15.72:0/177608385 conn(0x7f24980fff10 msgr2=0x7f2498102330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:46:00.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.243+0000 7f249e800640 1 -- 172.21.15.72:0/177608385 shutdown_connections 2024-04-26T23:46:00.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.243+0000 7f249e800640 1 -- 172.21.15.72:0/177608385 wait complete. 2024-04-26T23:46:00.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.244+0000 7f249e800640 1 Processor -- start 2024-04-26T23:46:00.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.244+0000 7f249e800640 1 -- start start 2024-04-26T23:46:00.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.244+0000 7f249e800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2498104980 0x7f249819abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:00.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.244+0000 7f249e800640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2498105b70 0x7f249819b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:00.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.244+0000 7f249e800640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2498106530 0x7f24981a2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:00.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.244+0000 7f249e800640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f24981104b0 con 0x7f2498105b70 2024-04-26T23:46:00.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.244+0000 7f249e800640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2498110330 con 0x7f2498104980 2024-04-26T23:46:00.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.244+0000 7f249e800640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f2498110630 con 0x7f2498106530 2024-04-26T23:46:00.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.245+0000 7f249ca00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2498105b70 0x7f249819b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:00.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.245+0000 7f249d400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2498104980 0x7f249819abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:00.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.245+0000 7f249de00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2498106530 0x7f24981a2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:00.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.245+0000 7f249ca00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2498105b70 0x7f249819b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57882/0 (socket says 172.21.15.72:57882) 2024-04-26T23:46:00.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.245+0000 7f249ca00640 1 -- 172.21.15.72:0/465701562 learned_addr learned my addr 172.21.15.72:0/465701562 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:46:00.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.245+0000 7f249de00640 1 -- 172.21.15.72:0/465701562 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2498104980 msgr2=0x7f249819abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:00.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.245+0000 7f249de00640 1 --2- 172.21.15.72:0/465701562 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2498104980 0x7f249819abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:00.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.245+0000 7f249de00640 1 -- 172.21.15.72:0/465701562 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2498105b70 msgr2=0x7f249819b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:00.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.245+0000 7f249de00640 1 --2- 172.21.15.72:0/465701562 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2498105b70 0x7f249819b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:00.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.245+0000 7f249de00640 1 -- 172.21.15.72:0/465701562 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24981a27d0 con 0x7f2498106530 2024-04-26T23:46:00.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.246+0000 7f249de00640 1 --2- 172.21.15.72:0/465701562 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2498106530 0x7f24981a2160 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2490009fd0 tx=0x7f249000ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:46:00.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.246+0000 7f2496000640 1 -- 172.21.15.72:0/465701562 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f249000ce20 con 0x7f2498106530 2024-04-26T23:46:00.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.246+0000 7f249e800640 1 -- 172.21.15.72:0/465701562 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24981a2ac0 con 0x7f2498106530 2024-04-26T23:46:00.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.246+0000 7f2496000640 1 -- 172.21.15.72:0/465701562 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2490005020 con 0x7f2498106530 2024-04-26T23:46:00.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.246+0000 7f2496000640 1 -- 172.21.15.72:0/465701562 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2490010690 con 0x7f2498106530 2024-04-26T23:46:00.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.246+0000 7f249e800640 1 -- 172.21.15.72:0/465701562 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24981a2fe0 con 0x7f2498106530 2024-04-26T23:46:00.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.247+0000 7f249e800640 1 -- 172.21.15.72:0/465701562 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f249810bcc0 con 0x7f2498106530 2024-04-26T23:46:00.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.249+0000 7f2496000640 1 -- 172.21.15.72:0/465701562 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f24900054d0 con 0x7f2498106530 2024-04-26T23:46:00.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.251+0000 7f2496000640 1 --2- 172.21.15.72:0/465701562 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f24800a3c40 0x7f24800a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:00.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.251+0000 7f2496000640 1 -- 172.21.15.72:0/465701562 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f24900d1450 con 0x7f2498106530 2024-04-26T23:46:00.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.251+0000 7f249d400640 1 --2- 172.21.15.72:0/465701562 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f24800a3c40 0x7f24800a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:00.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.252+0000 7f249d400640 1 --2- 172.21.15.72:0/465701562 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f24800a3c40 0x7f24800a6100 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f248c005fd0 tx=0x7f248c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:46:00.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.253+0000 7f2496000640 1 -- 172.21.15.72:0/465701562 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f249009d880 con 0x7f2498106530 2024-04-26T23:46:00.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.456+0000 7f249e800640 1 -- 172.21.15.72:0/465701562 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2498063530 con 0x7f2498106530 2024-04-26T23:46:00.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.459+0000 7f2496000640 1 -- 172.21.15.72:0/465701562 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24841 (secure 0 0 0) 0x7f249009d180 con 0x7f2498106530 2024-04-26T23:46:00.463 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:46:00.464 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":93,"fsid":"3c8453ae-0425-11ef-bc93-c7b262605968","created":"2024-04-26T23:33:48.088300+0000","modified":"2024-04-26T23:45:00.532006+0000","last_up_change":"2024-04-26T23:42:04.482891+0000","last_in_change":"2024-04-26T23:41:44.784415+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:37:44.428727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:43:00.431829+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:43:01.872977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:43:05.512050+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e20c9ebf-56f6-47c1-a11b-e1e615af042b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6803","nonce":1639298875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6805","nonce":1639298875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6809","nonce":1639298875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6807","nonce":1639298875}]},"public_addr":"172.21.15.72:6803/1639298875","cluster_addr":"172.21.15.72:6805/1639298875","heartbeat_back_addr":"172.21.15.72:6809/1639298875","heartbeat_front_addr":"172.21.15.72:6807/1639298875","state":["exists","up"]},{"osd":1,"uuid":"e6cac469-780c-4d66-b560-b686072371fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6801","nonce":4185342193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6803","nonce":4185342193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6807","nonce":4185342193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6805","nonce":4185342193}]},"public_addr":"172.21.15.76:6801/4185342193","cluster_addr":"172.21.15.76:6803/4185342193","heartbeat_back_addr":"172.21.15.76:6807/4185342193","heartbeat_front_addr":"172.21.15.76:6805/4185342193","state":["exists","up"]},{"osd":2,"uuid":"838a49fc-18de-446c-8336-67b3e05da231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6801","nonce":3033860799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6803","nonce":3033860799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6807","nonce":3033860799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6805","nonce":3033860799}]},"public_addr":"172.21.15.136:6801/3033860799","cluster_addr":"172.21.15.136:6803/3033860799","heartbeat_back_addr":"172.21.15.136:6807/3033860799","heartbeat_front_addr":"172.21.15.136:6805/3033860799","state":["exists","up"]},{"osd":3,"uuid":"e6da89be-5ba7-48a4-9306-f75498660e0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6811","nonce":2071107815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6813","nonce":2071107815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6817","nonce":2071107815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6815","nonce":2071107815}]},"public_addr":"172.21.15.72:6811/2071107815","cluster_addr":"172.21.15.72:6813/2071107815","heartbeat_back_addr":"172.21.15.72:6817/2071107815","heartbeat_front_addr":"172.21.15.72:6815/2071107815","state":["exists","up"]},{"osd":4,"uuid":"0b17c491-d798-488f-892d-521ef2fbc51e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6809","nonce":147449238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6811","nonce":147449238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6815","nonce":147449238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6813","nonce":147449238}]},"public_addr":"172.21.15.76:6809/147449238","cluster_addr":"172.21.15.76:6811/147449238","heartbeat_back_addr":"172.21.15.76:6815/147449238","heartbeat_front_addr":"172.21.15.76:6813/147449238","state":["exists","up"]},{"osd":5,"uuid":"c7e887b3-ab38-4608-9bfd-1ed49956e8af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6809","nonce":2320149212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6811","nonce":2320149212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6815","nonce":2320149212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6813","nonce":2320149212}]},"public_addr":"172.21.15.136:6809/2320149212","cluster_addr":"172.21.15.136:6811/2320149212","heartbeat_back_addr":"172.21.15.136:6815/2320149212","heartbeat_front_addr":"172.21.15.136:6813/2320149212","state":["exists","up"]},{"osd":6,"uuid":"f0bd604b-0f57-4a38-9295-eb200264f1ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6819","nonce":389499207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6821","nonce":389499207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6825","nonce":389499207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6823","nonce":389499207}]},"public_addr":"172.21.15.72:6819/389499207","cluster_addr":"172.21.15.72:6821/389499207","heartbeat_back_addr":"172.21.15.72:6825/389499207","heartbeat_front_addr":"172.21.15.72:6823/389499207","state":["exists","up"]},{"osd":7,"uuid":"1879d4e4-038e-4cd6-a3ec-3a940bcc3797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6817","nonce":375888575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6819","nonce":375888575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6823","nonce":375888575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6821","nonce":375888575}]},"public_addr":"172.21.15.76:6817/375888575","cluster_addr":"172.21.15.76:6819/375888575","heartbeat_back_addr":"172.21.15.76:6823/375888575","heartbeat_front_addr":"172.21.15.76:6821/375888575","state":["exists","up"]},{"osd":8,"uuid":"6f404062-97e7-4ccb-a582-76dc1ffae548","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6817","nonce":3446774052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6819","nonce":3446774052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6823","nonce":3446774052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6821","nonce":3446774052}]},"public_addr":"172.21.15.136:6817/3446774052","cluster_addr":"172.21.15.136:6819/3446774052","heartbeat_back_addr":"172.21.15.136:6823/3446774052","heartbeat_front_addr":"172.21.15.136:6821/3446774052","state":["exists","up"]},{"osd":9,"uuid":"2989ca68-3164-4667-9bcf-230dcbb7b8f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6827","nonce":3423789152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6829","nonce":3423789152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6832","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6833","nonce":3423789152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6831","nonce":3423789152}]},"public_addr":"172.21.15.72:6827/3423789152","cluster_addr":"172.21.15.72:6829/3423789152","heartbeat_back_addr":"172.21.15.72:6833/3423789152","heartbeat_front_addr":"172.21.15.72:6831/3423789152","state":["exists","up"]},{"osd":10,"uuid":"9ec6d46a-0913-4208-97b5-167188ee8cb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6825","nonce":1120991574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6827","nonce":1120991574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6831","nonce":1120991574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6829","nonce":1120991574}]},"public_addr":"172.21.15.76:6825/1120991574","cluster_addr":"172.21.15.76:6827/1120991574","heartbeat_back_addr":"172.21.15.76:6831/1120991574","heartbeat_front_addr":"172.21.15.76:6829/1120991574","state":["exists","up"]},{"osd":11,"uuid":"bdd0774f-6e58-4918-b4cc-afd4f1118d3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6825","nonce":1135995193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6827","nonce":1135995193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6831","nonce":1135995193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6829","nonce":1135995193}]},"public_addr":"172.21.15.136:6825/1135995193","cluster_addr":"172.21.15.136:6827/1135995193","heartbeat_back_addr":"172.21.15.136:6831/1135995193","heartbeat_front_addr":"172.21.15.136:6829/1135995193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:36:52.128788+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:37:17.523654+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:37:41.634677+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:38:15.060797+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:38:41.234111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:39:06.783560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:39:40.884386+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:40:08.340465+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:40:34.344858+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:41:08.525309+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:41:36.782624+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:42:02.756888+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":6}]},{"pgid":"3.37","mappings":[{"from":5,"to":7}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:0/3625071330":"2024-04-27T23:34:58.266218+0000","172.21.15.72:6801/2903979134":"2024-04-27T23:34:58.266218+0000","172.21.15.72:6800/2903979134":"2024-04-27T23:34:58.266218+0000","172.21.15.72:0/1519142576":"2024-04-27T23:34:58.266218+0000","172.21.15.72:0/3669626460":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/945623297":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/3479150073":"2024-04-27T23:34:24.233319+0000","172.21.15.72:6800/4175896290":"2024-04-27T23:34:24.233319+0000","172.21.15.72:6801/4175896290":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/3783175012":"2024-04-27T23:34:58.266218+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:46:00.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.461+0000 7f249e800640 1 -- 172.21.15.72:0/465701562 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f24800a3c40 msgr2=0x7f24800a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:00.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.461+0000 7f249e800640 1 --2- 172.21.15.72:0/465701562 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f24800a3c40 0x7f24800a6100 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f248c005fd0 tx=0x7f248c00b410 comp rx=0 tx=0).stop 2024-04-26T23:46:00.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.461+0000 7f249e800640 1 -- 172.21.15.72:0/465701562 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2498106530 msgr2=0x7f24981a2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:00.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.461+0000 7f249e800640 1 --2- 172.21.15.72:0/465701562 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2498106530 0x7f24981a2160 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2490009fd0 tx=0x7f249000ef10 comp rx=0 tx=0).stop 2024-04-26T23:46:00.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.462+0000 7f249e800640 1 -- 172.21.15.72:0/465701562 shutdown_connections 2024-04-26T23:46:00.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.462+0000 7f249e800640 1 --2- 172.21.15.72:0/465701562 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f24800a3c40 0x7f24800a6100 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:00.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.462+0000 7f249e800640 1 --2- 172.21.15.72:0/465701562 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2498106530 0x7f24981a2160 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:00.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.462+0000 7f249e800640 1 --2- 172.21.15.72:0/465701562 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2498105b70 0x7f249819b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:00.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.462+0000 7f249e800640 1 --2- 172.21.15.72:0/465701562 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2498104980 0x7f249819abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:00.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.462+0000 7f249e800640 1 -- 172.21.15.72:0/465701562 >> 172.21.15.72:0/465701562 conn(0x7f24980fff10 msgr2=0x7f2498108e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:46:00.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.462+0000 7f249e800640 1 -- 172.21.15.72:0/465701562 shutdown_connections 2024-04-26T23:46:00.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.462+0000 7f249e800640 1 -- 172.21.15.72:0/465701562 wait complete. 2024-04-26T23:46:00.487 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:37:44.428727+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T23:43:00.431829+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T23:43:01.872977+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T23:43:05.512050+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-26T23:46:00.488 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:46:00.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.687+0000 7f9ebb400640 1 -- 172.21.15.72:0/441002936 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9eb410a840 msgr2=0x7f9eb410ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:00.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.687+0000 7f9ebb400640 1 --2- 172.21.15.72:0/441002936 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9eb410a840 0x7f9eb410ccd0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9eac009ca0 tx=0x7f9eac02a8c0 comp rx=0 tx=0).stop 2024-04-26T23:46:00.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.687+0000 7f9ebb400640 1 -- 172.21.15.72:0/441002936 shutdown_connections 2024-04-26T23:46:00.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.687+0000 7f9ebb400640 1 --2- 172.21.15.72:0/441002936 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9eb410a840 0x7f9eb410ccd0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:00.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.687+0000 7f9ebb400640 1 --2- 172.21.15.72:0/441002936 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9eb4075390 0x7f9eb4075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:00.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.687+0000 7f9ebb400640 1 --2- 172.21.15.72:0/441002936 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9eb4076970 0x7f9eb4074dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:00.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.687+0000 7f9ebb400640 1 -- 172.21.15.72:0/441002936 >> 172.21.15.72:0/441002936 conn(0x7f9eb40fdf10 msgr2=0x7f9eb4100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:46:00.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.687+0000 7f9ebb400640 1 -- 172.21.15.72:0/441002936 shutdown_connections 2024-04-26T23:46:00.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.687+0000 7f9ebb400640 1 -- 172.21.15.72:0/441002936 wait complete. 2024-04-26T23:46:00.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.688+0000 7f9ebb400640 1 Processor -- start 2024-04-26T23:46:00.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.689+0000 7f9ebb400640 1 -- start start 2024-04-26T23:46:00.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.689+0000 7f9ebb400640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9eb4075390 0x7f9eb419ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:00.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.689+0000 7f9ebb400640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9eb4076970 0x7f9eb419b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:00.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.689+0000 7f9ebb400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9eb410a840 0x7f9eb41a2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:00.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.689+0000 7f9ebb400640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb41106e0 con 0x7f9eb4076970 2024-04-26T23:46:00.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.689+0000 7f9ebb400640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb4110560 con 0x7f9eb410a840 2024-04-26T23:46:00.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.689+0000 7f9ebb400640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb4110860 con 0x7f9eb4075390 2024-04-26T23:46:00.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.689+0000 7f9eb3e00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9eb4076970 0x7f9eb419b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:00.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.689+0000 7f9eb8a00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9eb4075390 0x7f9eb419ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:00.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.689+0000 7f9eb8a00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9eb4075390 0x7f9eb419ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.72:44514/0 (socket says 172.21.15.72:44514) 2024-04-26T23:46:00.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.689+0000 7f9eb8a00640 1 -- 172.21.15.72:0/2076103675 learned_addr learned my addr 172.21.15.72:0/2076103675 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:46:00.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.689+0000 7f9eb9400640 1 --2- 172.21.15.72:0/2076103675 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9eb410a840 0x7f9eb41a2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:00.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.690+0000 7f9eb8a00640 1 -- 172.21.15.72:0/2076103675 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9eb410a840 msgr2=0x7f9eb41a2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:00.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.690+0000 7f9eb8a00640 1 --2- 172.21.15.72:0/2076103675 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9eb410a840 0x7f9eb41a2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:00.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.690+0000 7f9eb8a00640 1 -- 172.21.15.72:0/2076103675 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9eb4076970 msgr2=0x7f9eb419b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:00.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.690+0000 7f9eb8a00640 1 --2- 172.21.15.72:0/2076103675 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9eb4076970 0x7f9eb419b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:00.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.690+0000 7f9eb8a00640 1 -- 172.21.15.72:0/2076103675 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9eb41a2990 con 0x7f9eb4075390 2024-04-26T23:46:00.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.690+0000 7f9eb8a00640 1 --2- 172.21.15.72:0/2076103675 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9eb4075390 0x7f9eb419ad70 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9ea800e9f0 tx=0x7f9ea800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:46:00.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.690+0000 7f9eb1600640 1 -- 172.21.15.72:0/2076103675 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea800cd80 con 0x7f9eb4075390 2024-04-26T23:46:00.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.690+0000 7f9eb1600640 1 -- 172.21.15.72:0/2076103675 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ea80108e0 con 0x7f9eb4075390 2024-04-26T23:46:00.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.690+0000 7f9ebb400640 1 -- 172.21.15.72:0/2076103675 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9eb41a2c80 con 0x7f9eb4075390 2024-04-26T23:46:00.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.690+0000 7f9eb1600640 1 -- 172.21.15.72:0/2076103675 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea8018980 con 0x7f9eb4075390 2024-04-26T23:46:00.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.690+0000 7f9ebb400640 1 -- 172.21.15.72:0/2076103675 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9eb41a31d0 con 0x7f9eb4075390 2024-04-26T23:46:00.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.692+0000 7f9ebb400640 1 -- 172.21.15.72:0/2076103675 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9eb4109570 con 0x7f9eb4075390 2024-04-26T23:46:00.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.694+0000 7f9eb1600640 1 -- 172.21.15.72:0/2076103675 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f9ea8010430 con 0x7f9eb4075390 2024-04-26T23:46:00.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.695+0000 7f9eb1600640 1 --2- 172.21.15.72:0/2076103675 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f9e9c0a3cc0 0x7f9e9c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:00.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.696+0000 7f9eb1600640 1 -- 172.21.15.72:0/2076103675 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f9ea80d0ca0 con 0x7f9eb4075390 2024-04-26T23:46:00.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.696+0000 7f9eb3e00640 1 --2- 172.21.15.72:0/2076103675 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f9e9c0a3cc0 0x7f9e9c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:00.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.696+0000 7f9eb3e00640 1 --2- 172.21.15.72:0/2076103675 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f9e9c0a3cc0 0x7f9e9c0a6180 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9eb419c280 tx=0x7f9ea0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:46:00.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.700+0000 7f9eb1600640 1 -- 172.21.15.72:0/2076103675 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9ea809ca80 con 0x7f9eb4075390 2024-04-26T23:46:00.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:00 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/465701562' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:46:00.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.902+0000 7f9ebb400640 1 -- 172.21.15.72:0/2076103675 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9eb402dd10 con 0x7f9eb4075390 2024-04-26T23:46:00.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.903+0000 7f9eb1600640 1 -- 172.21.15.72:0/2076103675 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f9ea80a0d80 con 0x7f9eb4075390 2024-04-26T23:46:00.907 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-04-26T23:46:00.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.906+0000 7f9ebb400640 1 -- 172.21.15.72:0/2076103675 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f9e9c0a3cc0 msgr2=0x7f9e9c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:00.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.906+0000 7f9ebb400640 1 --2- 172.21.15.72:0/2076103675 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f9e9c0a3cc0 0x7f9e9c0a6180 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9eb419c280 tx=0x7f9ea0005f90 comp rx=0 tx=0).stop 2024-04-26T23:46:00.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.906+0000 7f9ebb400640 1 -- 172.21.15.72:0/2076103675 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9eb4075390 msgr2=0x7f9eb419ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:00.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.906+0000 7f9ebb400640 1 --2- 172.21.15.72:0/2076103675 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9eb4075390 0x7f9eb419ad70 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9ea800e9f0 tx=0x7f9ea800eeb0 comp rx=0 tx=0).stop 2024-04-26T23:46:00.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.906+0000 7f9ebb400640 1 -- 172.21.15.72:0/2076103675 shutdown_connections 2024-04-26T23:46:00.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.906+0000 7f9ebb400640 1 --2- 172.21.15.72:0/2076103675 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f9e9c0a3cc0 0x7f9e9c0a6180 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:00.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.906+0000 7f9ebb400640 1 --2- 172.21.15.72:0/2076103675 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9eb410a840 0x7f9eb41a2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:00.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.906+0000 7f9ebb400640 1 --2- 172.21.15.72:0/2076103675 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9eb4076970 0x7f9eb419b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:00.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.906+0000 7f9ebb400640 1 --2- 172.21.15.72:0/2076103675 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9eb4075390 0x7f9eb419ad70 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:00.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.906+0000 7f9ebb400640 1 -- 172.21.15.72:0/2076103675 >> 172.21.15.72:0/2076103675 conn(0x7f9eb40fdf10 msgr2=0x7f9eb4106740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:46:00.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.906+0000 7f9ebb400640 1 -- 172.21.15.72:0/2076103675 shutdown_connections 2024-04-26T23:46:00.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:00.906+0000 7f9ebb400640 1 -- 172.21.15.72:0/2076103675 wait complete. 2024-04-26T23:46:00.929 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T23:46:00.930 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:00 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/465701562' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:46:01.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:00 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/465701562' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:46:01.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.132+0000 7f746a000640 1 -- 172.21.15.72:0/2405106240 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7464076970 msgr2=0x7f7464074dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:01.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.132+0000 7f746a000640 1 --2- 172.21.15.72:0/2405106240 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7464076970 0x7f7464074dc0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7454009900 tx=0x7f745402a400 comp rx=0 tx=0).stop 2024-04-26T23:46:01.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.132+0000 7f746a000640 1 -- 172.21.15.72:0/2405106240 shutdown_connections 2024-04-26T23:46:01.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.132+0000 7f746a000640 1 --2- 172.21.15.72:0/2405106240 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7464075390 0x7f7464075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:01.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.132+0000 7f746a000640 1 --2- 172.21.15.72:0/2405106240 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f746410a700 0x7f746410cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:01.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.132+0000 7f746a000640 1 --2- 172.21.15.72:0/2405106240 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7464076970 0x7f7464074dc0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:01.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.132+0000 7f746a000640 1 -- 172.21.15.72:0/2405106240 >> 172.21.15.72:0/2405106240 conn(0x7f74640fddd0 msgr2=0x7f74641001f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:46:01.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.132+0000 7f746a000640 1 -- 172.21.15.72:0/2405106240 shutdown_connections 2024-04-26T23:46:01.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.132+0000 7f746a000640 1 -- 172.21.15.72:0/2405106240 wait complete. 2024-04-26T23:46:01.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.133+0000 7f746a000640 1 Processor -- start 2024-04-26T23:46:01.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.134+0000 7f746a000640 1 -- start start 2024-04-26T23:46:01.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.134+0000 7f746a000640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7464075390 0x7f746419ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:01.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.134+0000 7f746a000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7464076970 0x7f746419b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:01.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.134+0000 7f746a000640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f746410a700 0x7f74641a21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:01.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.134+0000 7f746a000640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f74641105a0 con 0x7f7464075390 2024-04-26T23:46:01.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.134+0000 7f746a000640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f7464110420 con 0x7f7464076970 2024-04-26T23:46:01.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.134+0000 7f746a000640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f7464110720 con 0x7f746410a700 2024-04-26T23:46:01.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.134+0000 7f7468c00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7464075390 0x7f746419ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:01.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.134+0000 7f7468c00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7464075390 0x7f746419ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57916/0 (socket says 172.21.15.72:57916) 2024-04-26T23:46:01.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.134+0000 7f7468c00640 1 -- 172.21.15.72:0/3951149315 learned_addr learned my addr 172.21.15.72:0/3951149315 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:46:01.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.134+0000 7f7463e00640 1 --2- 172.21.15.72:0/3951149315 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7464076970 0x7f746419b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:01.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.134+0000 7f7469600640 1 --2- 172.21.15.72:0/3951149315 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f746410a700 0x7f74641a21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:01.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.135+0000 7f7468c00640 1 -- 172.21.15.72:0/3951149315 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f746410a700 msgr2=0x7f74641a21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:01.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.135+0000 7f7468c00640 1 --2- 172.21.15.72:0/3951149315 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f746410a700 0x7f74641a21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:01.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.135+0000 7f7468c00640 1 -- 172.21.15.72:0/3951149315 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7464076970 msgr2=0x7f746419b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:01.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.135+0000 7f7468c00640 1 --2- 172.21.15.72:0/3951149315 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7464076970 0x7f746419b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:01.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.135+0000 7f7468c00640 1 -- 172.21.15.72:0/3951149315 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74641a2870 con 0x7f7464075390 2024-04-26T23:46:01.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.135+0000 7f7463e00640 1 --2- 172.21.15.72:0/3951149315 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7464076970 0x7f746419b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:46:01.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.135+0000 7f7468c00640 1 --2- 172.21.15.72:0/3951149315 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7464075390 0x7f746419ac60 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f7454005ab0 tx=0x7f7454003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:46:01.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.136+0000 7f7461600640 1 -- 172.21.15.72:0/3951149315 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f745403f070 con 0x7f7464075390 2024-04-26T23:46:01.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.136+0000 7f746a000640 1 -- 172.21.15.72:0/3951149315 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74641a2b00 con 0x7f7464075390 2024-04-26T23:46:01.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.136+0000 7f7461600640 1 -- 172.21.15.72:0/3951149315 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7454031c50 con 0x7f7464075390 2024-04-26T23:46:01.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.136+0000 7f7461600640 1 -- 172.21.15.72:0/3951149315 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7454035030 con 0x7f7464075390 2024-04-26T23:46:01.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.136+0000 7f746a000640 1 -- 172.21.15.72:0/3951149315 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74641a2ff0 con 0x7f7464075390 2024-04-26T23:46:01.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.137+0000 7f746a000640 1 -- 172.21.15.72:0/3951149315 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7464109430 con 0x7f7464075390 2024-04-26T23:46:01.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.139+0000 7f7461600640 1 -- 172.21.15.72:0/3951149315 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7454042070 con 0x7f7464075390 2024-04-26T23:46:01.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.140+0000 7f7461600640 1 --2- 172.21.15.72:0/3951149315 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f744c0a3c40 0x7f744c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:01.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.140+0000 7f7461600640 1 -- 172.21.15.72:0/3951149315 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f74540f11b0 con 0x7f7464075390 2024-04-26T23:46:01.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.141+0000 7f7463e00640 1 --2- 172.21.15.72:0/3951149315 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f744c0a3c40 0x7f744c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:01.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.141+0000 7f7463e00640 1 --2- 172.21.15.72:0/3951149315 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f744c0a3c40 0x7f744c0a6100 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f7458009fc0 tx=0x7f7458005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:46:01.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.146+0000 7f7461600640 1 -- 172.21.15.72:0/3951149315 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f74540bcf90 con 0x7f7464075390 2024-04-26T23:46:01.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.348+0000 7f746a000640 1 -- 172.21.15.72:0/3951149315 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f746402dd20 con 0x7f7464075390 2024-04-26T23:46:01.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.349+0000 7f7461600640 1 -- 172.21.15.72:0/3951149315 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f74540c1290 con 0x7f7464075390 2024-04-26T23:46:01.353 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-26T23:46:01.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.352+0000 7f746a000640 1 -- 172.21.15.72:0/3951149315 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f744c0a3c40 msgr2=0x7f744c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:01.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.352+0000 7f746a000640 1 --2- 172.21.15.72:0/3951149315 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f744c0a3c40 0x7f744c0a6100 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f7458009fc0 tx=0x7f7458005f90 comp rx=0 tx=0).stop 2024-04-26T23:46:01.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.352+0000 7f746a000640 1 -- 172.21.15.72:0/3951149315 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7464075390 msgr2=0x7f746419ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:01.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.352+0000 7f746a000640 1 --2- 172.21.15.72:0/3951149315 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7464075390 0x7f746419ac60 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f7454005ab0 tx=0x7f7454003b90 comp rx=0 tx=0).stop 2024-04-26T23:46:01.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.352+0000 7f746a000640 1 -- 172.21.15.72:0/3951149315 shutdown_connections 2024-04-26T23:46:01.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.352+0000 7f746a000640 1 --2- 172.21.15.72:0/3951149315 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f744c0a3c40 0x7f744c0a6100 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:01.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.352+0000 7f746a000640 1 --2- 172.21.15.72:0/3951149315 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7464076970 0x7f746419b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:01.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.352+0000 7f746a000640 1 --2- 172.21.15.72:0/3951149315 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f746410a700 0x7f74641a21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:01.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.352+0000 7f746a000640 1 --2- 172.21.15.72:0/3951149315 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7464075390 0x7f746419ac60 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:01.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.352+0000 7f746a000640 1 -- 172.21.15.72:0/3951149315 >> 172.21.15.72:0/3951149315 conn(0x7f74640fddd0 msgr2=0x7f7464106600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:46:01.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.352+0000 7f746a000640 1 -- 172.21.15.72:0/3951149315 shutdown_connections 2024-04-26T23:46:01.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.352+0000 7f746a000640 1 -- 172.21.15.72:0/3951149315 wait complete. 2024-04-26T23:46:01.376 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T23:46:01.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.573+0000 7f2093e00640 1 -- 172.21.15.72:0/2548516579 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2094108d90 msgr2=0x7f20941091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:01.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.573+0000 7f2093e00640 1 --2- 172.21.15.72:0/2548516579 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2094108d90 0x7f20941091b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2084009900 tx=0x7f208402a410 comp rx=0 tx=0).stop 2024-04-26T23:46:01.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.573+0000 7f2093e00640 1 -- 172.21.15.72:0/2548516579 shutdown_connections 2024-04-26T23:46:01.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.573+0000 7f2093e00640 1 --2- 172.21.15.72:0/2548516579 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f209410a940 0x7f20941111d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:01.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.573+0000 7f2093e00640 1 --2- 172.21.15.72:0/2548516579 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2094109f80 0x7f209410a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:01.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.573+0000 7f2093e00640 1 --2- 172.21.15.72:0/2548516579 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2094108d90 0x7f20941091b0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:01.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.573+0000 7f2093e00640 1 -- 172.21.15.72:0/2548516579 >> 172.21.15.72:0/2548516579 conn(0x7f2094077d30 msgr2=0x7f209407a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:46:01.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.573+0000 7f2093e00640 1 -- 172.21.15.72:0/2548516579 shutdown_connections 2024-04-26T23:46:01.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.573+0000 7f2093e00640 1 -- 172.21.15.72:0/2548516579 wait complete. 2024-04-26T23:46:01.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.574+0000 7f2093e00640 1 Processor -- start 2024-04-26T23:46:01.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.574+0000 7f2093e00640 1 -- start start 2024-04-26T23:46:01.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.575+0000 7f2093e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2094108d90 0x7f209419f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:01.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.575+0000 7f2093e00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2094109f80 0x7f209419f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:01.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.575+0000 7f2093e00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f209410a940 0x7f20941a6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:01.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.575+0000 7f2093e00640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f2094114910 con 0x7f2094109f80 2024-04-26T23:46:01.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.575+0000 7f2093e00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2094114790 con 0x7f2094108d90 2024-04-26T23:46:01.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.575+0000 7f2093e00640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f2094114a90 con 0x7f209410a940 2024-04-26T23:46:01.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.575+0000 7f2092000640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2094109f80 0x7f209419f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:01.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.575+0000 7f2092a00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2094108d90 0x7f209419f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:01.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.575+0000 7f2092000640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2094109f80 0x7f209419f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57936/0 (socket says 172.21.15.72:57936) 2024-04-26T23:46:01.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.575+0000 7f2093400640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f209410a940 0x7f20941a6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:01.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.575+0000 7f2092000640 1 -- 172.21.15.72:0/608743343 learned_addr learned my addr 172.21.15.72:0/608743343 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:46:01.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.575+0000 7f2092a00640 1 --2- 172.21.15.72:0/608743343 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2094108d90 0x7f209419f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:34028/0 (socket says 172.21.15.72:34028) 2024-04-26T23:46:01.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.576+0000 7f2092000640 1 -- 172.21.15.72:0/608743343 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f209410a940 msgr2=0x7f20941a6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:01.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.576+0000 7f2092000640 1 --2- 172.21.15.72:0/608743343 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f209410a940 0x7f20941a6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:01.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.576+0000 7f2092000640 1 -- 172.21.15.72:0/608743343 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2094108d90 msgr2=0x7f209419f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:01.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.576+0000 7f2092000640 1 --2- 172.21.15.72:0/608743343 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2094108d90 0x7f209419f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:01.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.576+0000 7f2092000640 1 -- 172.21.15.72:0/608743343 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20941a6bd0 con 0x7f2094109f80 2024-04-26T23:46:01.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.576+0000 7f2093400640 1 --2- 172.21.15.72:0/608743343 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f209410a940 0x7f20941a6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:46:01.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.576+0000 7f2092000640 1 --2- 172.21.15.72:0/608743343 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2094109f80 0x7f209419f540 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f208000e9f0 tx=0x7f208000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:46:01.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.576+0000 7f208b400640 1 -- 172.21.15.72:0/608743343 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f208000cd80 con 0x7f2094109f80 2024-04-26T23:46:01.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.576+0000 7f208b400640 1 -- 172.21.15.72:0/608743343 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f20800108e0 con 0x7f2094109f80 2024-04-26T23:46:01.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.576+0000 7f2093e00640 1 -- 172.21.15.72:0/608743343 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20941a6ec0 con 0x7f2094109f80 2024-04-26T23:46:01.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.576+0000 7f208b400640 1 -- 172.21.15.72:0/608743343 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2080018980 con 0x7f2094109f80 2024-04-26T23:46:01.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.576+0000 7f2093e00640 1 -- 172.21.15.72:0/608743343 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20941a7380 con 0x7f2094109f80 2024-04-26T23:46:01.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.578+0000 7f2093e00640 1 -- 172.21.15.72:0/608743343 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20941100b0 con 0x7f2094109f80 2024-04-26T23:46:01.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.581+0000 7f208b400640 1 -- 172.21.15.72:0/608743343 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f2080010430 con 0x7f2094109f80 2024-04-26T23:46:01.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.583+0000 7f208b400640 1 --2- 172.21.15.72:0/608743343 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f20780a3c40 0x7f20780a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:01.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.583+0000 7f208b400640 1 -- 172.21.15.72:0/608743343 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f20800d0cb0 con 0x7f2094109f80 2024-04-26T23:46:01.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.583+0000 7f2092a00640 1 --2- 172.21.15.72:0/608743343 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f20780a3c40 0x7f20780a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:01.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.584+0000 7f2092a00640 1 --2- 172.21.15.72:0/608743343 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f20780a3c40 0x7f20780a6100 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f2084005ab0 tx=0x7f2084005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:46:01.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.587+0000 7f208b400640 1 -- 172.21.15.72:0/608743343 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f208009ca90 con 0x7f2094109f80 2024-04-26T23:46:01.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:01 smithi072 ceph-mon[28184]: pgmap v414: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:46:01.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:01 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/2076103675' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T23:46:01.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:01 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/3951149315' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T23:46:01.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.792+0000 7f2093e00640 1 -- 172.21.15.72:0/608743343 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f209402dcf0 con 0x7f2094109f80 2024-04-26T23:46:01.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.793+0000 7f208b400640 1 -- 172.21.15.72:0/608743343 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f20800a0d90 con 0x7f2094109f80 2024-04-26T23:46:01.798 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-04-26T23:46:01.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.796+0000 7f2093e00640 1 -- 172.21.15.72:0/608743343 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f20780a3c40 msgr2=0x7f20780a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:01.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.796+0000 7f2093e00640 1 --2- 172.21.15.72:0/608743343 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f20780a3c40 0x7f20780a6100 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f2084005ab0 tx=0x7f2084005a20 comp rx=0 tx=0).stop 2024-04-26T23:46:01.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.796+0000 7f2093e00640 1 -- 172.21.15.72:0/608743343 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2094109f80 msgr2=0x7f209419f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:01.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.796+0000 7f2093e00640 1 --2- 172.21.15.72:0/608743343 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2094109f80 0x7f209419f540 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f208000e9f0 tx=0x7f208000eeb0 comp rx=0 tx=0).stop 2024-04-26T23:46:01.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.796+0000 7f2093e00640 1 -- 172.21.15.72:0/608743343 shutdown_connections 2024-04-26T23:46:01.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.796+0000 7f2093e00640 1 --2- 172.21.15.72:0/608743343 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f20780a3c40 0x7f20780a6100 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:01.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.796+0000 7f2093e00640 1 --2- 172.21.15.72:0/608743343 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f209410a940 0x7f20941a6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:01.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.796+0000 7f2093e00640 1 --2- 172.21.15.72:0/608743343 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2094109f80 0x7f209419f540 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:01.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.796+0000 7f2093e00640 1 --2- 172.21.15.72:0/608743343 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2094108d90 0x7f209419f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:01.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.796+0000 7f2093e00640 1 -- 172.21.15.72:0/608743343 >> 172.21.15.72:0/608743343 conn(0x7f2094077d30 msgr2=0x7f209410d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:46:01.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.796+0000 7f2093e00640 1 -- 172.21.15.72:0/608743343 shutdown_connections 2024-04-26T23:46:01.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:01.796+0000 7f2093e00640 1 -- 172.21.15.72:0/608743343 wait complete. 2024-04-26T23:46:01.820 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-26T23:46:01.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:01 smithi076 ceph-mon[32774]: pgmap v414: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:46:01.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:01 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/2076103675' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T23:46:01.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:01 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/3951149315' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T23:46:02.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.013+0000 7f816ac00640 1 -- 172.21.15.72:0/4196947843 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8164106620 msgr2=0x7f816410ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:02.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.013+0000 7f816ac00640 1 --2- 172.21.15.72:0/4196947843 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8164106620 0x7f816410ceb0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f815c009be0 tx=0x7f815c02a6d0 comp rx=0 tx=0).stop 2024-04-26T23:46:02.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.013+0000 7f816ac00640 1 -- 172.21.15.72:0/4196947843 shutdown_connections 2024-04-26T23:46:02.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.013+0000 7f816ac00640 1 --2- 172.21.15.72:0/4196947843 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8164106620 0x7f816410ceb0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:02.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.013+0000 7f816ac00640 1 --2- 172.21.15.72:0/4196947843 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8164105c60 0x7f81641060e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:02.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.013+0000 7f816ac00640 1 --2- 172.21.15.72:0/4196947843 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8164104a70 0x7f8164104e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:02.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.013+0000 7f816ac00640 1 -- 172.21.15.72:0/4196947843 >> 172.21.15.72:0/4196947843 conn(0x7f81640fffe0 msgr2=0x7f8164102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:46:02.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.013+0000 7f816ac00640 1 -- 172.21.15.72:0/4196947843 shutdown_connections 2024-04-26T23:46:02.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.013+0000 7f816ac00640 1 -- 172.21.15.72:0/4196947843 wait complete. 2024-04-26T23:46:02.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.014+0000 7f816ac00640 1 Processor -- start 2024-04-26T23:46:02.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.014+0000 7f816ac00640 1 -- start start 2024-04-26T23:46:02.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.015+0000 7f816ac00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8164104a70 0x7f816419ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:02.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.015+0000 7f816ac00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8164105c60 0x7f816419b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:02.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.015+0000 7f816ac00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8164106620 0x7f81641a21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:02.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.015+0000 7f816ac00640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f81641105a0 con 0x7f8164104a70 2024-04-26T23:46:02.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.015+0000 7f816ac00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8164110420 con 0x7f8164106620 2024-04-26T23:46:02.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.015+0000 7f816ac00640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f8164110720 con 0x7f8164105c60 2024-04-26T23:46:02.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.015+0000 7f8169800640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8164104a70 0x7f816419ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:02.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.015+0000 7f8169800640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8164104a70 0x7f816419ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57962/0 (socket says 172.21.15.72:57962) 2024-04-26T23:46:02.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.015+0000 7f8169800640 1 -- 172.21.15.72:0/264878021 learned_addr learned my addr 172.21.15.72:0/264878021 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:46:02.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.015+0000 7f8168e00640 1 --2- 172.21.15.72:0/264878021 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8164105c60 0x7f816419b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:02.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.015+0000 7f816a200640 1 --2- 172.21.15.72:0/264878021 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8164106620 0x7f81641a21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:02.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.016+0000 7f8169800640 1 -- 172.21.15.72:0/264878021 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8164105c60 msgr2=0x7f816419b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:02.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.016+0000 7f8169800640 1 --2- 172.21.15.72:0/264878021 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8164105c60 0x7f816419b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:02.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.016+0000 7f8169800640 1 -- 172.21.15.72:0/264878021 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8164106620 msgr2=0x7f81641a21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:02.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.016+0000 7f8169800640 1 --2- 172.21.15.72:0/264878021 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8164106620 0x7f81641a21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:02.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.016+0000 7f8169800640 1 -- 172.21.15.72:0/264878021 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81641a2890 con 0x7f8164104a70 2024-04-26T23:46:02.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.016+0000 7f8169800640 1 --2- 172.21.15.72:0/264878021 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8164104a70 0x7f816419ac80 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f815800e9a0 tx=0x7f815800ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:46:02.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.016+0000 7f8162000640 1 -- 172.21.15.72:0/264878021 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f815800cd80 con 0x7f8164104a70 2024-04-26T23:46:02.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.016+0000 7f8162000640 1 -- 172.21.15.72:0/264878021 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f81580108e0 con 0x7f8164104a70 2024-04-26T23:46:02.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.016+0000 7f816ac00640 1 -- 172.21.15.72:0/264878021 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81641a2b80 con 0x7f8164104a70 2024-04-26T23:46:02.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.016+0000 7f8162000640 1 -- 172.21.15.72:0/264878021 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8158018960 con 0x7f8164104a70 2024-04-26T23:46:02.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.017+0000 7f816ac00640 1 -- 172.21.15.72:0/264878021 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81641a30d0 con 0x7f8164104a70 2024-04-26T23:46:02.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.018+0000 7f816ac00640 1 -- 172.21.15.72:0/264878021 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f816410be60 con 0x7f8164104a70 2024-04-26T23:46:02.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.020+0000 7f8162000640 1 -- 172.21.15.72:0/264878021 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f8158010a90 con 0x7f8164104a70 2024-04-26T23:46:02.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.021+0000 7f8162000640 1 --2- 172.21.15.72:0/264878021 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f814c0a3c40 0x7f814c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:02.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.021+0000 7f8162000640 1 -- 172.21.15.72:0/264878021 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f81580d0d40 con 0x7f8164104a70 2024-04-26T23:46:02.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.021+0000 7f8168e00640 1 --2- 172.21.15.72:0/264878021 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f814c0a3c40 0x7f814c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:02.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.022+0000 7f8168e00640 1 --2- 172.21.15.72:0/264878021 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f814c0a3c40 0x7f814c0a6100 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f8150009bd0 tx=0x7f8150005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:46:02.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.026+0000 7f8162000640 1 -- 172.21.15.72:0/264878021 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8158054060 con 0x7f8164104a70 2024-04-26T23:46:02.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:01 smithi136 ceph-mon[33050]: pgmap v414: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:46:02.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:01 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/2076103675' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T23:46:02.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:01 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/3951149315' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T23:46:02.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.225+0000 7f816ac00640 1 -- 172.21.15.72:0/264878021 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f816402dd50 con 0x7f8164104a70 2024-04-26T23:46:02.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.227+0000 7f8162000640 1 -- 172.21.15.72:0/264878021 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f815809d1d0 con 0x7f8164104a70 2024-04-26T23:46:02.231 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-26T23:46:02.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.229+0000 7f816ac00640 1 -- 172.21.15.72:0/264878021 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f814c0a3c40 msgr2=0x7f814c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:02.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.229+0000 7f816ac00640 1 --2- 172.21.15.72:0/264878021 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f814c0a3c40 0x7f814c0a6100 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f8150009bd0 tx=0x7f8150005f90 comp rx=0 tx=0).stop 2024-04-26T23:46:02.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.229+0000 7f816ac00640 1 -- 172.21.15.72:0/264878021 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8164104a70 msgr2=0x7f816419ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:02.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.229+0000 7f816ac00640 1 --2- 172.21.15.72:0/264878021 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8164104a70 0x7f816419ac80 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f815800e9a0 tx=0x7f815800ee60 comp rx=0 tx=0).stop 2024-04-26T23:46:02.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.229+0000 7f816ac00640 1 -- 172.21.15.72:0/264878021 shutdown_connections 2024-04-26T23:46:02.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.229+0000 7f816ac00640 1 --2- 172.21.15.72:0/264878021 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f814c0a3c40 0x7f814c0a6100 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:02.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.229+0000 7f816ac00640 1 --2- 172.21.15.72:0/264878021 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8164106620 0x7f81641a21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:02.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.229+0000 7f816ac00640 1 --2- 172.21.15.72:0/264878021 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8164105c60 0x7f816419b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:02.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.229+0000 7f816ac00640 1 --2- 172.21.15.72:0/264878021 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8164104a70 0x7f816419ac80 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:02.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.229+0000 7f816ac00640 1 -- 172.21.15.72:0/264878021 >> 172.21.15.72:0/264878021 conn(0x7f81640fffe0 msgr2=0x7f8164109030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:46:02.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.230+0000 7f816ac00640 1 -- 172.21.15.72:0/264878021 shutdown_connections 2024-04-26T23:46:02.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.230+0000 7f816ac00640 1 -- 172.21.15.72:0/264878021 wait complete. 2024-04-26T23:46:02.254 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:46:02.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.451+0000 7fb3c1c00640 1 -- 172.21.15.72:0/3057057910 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb3bc108430 msgr2=0x7fb3bc10a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:02.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.451+0000 7fb3c1c00640 1 --2- 172.21.15.72:0/3057057910 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb3bc108430 0x7fb3bc10a880 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb3b4009ca0 tx=0x7fb3b402a8c0 comp rx=0 tx=0).stop 2024-04-26T23:46:02.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.452+0000 7fb3c1c00640 1 -- 172.21.15.72:0/3057057910 shutdown_connections 2024-04-26T23:46:02.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.452+0000 7fb3c1c00640 1 --2- 172.21.15.72:0/3057057910 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb3bc108430 0x7fb3bc10a880 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:02.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.452+0000 7fb3c1c00640 1 --2- 172.21.15.72:0/3057057910 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3bc105b00 0x7fb3bc107ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:02.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.452+0000 7fb3c1c00640 1 --2- 172.21.15.72:0/3057057910 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb3bc1031d0 0x7fb3bc1055c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:02.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.452+0000 7fb3c1c00640 1 -- 172.21.15.72:0/3057057910 >> 172.21.15.72:0/3057057910 conn(0x7fb3bc0fcbe0 msgr2=0x7fb3bc0ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:46:02.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.452+0000 7fb3c1c00640 1 -- 172.21.15.72:0/3057057910 shutdown_connections 2024-04-26T23:46:02.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.452+0000 7fb3c1c00640 1 -- 172.21.15.72:0/3057057910 wait complete. 2024-04-26T23:46:02.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.453+0000 7fb3c1c00640 1 Processor -- start 2024-04-26T23:46:02.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.453+0000 7fb3c1c00640 1 -- start start 2024-04-26T23:46:02.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.453+0000 7fb3c1c00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3bc1031d0 0x7fb3bc198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:02.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.453+0000 7fb3c1c00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb3bc105b00 0x7fb3bc198e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:02.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.453+0000 7fb3c1c00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb3bc108430 0x7fb3bc19fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:02.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.453+0000 7fb3c1c00640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fb3bc10e2f0 con 0x7fb3bc1031d0 2024-04-26T23:46:02.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.453+0000 7fb3c1c00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fb3bc10e170 con 0x7fb3bc108430 2024-04-26T23:46:02.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.453+0000 7fb3c1c00640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fb3bc10e470 con 0x7fb3bc105b00 2024-04-26T23:46:02.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.454+0000 7fb3bb400640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb3bc105b00 0x7fb3bc198e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:02.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.454+0000 7fb3bb400640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb3bc105b00 0x7fb3bc198e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.72:44602/0 (socket says 172.21.15.72:44602) 2024-04-26T23:46:02.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.454+0000 7fb3bb400640 1 -- 172.21.15.72:0/3217728662 learned_addr learned my addr 172.21.15.72:0/3217728662 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:46:02.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.454+0000 7fb3bbe00640 1 --2- 172.21.15.72:0/3217728662 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3bc1031d0 0x7fb3bc198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:02.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.454+0000 7fb3c1200640 1 --2- 172.21.15.72:0/3217728662 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb3bc108430 0x7fb3bc19fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:02.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.454+0000 7fb3bb400640 1 -- 172.21.15.72:0/3217728662 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb3bc108430 msgr2=0x7fb3bc19fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:02.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.454+0000 7fb3bb400640 1 --2- 172.21.15.72:0/3217728662 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb3bc108430 0x7fb3bc19fec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:02.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.454+0000 7fb3bb400640 1 -- 172.21.15.72:0/3217728662 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3bc1031d0 msgr2=0x7fb3bc198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:02.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.454+0000 7fb3bb400640 1 --2- 172.21.15.72:0/3217728662 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3bc1031d0 0x7fb3bc198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:02.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.454+0000 7fb3bb400640 1 -- 172.21.15.72:0/3217728662 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3bc1a0530 con 0x7fb3bc105b00 2024-04-26T23:46:02.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.454+0000 7fb3c1200640 1 --2- 172.21.15.72:0/3217728662 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb3bc108430 0x7fb3bc19fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:46:02.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.454+0000 7fb3bb400640 1 --2- 172.21.15.72:0/3217728662 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb3bc105b00 0x7fb3bc198e50 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb3a800e9c0 tx=0x7fb3a800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:46:02.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.455+0000 7fb3b8c00640 1 -- 172.21.15.72:0/3217728662 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a800cd80 con 0x7fb3bc105b00 2024-04-26T23:46:02.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.455+0000 7fb3c1c00640 1 -- 172.21.15.72:0/3217728662 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3bc1a0820 con 0x7fb3bc105b00 2024-04-26T23:46:02.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.455+0000 7fb3b8c00640 1 -- 172.21.15.72:0/3217728662 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb3a80054d0 con 0x7fb3bc105b00 2024-04-26T23:46:02.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.455+0000 7fb3b8c00640 1 -- 172.21.15.72:0/3217728662 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a8010430 con 0x7fb3bc105b00 2024-04-26T23:46:02.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.455+0000 7fb3c1c00640 1 -- 172.21.15.72:0/3217728662 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3bc1a0dc0 con 0x7fb3bc105b00 2024-04-26T23:46:02.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.456+0000 7fb3c1c00640 1 -- 172.21.15.72:0/3217728662 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3bc0fe6d0 con 0x7fb3bc105b00 2024-04-26T23:46:02.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.458+0000 7fb3b8c00640 1 -- 172.21.15.72:0/3217728662 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fb3a8004fb0 con 0x7fb3bc105b00 2024-04-26T23:46:02.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.460+0000 7fb3b8c00640 1 --2- 172.21.15.72:0/3217728662 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fb3a00a3c40 0x7fb3a00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:02.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.460+0000 7fb3b8c00640 1 -- 172.21.15.72:0/3217728662 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7fb3a80d0fc0 con 0x7fb3bc105b00 2024-04-26T23:46:02.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.460+0000 7fb3bbe00640 1 --2- 172.21.15.72:0/3217728662 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fb3a00a3c40 0x7fb3a00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:02.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.461+0000 7fb3bbe00640 1 --2- 172.21.15.72:0/3217728662 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fb3a00a3c40 0x7fb3a00a6100 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac009bd0 tx=0x7fb3ac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:46:02.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.462+0000 7fb3b8c00640 1 -- 172.21.15.72:0/3217728662 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb3a8054060 con 0x7fb3bc105b00 2024-04-26T23:46:02.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.713+0000 7fb3c1c00640 1 -- 172.21.15.72:0/3217728662 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb3bc02dc90 con 0x7fb3bc105b00 2024-04-26T23:46:02.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.714+0000 7fb3b8c00640 1 -- 172.21.15.72:0/3217728662 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11874 (secure 0 0 0) 0x7fb3a809d3f0 con 0x7fb3bc105b00 2024-04-26T23:46:02.719 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:46:02.719 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/2386456166","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2386456166},{"type":"v1","addr":"172.21.15.72:6835","nonce":2386456166}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1528342053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1528342053},{"type":"v1","addr":"172.21.15.136:6833","nonce":1528342053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6833/2427238559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":2427238559},{"type":"v1","addr":"172.21.15.76:6833","nonce":2427238559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24491,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/4050689894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":4050689894},{"type":"v1","addr":"172.21.15.72:6837","nonce":4050689894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3681925367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3681925367},{"type":"v1","addr":"172.21.15.136:6835","nonce":3681925367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4251109085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4251109085},{"type":"v1","addr":"172.21.15.76:6835","nonce":4251109085}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T23:43:02.928649+0000","modified":"2024-04-26T23:43:30.190394+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24590,"mds_1":24571,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24514":{"gid":24514,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6837/3639308555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3639308555},{"type":"v1","addr":"172.21.15.136:6837","nonce":3639308555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.76:6837/2506930945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2506930945},{"type":"v1","addr":"172.21.15.76:6837","nonce":2506930945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6839/3938677630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3938677630},{"type":"v1","addr":"172.21.15.136:6839","nonce":3938677630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.72:6839/1403162102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6838","nonce":1403162102},{"type":"v1","addr":"172.21.15.72:6839","nonce":1403162102}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24571":{"gid":24571,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.72:6841/1356345077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6840","nonce":1356345077},{"type":"v1","addr":"172.21.15.72:6841","nonce":1356345077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.76:6839/3405829058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":3405829058},{"type":"v1","addr":"172.21.15.76:6839","nonce":3405829058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24553,24571,24590]},"id":1}]} 2024-04-26T23:46:02.719 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 23 2024-04-26T23:46:02.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.717+0000 7fb3c1c00640 1 -- 172.21.15.72:0/3217728662 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fb3a00a3c40 msgr2=0x7fb3a00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:02.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.717+0000 7fb3c1c00640 1 --2- 172.21.15.72:0/3217728662 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fb3a00a3c40 0x7fb3a00a6100 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac009bd0 tx=0x7fb3ac005f90 comp rx=0 tx=0).stop 2024-04-26T23:46:02.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.717+0000 7fb3c1c00640 1 -- 172.21.15.72:0/3217728662 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb3bc105b00 msgr2=0x7fb3bc198e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:02.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.717+0000 7fb3c1c00640 1 --2- 172.21.15.72:0/3217728662 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb3bc105b00 0x7fb3bc198e50 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb3a800e9c0 tx=0x7fb3a800ee80 comp rx=0 tx=0).stop 2024-04-26T23:46:02.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.717+0000 7fb3c1c00640 1 -- 172.21.15.72:0/3217728662 shutdown_connections 2024-04-26T23:46:02.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.717+0000 7fb3c1c00640 1 --2- 172.21.15.72:0/3217728662 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fb3a00a3c40 0x7fb3a00a6100 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:02.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.717+0000 7fb3c1c00640 1 --2- 172.21.15.72:0/3217728662 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb3bc108430 0x7fb3bc19fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:02.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.717+0000 7fb3c1c00640 1 --2- 172.21.15.72:0/3217728662 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb3bc105b00 0x7fb3bc198e50 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:02.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.717+0000 7fb3c1c00640 1 --2- 172.21.15.72:0/3217728662 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3bc1031d0 0x7fb3bc198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:02.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.717+0000 7fb3c1c00640 1 -- 172.21.15.72:0/3217728662 >> 172.21.15.72:0/3217728662 conn(0x7fb3bc0fcbe0 msgr2=0x7fb3bc103b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:46:02.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.717+0000 7fb3c1c00640 1 -- 172.21.15.72:0/3217728662 shutdown_connections 2024-04-26T23:46:02.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.718+0000 7fb3c1c00640 1 -- 172.21.15.72:0/3217728662 wait complete. 2024-04-26T23:46:02.743 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:46:02.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:02 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/608743343' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T23:46:02.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:02 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/264878021' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T23:46:02.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:02 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/608743343' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T23:46:02.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:02 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/264878021' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T23:46:02.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.943+0000 7ffa93200640 1 -- 172.21.15.72:0/2446672073 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa9410a940 msgr2=0x7ffa941111d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:02.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.943+0000 7ffa93200640 1 --2- 172.21.15.72:0/2446672073 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa9410a940 0x7ffa941111d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c009ca0 tx=0x7ffa8c02a8b0 comp rx=0 tx=0).stop 2024-04-26T23:46:02.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.944+0000 7ffa93200640 1 -- 172.21.15.72:0/2446672073 shutdown_connections 2024-04-26T23:46:02.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.944+0000 7ffa93200640 1 --2- 172.21.15.72:0/2446672073 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffa94109f80 0x7ffa9410a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:02.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.944+0000 7ffa93200640 1 --2- 172.21.15.72:0/2446672073 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa9410a940 0x7ffa941111d0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:02.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.944+0000 7ffa93200640 1 --2- 172.21.15.72:0/2446672073 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffa94108d90 0x7ffa941091b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:02.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.944+0000 7ffa93200640 1 -- 172.21.15.72:0/2446672073 >> 172.21.15.72:0/2446672073 conn(0x7ffa94077d30 msgr2=0x7ffa9407a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:46:02.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.944+0000 7ffa93200640 1 -- 172.21.15.72:0/2446672073 shutdown_connections 2024-04-26T23:46:02.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.944+0000 7ffa93200640 1 -- 172.21.15.72:0/2446672073 wait complete. 2024-04-26T23:46:02.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.945+0000 7ffa93200640 1 Processor -- start 2024-04-26T23:46:02.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.945+0000 7ffa93200640 1 -- start start 2024-04-26T23:46:02.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.945+0000 7ffa93200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffa94108d90 0x7ffa9419ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:02.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.945+0000 7ffa93200640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa94109f80 0x7ffa9419f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:02.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.945+0000 7ffa93200640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffa9410a940 0x7ffa941a6500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:02.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.945+0000 7ffa93200640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7ffa94114910 con 0x7ffa9410a940 2024-04-26T23:46:02.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.945+0000 7ffa93200640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ffa94114790 con 0x7ffa94108d90 2024-04-26T23:46:02.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.945+0000 7ffa93200640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ffa94114a90 con 0x7ffa94109f80 2024-04-26T23:46:02.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.945+0000 7ffa92800640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffa9410a940 0x7ffa941a6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:02.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.945+0000 7ffa89400640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa94109f80 0x7ffa9419f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:02.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.945+0000 7ffa91e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffa94108d90 0x7ffa9419ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:02.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.945+0000 7ffa89400640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa94109f80 0x7ffa9419f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.72:44622/0 (socket says 172.21.15.72:44622) 2024-04-26T23:46:02.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.945+0000 7ffa89400640 1 -- 172.21.15.72:0/1531362679 learned_addr learned my addr 172.21.15.72:0/1531362679 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:46:02.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.946+0000 7ffa91e00640 1 -- 172.21.15.72:0/1531362679 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa94109f80 msgr2=0x7ffa9419f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:02.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.946+0000 7ffa91e00640 1 --2- 172.21.15.72:0/1531362679 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa94109f80 0x7ffa9419f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:02.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.946+0000 7ffa91e00640 1 -- 172.21.15.72:0/1531362679 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffa9410a940 msgr2=0x7ffa941a6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:02.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.946+0000 7ffa91e00640 1 --2- 172.21.15.72:0/1531362679 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffa9410a940 0x7ffa941a6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:02.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.946+0000 7ffa91e00640 1 -- 172.21.15.72:0/1531362679 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa941a6b70 con 0x7ffa94108d90 2024-04-26T23:46:02.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.946+0000 7ffa92800640 1 --2- 172.21.15.72:0/1531362679 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffa9410a940 0x7ffa941a6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:46:02.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.946+0000 7ffa91e00640 1 --2- 172.21.15.72:0/1531362679 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffa94108d90 0x7ffa9419ef50 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ffa8400e9c0 tx=0x7ffa8400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:46:02.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.946+0000 7ffa8b400640 1 -- 172.21.15.72:0/1531362679 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa8400ccc0 con 0x7ffa94108d90 2024-04-26T23:46:02.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.946+0000 7ffa8b400640 1 -- 172.21.15.72:0/1531362679 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa840035f0 con 0x7ffa94108d90 2024-04-26T23:46:02.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.946+0000 7ffa93200640 1 -- 172.21.15.72:0/1531362679 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa941a6e60 con 0x7ffa94108d90 2024-04-26T23:46:02.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.947+0000 7ffa8b400640 1 -- 172.21.15.72:0/1531362679 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa8400ce60 con 0x7ffa94108d90 2024-04-26T23:46:02.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.947+0000 7ffa93200640 1 -- 172.21.15.72:0/1531362679 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa941a72c0 con 0x7ffa94108d90 2024-04-26T23:46:02.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.948+0000 7ffa93200640 1 -- 172.21.15.72:0/1531362679 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa941100b0 con 0x7ffa94108d90 2024-04-26T23:46:02.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.950+0000 7ffa8b400640 1 -- 172.21.15.72:0/1531362679 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ffa84004000 con 0x7ffa94108d90 2024-04-26T23:46:02.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.951+0000 7ffa8b400640 1 --2- 172.21.15.72:0/1531362679 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ffa780a3c40 0x7ffa780a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:02.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.951+0000 7ffa8b400640 1 -- 172.21.15.72:0/1531362679 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7ffa840d0e10 con 0x7ffa94108d90 2024-04-26T23:46:02.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.951+0000 7ffa89400640 1 --2- 172.21.15.72:0/1531362679 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ffa780a3c40 0x7ffa780a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:02.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.952+0000 7ffa89400640 1 --2- 172.21.15.72:0/1531362679 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ffa780a3c40 0x7ffa780a6100 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ffa941a0460 tx=0x7ffa7c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:46:02.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:02.955+0000 7ffa8b400640 1 -- 172.21.15.72:0/1531362679 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ffa840d4030 con 0x7ffa94108d90 2024-04-26T23:46:03.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:02 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/608743343' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T23:46:03.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:02 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/264878021' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T23:46:03.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.159+0000 7ffa93200640 1 -- 172.21.15.72:0/1531362679 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ffa9402dd30 con 0x7ffa94108d90 2024-04-26T23:46:03.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.162+0000 7ffa8b400640 1 -- 172.21.15.72:0/1531362679 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24841 (secure 0 0 0) 0x7ffa8409d240 con 0x7ffa94108d90 2024-04-26T23:46:03.166 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:46:03.167 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":93,"fsid":"3c8453ae-0425-11ef-bc93-c7b262605968","created":"2024-04-26T23:33:48.088300+0000","modified":"2024-04-26T23:45:00.532006+0000","last_up_change":"2024-04-26T23:42:04.482891+0000","last_in_change":"2024-04-26T23:41:44.784415+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:37:44.428727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:43:00.431829+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:43:01.872977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:43:05.512050+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e20c9ebf-56f6-47c1-a11b-e1e615af042b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6803","nonce":1639298875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6805","nonce":1639298875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6809","nonce":1639298875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6807","nonce":1639298875}]},"public_addr":"172.21.15.72:6803/1639298875","cluster_addr":"172.21.15.72:6805/1639298875","heartbeat_back_addr":"172.21.15.72:6809/1639298875","heartbeat_front_addr":"172.21.15.72:6807/1639298875","state":["exists","up"]},{"osd":1,"uuid":"e6cac469-780c-4d66-b560-b686072371fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6801","nonce":4185342193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6803","nonce":4185342193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6807","nonce":4185342193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6805","nonce":4185342193}]},"public_addr":"172.21.15.76:6801/4185342193","cluster_addr":"172.21.15.76:6803/4185342193","heartbeat_back_addr":"172.21.15.76:6807/4185342193","heartbeat_front_addr":"172.21.15.76:6805/4185342193","state":["exists","up"]},{"osd":2,"uuid":"838a49fc-18de-446c-8336-67b3e05da231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6801","nonce":3033860799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6803","nonce":3033860799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6807","nonce":3033860799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6805","nonce":3033860799}]},"public_addr":"172.21.15.136:6801/3033860799","cluster_addr":"172.21.15.136:6803/3033860799","heartbeat_back_addr":"172.21.15.136:6807/3033860799","heartbeat_front_addr":"172.21.15.136:6805/3033860799","state":["exists","up"]},{"osd":3,"uuid":"e6da89be-5ba7-48a4-9306-f75498660e0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6811","nonce":2071107815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6813","nonce":2071107815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6817","nonce":2071107815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6815","nonce":2071107815}]},"public_addr":"172.21.15.72:6811/2071107815","cluster_addr":"172.21.15.72:6813/2071107815","heartbeat_back_addr":"172.21.15.72:6817/2071107815","heartbeat_front_addr":"172.21.15.72:6815/2071107815","state":["exists","up"]},{"osd":4,"uuid":"0b17c491-d798-488f-892d-521ef2fbc51e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6809","nonce":147449238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6811","nonce":147449238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6815","nonce":147449238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6813","nonce":147449238}]},"public_addr":"172.21.15.76:6809/147449238","cluster_addr":"172.21.15.76:6811/147449238","heartbeat_back_addr":"172.21.15.76:6815/147449238","heartbeat_front_addr":"172.21.15.76:6813/147449238","state":["exists","up"]},{"osd":5,"uuid":"c7e887b3-ab38-4608-9bfd-1ed49956e8af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6809","nonce":2320149212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6811","nonce":2320149212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6815","nonce":2320149212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6813","nonce":2320149212}]},"public_addr":"172.21.15.136:6809/2320149212","cluster_addr":"172.21.15.136:6811/2320149212","heartbeat_back_addr":"172.21.15.136:6815/2320149212","heartbeat_front_addr":"172.21.15.136:6813/2320149212","state":["exists","up"]},{"osd":6,"uuid":"f0bd604b-0f57-4a38-9295-eb200264f1ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6819","nonce":389499207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6821","nonce":389499207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6825","nonce":389499207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6823","nonce":389499207}]},"public_addr":"172.21.15.72:6819/389499207","cluster_addr":"172.21.15.72:6821/389499207","heartbeat_back_addr":"172.21.15.72:6825/389499207","heartbeat_front_addr":"172.21.15.72:6823/389499207","state":["exists","up"]},{"osd":7,"uuid":"1879d4e4-038e-4cd6-a3ec-3a940bcc3797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6817","nonce":375888575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6819","nonce":375888575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6823","nonce":375888575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6821","nonce":375888575}]},"public_addr":"172.21.15.76:6817/375888575","cluster_addr":"172.21.15.76:6819/375888575","heartbeat_back_addr":"172.21.15.76:6823/375888575","heartbeat_front_addr":"172.21.15.76:6821/375888575","state":["exists","up"]},{"osd":8,"uuid":"6f404062-97e7-4ccb-a582-76dc1ffae548","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6817","nonce":3446774052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6819","nonce":3446774052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6823","nonce":3446774052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6821","nonce":3446774052}]},"public_addr":"172.21.15.136:6817/3446774052","cluster_addr":"172.21.15.136:6819/3446774052","heartbeat_back_addr":"172.21.15.136:6823/3446774052","heartbeat_front_addr":"172.21.15.136:6821/3446774052","state":["exists","up"]},{"osd":9,"uuid":"2989ca68-3164-4667-9bcf-230dcbb7b8f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6827","nonce":3423789152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6829","nonce":3423789152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6832","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6833","nonce":3423789152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6831","nonce":3423789152}]},"public_addr":"172.21.15.72:6827/3423789152","cluster_addr":"172.21.15.72:6829/3423789152","heartbeat_back_addr":"172.21.15.72:6833/3423789152","heartbeat_front_addr":"172.21.15.72:6831/3423789152","state":["exists","up"]},{"osd":10,"uuid":"9ec6d46a-0913-4208-97b5-167188ee8cb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6825","nonce":1120991574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6827","nonce":1120991574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6831","nonce":1120991574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6829","nonce":1120991574}]},"public_addr":"172.21.15.76:6825/1120991574","cluster_addr":"172.21.15.76:6827/1120991574","heartbeat_back_addr":"172.21.15.76:6831/1120991574","heartbeat_front_addr":"172.21.15.76:6829/1120991574","state":["exists","up"]},{"osd":11,"uuid":"bdd0774f-6e58-4918-b4cc-afd4f1118d3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6825","nonce":1135995193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6827","nonce":1135995193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6831","nonce":1135995193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6829","nonce":1135995193}]},"public_addr":"172.21.15.136:6825/1135995193","cluster_addr":"172.21.15.136:6827/1135995193","heartbeat_back_addr":"172.21.15.136:6831/1135995193","heartbeat_front_addr":"172.21.15.136:6829/1135995193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:36:52.128788+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:37:17.523654+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:37:41.634677+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:38:15.060797+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:38:41.234111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:39:06.783560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:39:40.884386+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:40:08.340465+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:40:34.344858+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:41:08.525309+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:41:36.782624+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:42:02.756888+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":6}]},{"pgid":"3.37","mappings":[{"from":5,"to":7}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:0/3625071330":"2024-04-27T23:34:58.266218+0000","172.21.15.72:6801/2903979134":"2024-04-27T23:34:58.266218+0000","172.21.15.72:6800/2903979134":"2024-04-27T23:34:58.266218+0000","172.21.15.72:0/1519142576":"2024-04-27T23:34:58.266218+0000","172.21.15.72:0/3669626460":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/945623297":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/3479150073":"2024-04-27T23:34:24.233319+0000","172.21.15.72:6800/4175896290":"2024-04-27T23:34:24.233319+0000","172.21.15.72:6801/4175896290":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/3783175012":"2024-04-27T23:34:58.266218+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:46:03.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.164+0000 7ffa93200640 1 -- 172.21.15.72:0/1531362679 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ffa780a3c40 msgr2=0x7ffa780a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:03.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.164+0000 7ffa93200640 1 --2- 172.21.15.72:0/1531362679 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ffa780a3c40 0x7ffa780a6100 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ffa941a0460 tx=0x7ffa7c005f90 comp rx=0 tx=0).stop 2024-04-26T23:46:03.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.164+0000 7ffa93200640 1 -- 172.21.15.72:0/1531362679 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffa94108d90 msgr2=0x7ffa9419ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:03.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.164+0000 7ffa93200640 1 --2- 172.21.15.72:0/1531362679 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffa94108d90 0x7ffa9419ef50 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ffa8400e9c0 tx=0x7ffa8400ee80 comp rx=0 tx=0).stop 2024-04-26T23:46:03.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.164+0000 7ffa93200640 1 -- 172.21.15.72:0/1531362679 shutdown_connections 2024-04-26T23:46:03.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.164+0000 7ffa93200640 1 --2- 172.21.15.72:0/1531362679 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ffa780a3c40 0x7ffa780a6100 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:03.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.164+0000 7ffa93200640 1 --2- 172.21.15.72:0/1531362679 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffa9410a940 0x7ffa941a6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:03.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.164+0000 7ffa93200640 1 --2- 172.21.15.72:0/1531362679 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa94109f80 0x7ffa9419f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:03.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.164+0000 7ffa93200640 1 --2- 172.21.15.72:0/1531362679 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffa94108d90 0x7ffa9419ef50 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:03.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.164+0000 7ffa93200640 1 -- 172.21.15.72:0/1531362679 >> 172.21.15.72:0/1531362679 conn(0x7ffa94077d30 msgr2=0x7ffa9410d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:46:03.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.165+0000 7ffa93200640 1 -- 172.21.15.72:0/1531362679 shutdown_connections 2024-04-26T23:46:03.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.165+0000 7ffa93200640 1 -- 172.21.15.72:0/1531362679 wait complete. 2024-04-26T23:46:03.189 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:37:44.428727+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T23:43:00.431829+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T23:43:01.872977+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T23:43:05.512050+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-26T23:46:03.190 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T23:46:03.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.388+0000 7f73d3400640 1 -- 172.21.15.72:0/3817818580 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73cc108f90 msgr2=0x7f73cc1093b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:03.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.388+0000 7f73d3400640 1 --2- 172.21.15.72:0/3817818580 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73cc108f90 0x7f73cc1093b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f73c0009900 tx=0x7f73c002a410 comp rx=0 tx=0).stop 2024-04-26T23:46:03.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.389+0000 7f73d3400640 1 -- 172.21.15.72:0/3817818580 shutdown_connections 2024-04-26T23:46:03.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.389+0000 7f73d3400640 1 --2- 172.21.15.72:0/3817818580 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f73cc10a180 0x7f73cc10a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:03.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.389+0000 7f73d3400640 1 --2- 172.21.15.72:0/3817818580 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73cc10ab40 0x7f73cc1113d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:03.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.389+0000 7f73d3400640 1 --2- 172.21.15.72:0/3817818580 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73cc108f90 0x7f73cc1093b0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:03.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.389+0000 7f73d3400640 1 -- 172.21.15.72:0/3817818580 >> 172.21.15.72:0/3817818580 conn(0x7f73cc077f30 msgr2=0x7f73cc07a370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:46:03.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.389+0000 7f73d3400640 1 -- 172.21.15.72:0/3817818580 shutdown_connections 2024-04-26T23:46:03.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.389+0000 7f73d3400640 1 -- 172.21.15.72:0/3817818580 wait complete. 2024-04-26T23:46:03.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.390+0000 7f73d3400640 1 Processor -- start 2024-04-26T23:46:03.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.390+0000 7f73d3400640 1 -- start start 2024-04-26T23:46:03.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.390+0000 7f73d3400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f73cc108f90 0x7f73cc19f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:03.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.390+0000 7f73d3400640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73cc10a180 0x7f73cc19f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:03.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.390+0000 7f73d3400640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73cc10ab40 0x7f73cc1a67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:03.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.390+0000 7f73d3400640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f73cc114ad0 con 0x7f73cc10ab40 2024-04-26T23:46:03.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.390+0000 7f73d3400640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f73cc114950 con 0x7f73cc108f90 2024-04-26T23:46:03.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.390+0000 7f73d3400640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f73cc114c50 con 0x7f73cc10a180 2024-04-26T23:46:03.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.391+0000 7f73d1400640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73cc10ab40 0x7f73cc1a67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:03.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.391+0000 7f73d0a00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f73cc108f90 0x7f73cc19f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:03.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.391+0000 7f73d0a00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f73cc108f90 0x7f73cc19f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:34112/0 (socket says 172.21.15.72:34112) 2024-04-26T23:46:03.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.391+0000 7f73d0a00640 1 -- 172.21.15.72:0/382771067 learned_addr learned my addr 172.21.15.72:0/382771067 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:46:03.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.391+0000 7f73cbe00640 1 --2- 172.21.15.72:0/382771067 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73cc10a180 0x7f73cc19f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:03.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.391+0000 7f73d0a00640 1 -- 172.21.15.72:0/382771067 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73cc10a180 msgr2=0x7f73cc19f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:03.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.391+0000 7f73d0a00640 1 --2- 172.21.15.72:0/382771067 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73cc10a180 0x7f73cc19f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:03.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.391+0000 7f73d0a00640 1 -- 172.21.15.72:0/382771067 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73cc10ab40 msgr2=0x7f73cc1a67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:03.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.391+0000 7f73d0a00640 1 --2- 172.21.15.72:0/382771067 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73cc10ab40 0x7f73cc1a67a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:03.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.391+0000 7f73d0a00640 1 -- 172.21.15.72:0/382771067 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73cc1a6e10 con 0x7f73cc108f90 2024-04-26T23:46:03.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.391+0000 7f73d1400640 1 --2- 172.21.15.72:0/382771067 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73cc10ab40 0x7f73cc1a67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:46:03.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.391+0000 7f73d0a00640 1 --2- 172.21.15.72:0/382771067 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f73cc108f90 0x7f73cc19f1f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f73c0006010 tx=0x7f73c00042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:46:03.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.392+0000 7f73c9600640 1 -- 172.21.15.72:0/382771067 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73c003f030 con 0x7f73cc108f90 2024-04-26T23:46:03.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.392+0000 7f73d3400640 1 -- 172.21.15.72:0/382771067 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73cc1a70a0 con 0x7f73cc108f90 2024-04-26T23:46:03.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.392+0000 7f73c9600640 1 -- 172.21.15.72:0/382771067 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73c0031910 con 0x7f73cc108f90 2024-04-26T23:46:03.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.392+0000 7f73c9600640 1 -- 172.21.15.72:0/382771067 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73c0035070 con 0x7f73cc108f90 2024-04-26T23:46:03.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.392+0000 7f73d3400640 1 -- 172.21.15.72:0/382771067 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73cc1a75e0 con 0x7f73cc108f90 2024-04-26T23:46:03.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.393+0000 7f73d3400640 1 -- 172.21.15.72:0/382771067 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73cc1103c0 con 0x7f73cc108f90 2024-04-26T23:46:03.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.396+0000 7f73c9600640 1 -- 172.21.15.72:0/382771067 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f73c0042030 con 0x7f73cc108f90 2024-04-26T23:46:03.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.397+0000 7f73c9600640 1 --2- 172.21.15.72:0/382771067 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f73b40a3c40 0x7f73b40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:03.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.397+0000 7f73c9600640 1 -- 172.21.15.72:0/382771067 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f73c00f12a0 con 0x7f73cc108f90 2024-04-26T23:46:03.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.398+0000 7f73cbe00640 1 --2- 172.21.15.72:0/382771067 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f73b40a3c40 0x7f73b40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:03.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.398+0000 7f73cbe00640 1 --2- 172.21.15.72:0/382771067 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f73b40a3c40 0x7f73b40a6100 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f73b80095f0 tx=0x7f73b8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:46:03.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.399+0000 7f73c9600640 1 -- 172.21.15.72:0/382771067 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f73c0003e60 con 0x7f73cc108f90 2024-04-26T23:46:03.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.607+0000 7f73d3400640 1 -- 172.21.15.72:0/382771067 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f73cc063530 con 0x7f73cc108f90 2024-04-26T23:46:03.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.608+0000 7f73c9600640 1 -- 172.21.15.72:0/382771067 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f73c00bd080 con 0x7f73cc108f90 2024-04-26T23:46:03.612 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-04-26T23:46:03.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.611+0000 7f73d3400640 1 -- 172.21.15.72:0/382771067 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f73b40a3c40 msgr2=0x7f73b40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:03.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.611+0000 7f73d3400640 1 --2- 172.21.15.72:0/382771067 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f73b40a3c40 0x7f73b40a6100 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f73b80095f0 tx=0x7f73b8005f90 comp rx=0 tx=0).stop 2024-04-26T23:46:03.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.611+0000 7f73d3400640 1 -- 172.21.15.72:0/382771067 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f73cc108f90 msgr2=0x7f73cc19f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:03.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.611+0000 7f73d3400640 1 --2- 172.21.15.72:0/382771067 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f73cc108f90 0x7f73cc19f1f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f73c0006010 tx=0x7f73c00042f0 comp rx=0 tx=0).stop 2024-04-26T23:46:03.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.611+0000 7f73d3400640 1 -- 172.21.15.72:0/382771067 shutdown_connections 2024-04-26T23:46:03.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.611+0000 7f73d3400640 1 --2- 172.21.15.72:0/382771067 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f73b40a3c40 0x7f73b40a6100 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:03.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.611+0000 7f73d3400640 1 --2- 172.21.15.72:0/382771067 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73cc10ab40 0x7f73cc1a67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:03.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.611+0000 7f73d3400640 1 --2- 172.21.15.72:0/382771067 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73cc10a180 0x7f73cc19f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:03.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.611+0000 7f73d3400640 1 --2- 172.21.15.72:0/382771067 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f73cc108f90 0x7f73cc19f1f0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:03.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.611+0000 7f73d3400640 1 -- 172.21.15.72:0/382771067 >> 172.21.15.72:0/382771067 conn(0x7f73cc077f30 msgr2=0x7f73cc10d590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:46:03.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.611+0000 7f73d3400640 1 -- 172.21.15.72:0/382771067 shutdown_connections 2024-04-26T23:46:03.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.611+0000 7f73d3400640 1 -- 172.21.15.72:0/382771067 wait complete. 2024-04-26T23:46:03.634 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T23:46:03.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.832+0000 7ff1e0200640 1 -- 172.21.15.72:0/3205384712 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff1d8105d80 msgr2=0x7ff1d8106200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:03.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.832+0000 7ff1e0200640 1 --2- 172.21.15.72:0/3205384712 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff1d8105d80 0x7ff1d8106200 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff1c4009900 tx=0x7ff1c402a3f0 comp rx=0 tx=0).stop 2024-04-26T23:46:03.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.832+0000 7ff1e0200640 1 -- 172.21.15.72:0/3205384712 shutdown_connections 2024-04-26T23:46:03.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.832+0000 7ff1e0200640 1 --2- 172.21.15.72:0/3205384712 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff1d8105d80 0x7ff1d8106200 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:03.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.832+0000 7ff1e0200640 1 --2- 172.21.15.72:0/3205384712 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1d8106740 0x7ff1d810cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:03.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.832+0000 7ff1e0200640 1 --2- 172.21.15.72:0/3205384712 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1d8104b90 0x7ff1d8104fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:03.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.832+0000 7ff1e0200640 1 -- 172.21.15.72:0/3205384712 >> 172.21.15.72:0/3205384712 conn(0x7ff1d81000e0 msgr2=0x7ff1d8102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:46:03.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.832+0000 7ff1e0200640 1 -- 172.21.15.72:0/3205384712 shutdown_connections 2024-04-26T23:46:03.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.832+0000 7ff1e0200640 1 -- 172.21.15.72:0/3205384712 wait complete. 2024-04-26T23:46:03.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.833+0000 7ff1e0200640 1 Processor -- start 2024-04-26T23:46:03.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.833+0000 7ff1e0200640 1 -- start start 2024-04-26T23:46:03.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.834+0000 7ff1e0200640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1d8104b90 0x7ff1d819adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:03.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.834+0000 7ff1e0200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff1d8105d80 0x7ff1d819b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:03.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.834+0000 7ff1e0200640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1d8106740 0x7ff1d81a2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:03.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.834+0000 7ff1e0200640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d8110710 con 0x7ff1d8104b90 2024-04-26T23:46:03.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.834+0000 7ff1e0200640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d8110590 con 0x7ff1d8105d80 2024-04-26T23:46:03.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.834+0000 7ff1e0200640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d8110890 con 0x7ff1d8106740 2024-04-26T23:46:03.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.834+0000 7ff1dd800640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1d8104b90 0x7ff1d819adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:03.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.834+0000 7ff1dd800640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1d8104b90 0x7ff1d819adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58042/0 (socket says 172.21.15.72:58042) 2024-04-26T23:46:03.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.834+0000 7ff1dd800640 1 -- 172.21.15.72:0/2136750648 learned_addr learned my addr 172.21.15.72:0/2136750648 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:46:03.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.834+0000 7ff1dce00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff1d8105d80 0x7ff1d819b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:03.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.834+0000 7ff1de200640 1 --2- 172.21.15.72:0/2136750648 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1d8106740 0x7ff1d81a2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:03.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.834+0000 7ff1dd800640 1 -- 172.21.15.72:0/2136750648 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1d8106740 msgr2=0x7ff1d81a2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:03.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.834+0000 7ff1dd800640 1 --2- 172.21.15.72:0/2136750648 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1d8106740 0x7ff1d81a2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:03.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.834+0000 7ff1dd800640 1 -- 172.21.15.72:0/2136750648 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff1d8105d80 msgr2=0x7ff1d819b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:03.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.834+0000 7ff1dd800640 1 --2- 172.21.15.72:0/2136750648 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff1d8105d80 0x7ff1d819b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:03.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.834+0000 7ff1dd800640 1 -- 172.21.15.72:0/2136750648 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1d81a29c0 con 0x7ff1d8104b90 2024-04-26T23:46:03.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.835+0000 7ff1dd800640 1 --2- 172.21.15.72:0/2136750648 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1d8104b90 0x7ff1d819adb0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc00e9c0 tx=0x7ff1cc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:46:03.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.835+0000 7ff1d6000640 1 -- 172.21.15.72:0/2136750648 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1cc00cd80 con 0x7ff1d8104b90 2024-04-26T23:46:03.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.835+0000 7ff1e0200640 1 -- 172.21.15.72:0/2136750648 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1d81a2cb0 con 0x7ff1d8104b90 2024-04-26T23:46:03.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.835+0000 7ff1d6000640 1 -- 172.21.15.72:0/2136750648 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1cc0035f0 con 0x7ff1d8104b90 2024-04-26T23:46:03.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.835+0000 7ff1d6000640 1 -- 172.21.15.72:0/2136750648 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1cc010690 con 0x7ff1d8104b90 2024-04-26T23:46:03.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.835+0000 7ff1e0200640 1 -- 172.21.15.72:0/2136750648 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1d81a31a0 con 0x7ff1d8104b90 2024-04-26T23:46:03.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.836+0000 7ff1e0200640 1 -- 172.21.15.72:0/2136750648 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1d810beb0 con 0x7ff1d8104b90 2024-04-26T23:46:03.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.838+0000 7ff1d6000640 1 -- 172.21.15.72:0/2136750648 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff1cc0037a0 con 0x7ff1d8104b90 2024-04-26T23:46:03.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.840+0000 7ff1d6000640 1 --2- 172.21.15.72:0/2136750648 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff1c00a3c40 0x7ff1c00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:03.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.840+0000 7ff1d6000640 1 -- 172.21.15.72:0/2136750648 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7ff1cc0d0e80 con 0x7ff1d8104b90 2024-04-26T23:46:03.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.840+0000 7ff1dce00640 1 --2- 172.21.15.72:0/2136750648 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff1c00a3c40 0x7ff1c00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:03.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.841+0000 7ff1dce00640 1 --2- 172.21.15.72:0/2136750648 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff1c00a3c40 0x7ff1c00a6100 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff1c4031500 tx=0x7ff1c4005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:46:03.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:03.843+0000 7ff1d6000640 1 -- 172.21.15.72:0/2136750648 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff1cc09d2b0 con 0x7ff1d8104b90 2024-04-26T23:46:03.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:03 smithi076 ceph-mon[32774]: pgmap v415: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:46:03.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:03 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/3217728662' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:46:03.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:03 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/1531362679' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:46:04.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:03 smithi072 ceph-mon[28184]: pgmap v415: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:46:04.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:03 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/3217728662' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:46:04.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:03 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/1531362679' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:46:04.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:03 smithi136 ceph-mon[33050]: pgmap v415: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:46:04.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:03 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/3217728662' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:46:04.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:03 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/1531362679' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:46:04.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.064+0000 7ff1e0200640 1 -- 172.21.15.72:0/2136750648 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff1d802dad0 con 0x7ff1d8104b90 2024-04-26T23:46:04.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.065+0000 7ff1d6000640 1 -- 172.21.15.72:0/2136750648 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7ff1d802dad0 con 0x7ff1d8104b90 2024-04-26T23:46:04.070 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-26T23:46:04.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.068+0000 7ff1e0200640 1 -- 172.21.15.72:0/2136750648 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff1c00a3c40 msgr2=0x7ff1c00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:04.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.068+0000 7ff1e0200640 1 --2- 172.21.15.72:0/2136750648 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff1c00a3c40 0x7ff1c00a6100 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff1c4031500 tx=0x7ff1c4005a20 comp rx=0 tx=0).stop 2024-04-26T23:46:04.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.068+0000 7ff1e0200640 1 -- 172.21.15.72:0/2136750648 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1d8104b90 msgr2=0x7ff1d819adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:04.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.068+0000 7ff1e0200640 1 --2- 172.21.15.72:0/2136750648 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1d8104b90 0x7ff1d819adb0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc00e9c0 tx=0x7ff1cc00ee80 comp rx=0 tx=0).stop 2024-04-26T23:46:04.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.068+0000 7ff1e0200640 1 -- 172.21.15.72:0/2136750648 shutdown_connections 2024-04-26T23:46:04.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.068+0000 7ff1e0200640 1 --2- 172.21.15.72:0/2136750648 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ff1c00a3c40 0x7ff1c00a6100 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:04.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.068+0000 7ff1e0200640 1 --2- 172.21.15.72:0/2136750648 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff1d8105d80 0x7ff1d819b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:04.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.068+0000 7ff1e0200640 1 --2- 172.21.15.72:0/2136750648 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1d8106740 0x7ff1d81a2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:04.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.068+0000 7ff1e0200640 1 --2- 172.21.15.72:0/2136750648 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1d8104b90 0x7ff1d819adb0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:04.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.068+0000 7ff1e0200640 1 -- 172.21.15.72:0/2136750648 >> 172.21.15.72:0/2136750648 conn(0x7ff1d81000e0 msgr2=0x7ff1d8109080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:46:04.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.068+0000 7ff1e0200640 1 -- 172.21.15.72:0/2136750648 shutdown_connections 2024-04-26T23:46:04.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.068+0000 7ff1e0200640 1 -- 172.21.15.72:0/2136750648 wait complete. 2024-04-26T23:46:04.091 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T23:46:04.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.293+0000 7fdab4a00640 1 -- 172.21.15.72:0/1237250000 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdab0102780 msgr2=0x7fdab0102ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:04.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.293+0000 7fdab4a00640 1 --2- 172.21.15.72:0/1237250000 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdab0102780 0x7fdab0102ba0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fdaa0009900 tx=0x7fdaa002a410 comp rx=0 tx=0).stop 2024-04-26T23:46:04.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.293+0000 7fdab4a00640 1 -- 172.21.15.72:0/1237250000 shutdown_connections 2024-04-26T23:46:04.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.293+0000 7fdab4a00640 1 --2- 172.21.15.72:0/1237250000 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdab0103970 0x7fdab0103df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:04.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.293+0000 7fdab4a00640 1 --2- 172.21.15.72:0/1237250000 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdab0104330 0x7fdab010abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:04.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.293+0000 7fdab4a00640 1 --2- 172.21.15.72:0/1237250000 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdab0102780 0x7fdab0102ba0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:04.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.293+0000 7fdab4a00640 1 -- 172.21.15.72:0/1237250000 >> 172.21.15.72:0/1237250000 conn(0x7fdab00fdcf0 msgr2=0x7fdab0100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:46:04.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.293+0000 7fdab4a00640 1 -- 172.21.15.72:0/1237250000 shutdown_connections 2024-04-26T23:46:04.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.293+0000 7fdab4a00640 1 -- 172.21.15.72:0/1237250000 wait complete. 2024-04-26T23:46:04.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.294+0000 7fdab4a00640 1 Processor -- start 2024-04-26T23:46:04.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.294+0000 7fdab4a00640 1 -- start start 2024-04-26T23:46:04.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.295+0000 7fdab4a00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdab0102780 0x7fdab019ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:04.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.295+0000 7fdab4a00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdab0103970 0x7fdab019b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:04.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.295+0000 7fdab4a00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdab0104330 0x7fdab01a20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:04.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.295+0000 7fdab4a00640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fdab010e2b0 con 0x7fdab0103970 2024-04-26T23:46:04.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.295+0000 7fdab4a00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fdab010e130 con 0x7fdab0104330 2024-04-26T23:46:04.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.295+0000 7fdab4a00640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fdab010e430 con 0x7fdab0102780 2024-04-26T23:46:04.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.295+0000 7fdaae800640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdab0102780 0x7fdab019ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:04.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.295+0000 7fdaae800640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdab0102780 0x7fdab019ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.72:44684/0 (socket says 172.21.15.72:44684) 2024-04-26T23:46:04.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.295+0000 7fdaae800640 1 -- 172.21.15.72:0/3055164637 learned_addr learned my addr 172.21.15.72:0/3055164637 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:46:04.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.295+0000 7fdaaf200640 1 --2- 172.21.15.72:0/3055164637 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdab0104330 0x7fdab01a20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:04.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.295+0000 7fdaade00640 1 --2- 172.21.15.72:0/3055164637 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdab0103970 0x7fdab019b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:04.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.296+0000 7fdaaf200640 1 -- 172.21.15.72:0/3055164637 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdab0102780 msgr2=0x7fdab019ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:04.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.296+0000 7fdaaf200640 1 --2- 172.21.15.72:0/3055164637 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdab0102780 0x7fdab019ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:04.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.296+0000 7fdaaf200640 1 -- 172.21.15.72:0/3055164637 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdab0103970 msgr2=0x7fdab019b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:04.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.296+0000 7fdaaf200640 1 --2- 172.21.15.72:0/3055164637 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdab0103970 0x7fdab019b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:04.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.296+0000 7fdaaf200640 1 -- 172.21.15.72:0/3055164637 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdab01a2760 con 0x7fdab0104330 2024-04-26T23:46:04.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.296+0000 7fdaae800640 1 --2- 172.21.15.72:0/3055164637 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdab0102780 0x7fdab019ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:46:04.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.296+0000 7fdaaf200640 1 --2- 172.21.15.72:0/3055164637 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdab0104330 0x7fdab01a20f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fdaa8009fd0 tx=0x7fdaa800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:46:04.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.296+0000 7fdaa7400640 1 -- 172.21.15.72:0/3055164637 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa800ce20 con 0x7fdab0104330 2024-04-26T23:46:04.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.296+0000 7fdab4a00640 1 -- 172.21.15.72:0/3055164637 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdab01a2a50 con 0x7fdab0104330 2024-04-26T23:46:04.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.296+0000 7fdaa7400640 1 -- 172.21.15.72:0/3055164637 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdaa8005020 con 0x7fdab0104330 2024-04-26T23:46:04.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.296+0000 7fdaa7400640 1 -- 172.21.15.72:0/3055164637 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa8010690 con 0x7fdab0104330 2024-04-26T23:46:04.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.296+0000 7fdab4a00640 1 -- 172.21.15.72:0/3055164637 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdab01a2fa0 con 0x7fdab0104330 2024-04-26T23:46:04.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.297+0000 7fdab4a00640 1 -- 172.21.15.72:0/3055164637 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdab01a3230 con 0x7fdab0104330 2024-04-26T23:46:04.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.299+0000 7fdaa7400640 1 -- 172.21.15.72:0/3055164637 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fdaa80054d0 con 0x7fdab0104330 2024-04-26T23:46:04.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.301+0000 7fdaa7400640 1 --2- 172.21.15.72:0/3055164637 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fda940a3c40 0x7fda940a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:04.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.301+0000 7fdaa7400640 1 -- 172.21.15.72:0/3055164637 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7fdaa80d0ff0 con 0x7fdab0104330 2024-04-26T23:46:04.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.301+0000 7fdaae800640 1 --2- 172.21.15.72:0/3055164637 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fda940a3c40 0x7fda940a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:04.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.302+0000 7fdaae800640 1 --2- 172.21.15.72:0/3055164637 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fda940a3c40 0x7fda940a6100 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fdaa0032fd0 tx=0x7fdaa00314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:46:04.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.303+0000 7fdaa7400640 1 -- 172.21.15.72:0/3055164637 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fdaa809d460 con 0x7fdab0104330 2024-04-26T23:46:04.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.491+0000 7fdab4a00640 1 -- 172.21.15.72:0/3055164637 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fdab01a3400 con 0x7fdab0104330 2024-04-26T23:46:04.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.492+0000 7fdaa7400640 1 -- 172.21.15.72:0/3055164637 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7fdaa809cd60 con 0x7fdab0104330 2024-04-26T23:46:04.496 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-04-26T23:46:04.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.494+0000 7fdab4a00640 1 -- 172.21.15.72:0/3055164637 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fda940a3c40 msgr2=0x7fda940a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:04.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.494+0000 7fdab4a00640 1 --2- 172.21.15.72:0/3055164637 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fda940a3c40 0x7fda940a6100 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fdaa0032fd0 tx=0x7fdaa00314a0 comp rx=0 tx=0).stop 2024-04-26T23:46:04.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.494+0000 7fdab4a00640 1 -- 172.21.15.72:0/3055164637 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdab0104330 msgr2=0x7fdab01a20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:04.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.494+0000 7fdab4a00640 1 --2- 172.21.15.72:0/3055164637 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdab0104330 0x7fdab01a20f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fdaa8009fd0 tx=0x7fdaa800ef10 comp rx=0 tx=0).stop 2024-04-26T23:46:04.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.494+0000 7fdab4a00640 1 -- 172.21.15.72:0/3055164637 shutdown_connections 2024-04-26T23:46:04.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.494+0000 7fdab4a00640 1 --2- 172.21.15.72:0/3055164637 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fda940a3c40 0x7fda940a6100 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:04.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.494+0000 7fdab4a00640 1 --2- 172.21.15.72:0/3055164637 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdab0104330 0x7fdab01a20f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:04.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.494+0000 7fdab4a00640 1 --2- 172.21.15.72:0/3055164637 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdab0103970 0x7fdab019b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:04.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.495+0000 7fdab4a00640 1 --2- 172.21.15.72:0/3055164637 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdab0102780 0x7fdab019ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:04.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.495+0000 7fdab4a00640 1 -- 172.21.15.72:0/3055164637 >> 172.21.15.72:0/3055164637 conn(0x7fdab00fdcf0 msgr2=0x7fdab0074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:46:04.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.495+0000 7fdab4a00640 1 -- 172.21.15.72:0/3055164637 shutdown_connections 2024-04-26T23:46:04.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.495+0000 7fdab4a00640 1 -- 172.21.15.72:0/3055164637 wait complete. 2024-04-26T23:46:04.516 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-26T23:46:04.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.709+0000 7fce1cc00640 1 -- 172.21.15.72:0/2830191654 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fce18101a20 msgr2=0x7fce18101e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:04.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.709+0000 7fce1cc00640 1 --2- 172.21.15.72:0/2830191654 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fce18101a20 0x7fce18101e80 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fce04009900 tx=0x7fce0402a410 comp rx=0 tx=0).stop 2024-04-26T23:46:04.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.709+0000 7fce1cc00640 1 -- 172.21.15.72:0/2830191654 shutdown_connections 2024-04-26T23:46:04.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.709+0000 7fce1cc00640 1 --2- 172.21.15.72:0/2830191654 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fce181023c0 0x7fce18102840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:04.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.709+0000 7fce1cc00640 1 --2- 172.21.15.72:0/2830191654 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fce18101a20 0x7fce18101e80 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:04.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.709+0000 7fce1cc00640 1 --2- 172.21.15.72:0/2830191654 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce181010c0 0x7fce181014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:04.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.709+0000 7fce1cc00640 1 -- 172.21.15.72:0/2830191654 >> 172.21.15.72:0/2830191654 conn(0x7fce180fcce0 msgr2=0x7fce180ff100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:46:04.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.709+0000 7fce1cc00640 1 -- 172.21.15.72:0/2830191654 shutdown_connections 2024-04-26T23:46:04.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.709+0000 7fce1cc00640 1 -- 172.21.15.72:0/2830191654 wait complete. 2024-04-26T23:46:04.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.710+0000 7fce1cc00640 1 Processor -- start 2024-04-26T23:46:04.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.711+0000 7fce1cc00640 1 -- start start 2024-04-26T23:46:04.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.711+0000 7fce1cc00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fce181010c0 0x7fce1819ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:04.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.711+0000 7fce1cc00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fce18101a20 0x7fce1819b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:04.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.711+0000 7fce1cc00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce181023c0 0x7fce181a21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:04.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.711+0000 7fce1cc00640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fce18110540 con 0x7fce18101a20 2024-04-26T23:46:04.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.711+0000 7fce1cc00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fce181103c0 con 0x7fce181010c0 2024-04-26T23:46:04.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.711+0000 7fce1cc00640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fce181106c0 con 0x7fce181023c0 2024-04-26T23:46:04.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.711+0000 7fce16a00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fce18101a20 0x7fce1819b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:04.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.711+0000 7fce16a00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fce18101a20 0x7fce1819b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58086/0 (socket says 172.21.15.72:58086) 2024-04-26T23:46:04.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.711+0000 7fce16a00640 1 -- 172.21.15.72:0/2754589002 learned_addr learned my addr 172.21.15.72:0/2754589002 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:46:04.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.711+0000 7fce17400640 1 --2- 172.21.15.72:0/2754589002 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fce181010c0 0x7fce1819ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:04.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.712+0000 7fce17e00640 1 --2- 172.21.15.72:0/2754589002 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce181023c0 0x7fce181a21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:04.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.712+0000 7fce16a00640 1 -- 172.21.15.72:0/2754589002 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce181023c0 msgr2=0x7fce181a21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:04.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.712+0000 7fce16a00640 1 --2- 172.21.15.72:0/2754589002 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce181023c0 0x7fce181a21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:04.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.712+0000 7fce16a00640 1 -- 172.21.15.72:0/2754589002 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fce181010c0 msgr2=0x7fce1819ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:04.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.712+0000 7fce16a00640 1 --2- 172.21.15.72:0/2754589002 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fce181010c0 0x7fce1819ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:04.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.712+0000 7fce16a00640 1 -- 172.21.15.72:0/2754589002 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce181a2880 con 0x7fce18101a20 2024-04-26T23:46:04.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.712+0000 7fce16a00640 1 --2- 172.21.15.72:0/2754589002 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fce18101a20 0x7fce1819b1b0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fce04005940 tx=0x7fce04031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:46:04.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.713+0000 7fce0fe00640 1 -- 172.21.15.72:0/2754589002 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce0403f030 con 0x7fce18101a20 2024-04-26T23:46:04.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.713+0000 7fce0fe00640 1 -- 172.21.15.72:0/2754589002 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce04003aa0 con 0x7fce18101a20 2024-04-26T23:46:04.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.713+0000 7fce1cc00640 1 -- 172.21.15.72:0/2754589002 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce181a2b10 con 0x7fce18101a20 2024-04-26T23:46:04.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.713+0000 7fce0fe00640 1 -- 172.21.15.72:0/2754589002 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce04035070 con 0x7fce18101a20 2024-04-26T23:46:04.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.713+0000 7fce1cc00640 1 -- 172.21.15.72:0/2754589002 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce181a3000 con 0x7fce18101a20 2024-04-26T23:46:04.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.714+0000 7fce1cc00640 1 -- 172.21.15.72:0/2754589002 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce1810c050 con 0x7fce18101a20 2024-04-26T23:46:04.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.716+0000 7fce0fe00640 1 -- 172.21.15.72:0/2754589002 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fce04042030 con 0x7fce18101a20 2024-04-26T23:46:04.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.717+0000 7fce0fe00640 1 --2- 172.21.15.72:0/2754589002 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fcdfc0a3c40 0x7fcdfc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:04.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.717+0000 7fce0fe00640 1 -- 172.21.15.72:0/2754589002 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7fce040f0f90 con 0x7fce18101a20 2024-04-26T23:46:04.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.717+0000 7fce17400640 1 --2- 172.21.15.72:0/2754589002 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fcdfc0a3c40 0x7fcdfc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:04.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.718+0000 7fce17400640 1 --2- 172.21.15.72:0/2754589002 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fcdfc0a3c40 0x7fcdfc0a6100 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fce1819c140 tx=0x7fce0800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:46:04.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.721+0000 7fce0fe00640 1 -- 172.21.15.72:0/2754589002 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fce040bd420 con 0x7fce18101a20 2024-04-26T23:46:04.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:04 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/382771067' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T23:46:04.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:04 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/2136750648' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T23:46:04.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:04 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/3055164637' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T23:46:04.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.922+0000 7fce1cc00640 1 -- 172.21.15.72:0/2754589002 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fce1802dd20 con 0x7fce18101a20 2024-04-26T23:46:04.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.923+0000 7fce0fe00640 1 -- 172.21.15.72:0/2754589002 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7fce040bcd20 con 0x7fce18101a20 2024-04-26T23:46:04.927 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-26T23:46:04.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:04 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/382771067' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T23:46:04.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:04 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/2136750648' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T23:46:04.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:04 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/3055164637' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T23:46:04.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.925+0000 7fce1cc00640 1 -- 172.21.15.72:0/2754589002 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fcdfc0a3c40 msgr2=0x7fcdfc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:04.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.925+0000 7fce1cc00640 1 --2- 172.21.15.72:0/2754589002 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fcdfc0a3c40 0x7fcdfc0a6100 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fce1819c140 tx=0x7fce0800b390 comp rx=0 tx=0).stop 2024-04-26T23:46:04.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.926+0000 7fce1cc00640 1 -- 172.21.15.72:0/2754589002 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fce18101a20 msgr2=0x7fce1819b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:04.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.926+0000 7fce1cc00640 1 --2- 172.21.15.72:0/2754589002 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fce18101a20 0x7fce1819b1b0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fce04005940 tx=0x7fce04031a50 comp rx=0 tx=0).stop 2024-04-26T23:46:04.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.926+0000 7fce1cc00640 1 -- 172.21.15.72:0/2754589002 shutdown_connections 2024-04-26T23:46:04.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.926+0000 7fce1cc00640 1 --2- 172.21.15.72:0/2754589002 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fcdfc0a3c40 0x7fcdfc0a6100 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:04.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.926+0000 7fce1cc00640 1 --2- 172.21.15.72:0/2754589002 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce181023c0 0x7fce181a21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:04.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.926+0000 7fce1cc00640 1 --2- 172.21.15.72:0/2754589002 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fce18101a20 0x7fce1819b1b0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:04.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.926+0000 7fce1cc00640 1 --2- 172.21.15.72:0/2754589002 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fce181010c0 0x7fce1819ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:04.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.926+0000 7fce1cc00640 1 -- 172.21.15.72:0/2754589002 >> 172.21.15.72:0/2754589002 conn(0x7fce180fcce0 msgr2=0x7fce180fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:46:04.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.926+0000 7fce1cc00640 1 -- 172.21.15.72:0/2754589002 shutdown_connections 2024-04-26T23:46:04.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:04.926+0000 7fce1cc00640 1 -- 172.21.15.72:0/2754589002 wait complete. 2024-04-26T23:46:04.949 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T23:46:05.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:04 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/382771067' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T23:46:05.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:04 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/2136750648' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T23:46:05.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:04 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/3055164637' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T23:46:05.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.145+0000 7f4b72000640 1 -- 172.21.15.72:0/3640015373 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b6c104b90 msgr2=0x7f4b6c104fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:05.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.145+0000 7f4b72000640 1 --2- 172.21.15.72:0/3640015373 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b6c104b90 0x7f4b6c104fb0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c009900 tx=0x7f4b5c02a3f0 comp rx=0 tx=0).stop 2024-04-26T23:46:05.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.145+0000 7f4b72000640 1 -- 172.21.15.72:0/3640015373 shutdown_connections 2024-04-26T23:46:05.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.145+0000 7f4b72000640 1 --2- 172.21.15.72:0/3640015373 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4b6c106740 0x7f4b6c10cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:05.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.145+0000 7f4b72000640 1 --2- 172.21.15.72:0/3640015373 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b6c105d80 0x7f4b6c106200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:05.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.145+0000 7f4b72000640 1 --2- 172.21.15.72:0/3640015373 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b6c104b90 0x7f4b6c104fb0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:05.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.145+0000 7f4b72000640 1 -- 172.21.15.72:0/3640015373 >> 172.21.15.72:0/3640015373 conn(0x7f4b6c100120 msgr2=0x7f4b6c102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:46:05.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.145+0000 7f4b72000640 1 -- 172.21.15.72:0/3640015373 shutdown_connections 2024-04-26T23:46:05.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.145+0000 7f4b72000640 1 -- 172.21.15.72:0/3640015373 wait complete. 2024-04-26T23:46:05.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.146+0000 7f4b72000640 1 Processor -- start 2024-04-26T23:46:05.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.147+0000 7f4b72000640 1 -- start start 2024-04-26T23:46:05.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.147+0000 7f4b72000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4b6c104b90 0x7f4b6c19adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:05.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.147+0000 7f4b72000640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b6c105d80 0x7f4b6c19b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:05.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.147+0000 7f4b72000640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b6c106740 0x7f4b6c1a2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:05.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.147+0000 7f4b72000640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6c1106e0 con 0x7f4b6c106740 2024-04-26T23:46:05.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.147+0000 7f4b72000640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6c110560 con 0x7f4b6c104b90 2024-04-26T23:46:05.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.147+0000 7f4b72000640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6c110860 con 0x7f4b6c105d80 2024-04-26T23:46:05.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.147+0000 7f4b6be00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b6c106740 0x7f4b6c1a2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:05.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.147+0000 7f4b6be00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b6c106740 0x7f4b6c1a2330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58094/0 (socket says 172.21.15.72:58094) 2024-04-26T23:46:05.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.147+0000 7f4b6be00640 1 -- 172.21.15.72:0/3577055942 learned_addr learned my addr 172.21.15.72:0/3577055942 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:46:05.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.147+0000 7f4b6b400640 1 --2- 172.21.15.72:0/3577055942 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4b6c104b90 0x7f4b6c19adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:05.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.147+0000 7f4b63e00640 1 --2- 172.21.15.72:0/3577055942 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b6c105d80 0x7f4b6c19b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:05.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.148+0000 7f4b6be00640 1 -- 172.21.15.72:0/3577055942 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b6c105d80 msgr2=0x7f4b6c19b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:05.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.148+0000 7f4b6be00640 1 --2- 172.21.15.72:0/3577055942 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b6c105d80 0x7f4b6c19b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:05.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.148+0000 7f4b6be00640 1 -- 172.21.15.72:0/3577055942 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4b6c104b90 msgr2=0x7f4b6c19adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:05.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.148+0000 7f4b6be00640 1 --2- 172.21.15.72:0/3577055942 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4b6c104b90 0x7f4b6c19adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:05.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.148+0000 7f4b6be00640 1 -- 172.21.15.72:0/3577055942 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b6c1a29d0 con 0x7f4b6c106740 2024-04-26T23:46:05.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.148+0000 7f4b6b400640 1 --2- 172.21.15.72:0/3577055942 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4b6c104b90 0x7f4b6c19adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:46:05.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.148+0000 7f4b6be00640 1 --2- 172.21.15.72:0/3577055942 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b6c106740 0x7f4b6c1a2330 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4b6400eb10 tx=0x7f4b6400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:46:05.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.148+0000 7f4b68c00640 1 -- 172.21.15.72:0/3577055942 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b64004b20 con 0x7f4b6c106740 2024-04-26T23:46:05.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.148+0000 7f4b68c00640 1 -- 172.21.15.72:0/3577055942 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b64004cc0 con 0x7f4b6c106740 2024-04-26T23:46:05.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.148+0000 7f4b72000640 1 -- 172.21.15.72:0/3577055942 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b6c1a2cc0 con 0x7f4b6c106740 2024-04-26T23:46:05.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.148+0000 7f4b68c00640 1 -- 172.21.15.72:0/3577055942 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b64010670 con 0x7f4b6c106740 2024-04-26T23:46:05.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.148+0000 7f4b72000640 1 -- 172.21.15.72:0/3577055942 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b6c1a3190 con 0x7f4b6c106740 2024-04-26T23:46:05.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.149+0000 7f4b72000640 1 -- 172.21.15.72:0/3577055942 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b6c10bf80 con 0x7f4b6c106740 2024-04-26T23:46:05.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.151+0000 7f4b68c00640 1 -- 172.21.15.72:0/3577055942 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4b64005020 con 0x7f4b6c106740 2024-04-26T23:46:05.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.153+0000 7f4b68c00640 1 --2- 172.21.15.72:0/3577055942 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f4b500a3c40 0x7f4b500a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:05.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.153+0000 7f4b68c00640 1 -- 172.21.15.72:0/3577055942 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f4b640d0fe0 con 0x7f4b6c106740 2024-04-26T23:46:05.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.153+0000 7f4b6b400640 1 --2- 172.21.15.72:0/3577055942 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f4b500a3c40 0x7f4b500a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:05.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.154+0000 7f4b6b400640 1 --2- 172.21.15.72:0/3577055942 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f4b500a3c40 0x7f4b500a6100 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c005ab0 tx=0x7f4b5c005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:46:05.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.157+0000 7f4b68c00640 1 -- 172.21.15.72:0/3577055942 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4b6409d450 con 0x7f4b6c106740 2024-04-26T23:46:05.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.382+0000 7f4b72000640 1 -- 172.21.15.72:0/3577055942 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4b6c02dd20 con 0x7f4b6c106740 2024-04-26T23:46:05.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.384+0000 7f4b68c00640 1 -- 172.21.15.72:0/3577055942 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11874 (secure 0 0 0) 0x7f4b6409cd50 con 0x7f4b6c106740 2024-04-26T23:46:05.389 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:46:05.389 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/2386456166","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2386456166},{"type":"v1","addr":"172.21.15.72:6835","nonce":2386456166}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1528342053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1528342053},{"type":"v1","addr":"172.21.15.136:6833","nonce":1528342053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6833/2427238559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":2427238559},{"type":"v1","addr":"172.21.15.76:6833","nonce":2427238559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24491,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/4050689894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":4050689894},{"type":"v1","addr":"172.21.15.72:6837","nonce":4050689894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3681925367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3681925367},{"type":"v1","addr":"172.21.15.136:6835","nonce":3681925367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4251109085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4251109085},{"type":"v1","addr":"172.21.15.76:6835","nonce":4251109085}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T23:43:02.928649+0000","modified":"2024-04-26T23:43:30.190394+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24590,"mds_1":24571,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24514":{"gid":24514,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6837/3639308555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3639308555},{"type":"v1","addr":"172.21.15.136:6837","nonce":3639308555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.76:6837/2506930945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2506930945},{"type":"v1","addr":"172.21.15.76:6837","nonce":2506930945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6839/3938677630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3938677630},{"type":"v1","addr":"172.21.15.136:6839","nonce":3938677630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.72:6839/1403162102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6838","nonce":1403162102},{"type":"v1","addr":"172.21.15.72:6839","nonce":1403162102}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24571":{"gid":24571,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.72:6841/1356345077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6840","nonce":1356345077},{"type":"v1","addr":"172.21.15.72:6841","nonce":1356345077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.76:6839/3405829058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":3405829058},{"type":"v1","addr":"172.21.15.76:6839","nonce":3405829058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24553,24571,24590]},"id":1}]} 2024-04-26T23:46:05.389 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 23 2024-04-26T23:46:05.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.387+0000 7f4b72000640 1 -- 172.21.15.72:0/3577055942 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f4b500a3c40 msgr2=0x7f4b500a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:05.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.387+0000 7f4b72000640 1 --2- 172.21.15.72:0/3577055942 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f4b500a3c40 0x7f4b500a6100 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c005ab0 tx=0x7f4b5c005a20 comp rx=0 tx=0).stop 2024-04-26T23:46:05.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.387+0000 7f4b72000640 1 -- 172.21.15.72:0/3577055942 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b6c106740 msgr2=0x7f4b6c1a2330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:05.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.387+0000 7f4b72000640 1 --2- 172.21.15.72:0/3577055942 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b6c106740 0x7f4b6c1a2330 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4b6400eb10 tx=0x7f4b6400c2d0 comp rx=0 tx=0).stop 2024-04-26T23:46:05.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.387+0000 7f4b72000640 1 -- 172.21.15.72:0/3577055942 shutdown_connections 2024-04-26T23:46:05.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.387+0000 7f4b72000640 1 --2- 172.21.15.72:0/3577055942 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f4b500a3c40 0x7f4b500a6100 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:05.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.387+0000 7f4b72000640 1 --2- 172.21.15.72:0/3577055942 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b6c106740 0x7f4b6c1a2330 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:05.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.387+0000 7f4b72000640 1 --2- 172.21.15.72:0/3577055942 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b6c105d80 0x7f4b6c19b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:05.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.387+0000 7f4b72000640 1 --2- 172.21.15.72:0/3577055942 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4b6c104b90 0x7f4b6c19adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:05.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.387+0000 7f4b72000640 1 -- 172.21.15.72:0/3577055942 >> 172.21.15.72:0/3577055942 conn(0x7f4b6c100120 msgr2=0x7f4b6c109150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:46:05.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.387+0000 7f4b72000640 1 -- 172.21.15.72:0/3577055942 shutdown_connections 2024-04-26T23:46:05.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.388+0000 7f4b72000640 1 -- 172.21.15.72:0/3577055942 wait complete. 2024-04-26T23:46:05.409 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T23:46:05.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.593+0000 7f149f600640 1 -- 172.21.15.72:0/452267751 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1498104500 msgr2=0x7f149810ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:05.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.593+0000 7f149f600640 1 --2- 172.21.15.72:0/452267751 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1498104500 0x7f149810ad90 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1490009900 tx=0x7f149002a3f0 comp rx=0 tx=0).stop 2024-04-26T23:46:05.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.593+0000 7f149f600640 1 -- 172.21.15.72:0/452267751 shutdown_connections 2024-04-26T23:46:05.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.593+0000 7f149f600640 1 --2- 172.21.15.72:0/452267751 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1498103b40 0x7f1498103fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:05.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.593+0000 7f149f600640 1 --2- 172.21.15.72:0/452267751 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1498104500 0x7f149810ad90 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:05.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.593+0000 7f149f600640 1 --2- 172.21.15.72:0/452267751 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1498102950 0x7f1498102d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:05.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.593+0000 7f149f600640 1 -- 172.21.15.72:0/452267751 >> 172.21.15.72:0/452267751 conn(0x7f14980fdee0 msgr2=0x7f1498100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:46:05.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.593+0000 7f149f600640 1 -- 172.21.15.72:0/452267751 shutdown_connections 2024-04-26T23:46:05.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.593+0000 7f149f600640 1 -- 172.21.15.72:0/452267751 wait complete. 2024-04-26T23:46:05.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.594+0000 7f149f600640 1 Processor -- start 2024-04-26T23:46:05.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.594+0000 7f149f600640 1 -- start start 2024-04-26T23:46:05.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.594+0000 7f149f600640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1498102950 0x7f149819adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:05.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.595+0000 7f149f600640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1498103b40 0x7f149819b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:05.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.595+0000 7f149f600640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1498104500 0x7f14981a2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:05.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.595+0000 7f149f600640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f149810e4d0 con 0x7f1498104500 2024-04-26T23:46:05.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.595+0000 7f149f600640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f149810e350 con 0x7f1498103b40 2024-04-26T23:46:05.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.595+0000 7f149f600640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f149810e650 con 0x7f1498102950 2024-04-26T23:46:05.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.595+0000 7f149cc00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1498102950 0x7f149819adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:05.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.595+0000 7f149cc00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1498102950 0x7f149819adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.72:44736/0 (socket says 172.21.15.72:44736) 2024-04-26T23:46:05.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.595+0000 7f149cc00640 1 -- 172.21.15.72:0/881424407 learned_addr learned my addr 172.21.15.72:0/881424407 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:46:05.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.595+0000 7f149d600640 1 --2- 172.21.15.72:0/881424407 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1498104500 0x7f14981a2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:05.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.595+0000 7f1497e00640 1 --2- 172.21.15.72:0/881424407 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1498103b40 0x7f149819b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:05.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.595+0000 7f149cc00640 1 -- 172.21.15.72:0/881424407 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1498103b40 msgr2=0x7f149819b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:05.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.595+0000 7f149cc00640 1 --2- 172.21.15.72:0/881424407 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1498103b40 0x7f149819b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:05.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.595+0000 7f149cc00640 1 -- 172.21.15.72:0/881424407 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1498104500 msgr2=0x7f14981a2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:05.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.595+0000 7f149cc00640 1 --2- 172.21.15.72:0/881424407 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1498104500 0x7f14981a2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:05.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.595+0000 7f149cc00640 1 -- 172.21.15.72:0/881424407 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14981a29c0 con 0x7f1498102950 2024-04-26T23:46:05.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.596+0000 7f149cc00640 1 --2- 172.21.15.72:0/881424407 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1498102950 0x7f149819adb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f148000e9c0 tx=0x7f148000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:46:05.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.596+0000 7f1495600640 1 -- 172.21.15.72:0/881424407 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f148000cd80 con 0x7f1498102950 2024-04-26T23:46:05.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.596+0000 7f1495600640 1 -- 172.21.15.72:0/881424407 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14800035f0 con 0x7f1498102950 2024-04-26T23:46:05.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.596+0000 7f149f600640 1 -- 172.21.15.72:0/881424407 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14981a2cb0 con 0x7f1498102950 2024-04-26T23:46:05.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.596+0000 7f1495600640 1 -- 172.21.15.72:0/881424407 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1480010690 con 0x7f1498102950 2024-04-26T23:46:05.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.596+0000 7f149f600640 1 -- 172.21.15.72:0/881424407 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14981a31b0 con 0x7f1498102950 2024-04-26T23:46:05.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.597+0000 7f149f600640 1 -- 172.21.15.72:0/881424407 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14981a33e0 con 0x7f1498102950 2024-04-26T23:46:05.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.599+0000 7f1495600640 1 -- 172.21.15.72:0/881424407 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1480004000 con 0x7f1498102950 2024-04-26T23:46:05.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.601+0000 7f1495600640 1 --2- 172.21.15.72:0/881424407 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f147c0a3cc0 0x7f147c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:05.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.601+0000 7f1495600640 1 -- 172.21.15.72:0/881424407 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f14800d0ed0 con 0x7f1498102950 2024-04-26T23:46:05.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.601+0000 7f1497e00640 1 --2- 172.21.15.72:0/881424407 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f147c0a3cc0 0x7f147c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:05.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.602+0000 7f1497e00640 1 --2- 172.21.15.72:0/881424407 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f147c0a3cc0 0x7f147c0a6180 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f1488005fd0 tx=0x7f1488009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:46:05.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.605+0000 7f1495600640 1 -- 172.21.15.72:0/881424407 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f148009d300 con 0x7f1498102950 2024-04-26T23:46:05.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:05 smithi072 ceph-mon[28184]: pgmap v416: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:46:05.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:05 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/2754589002' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T23:46:05.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:05 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/3577055942' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:46:05.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.793+0000 7f149f600640 1 -- 172.21.15.72:0/881424407 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f14981a3650 con 0x7f1498102950 2024-04-26T23:46:05.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.795+0000 7f1495600640 1 -- 172.21.15.72:0/881424407 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24841 (secure 0 0 0) 0x7f148009cc00 con 0x7f1498102950 2024-04-26T23:46:05.800 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-26T23:46:05.800 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":93,"fsid":"3c8453ae-0425-11ef-bc93-c7b262605968","created":"2024-04-26T23:33:48.088300+0000","modified":"2024-04-26T23:45:00.532006+0000","last_up_change":"2024-04-26T23:42:04.482891+0000","last_in_change":"2024-04-26T23:41:44.784415+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:37:44.428727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:43:00.431829+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:43:01.872977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:43:05.512050+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e20c9ebf-56f6-47c1-a11b-e1e615af042b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6803","nonce":1639298875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6805","nonce":1639298875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6809","nonce":1639298875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6807","nonce":1639298875}]},"public_addr":"172.21.15.72:6803/1639298875","cluster_addr":"172.21.15.72:6805/1639298875","heartbeat_back_addr":"172.21.15.72:6809/1639298875","heartbeat_front_addr":"172.21.15.72:6807/1639298875","state":["exists","up"]},{"osd":1,"uuid":"e6cac469-780c-4d66-b560-b686072371fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6801","nonce":4185342193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6803","nonce":4185342193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6807","nonce":4185342193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6805","nonce":4185342193}]},"public_addr":"172.21.15.76:6801/4185342193","cluster_addr":"172.21.15.76:6803/4185342193","heartbeat_back_addr":"172.21.15.76:6807/4185342193","heartbeat_front_addr":"172.21.15.76:6805/4185342193","state":["exists","up"]},{"osd":2,"uuid":"838a49fc-18de-446c-8336-67b3e05da231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6801","nonce":3033860799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6803","nonce":3033860799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6807","nonce":3033860799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6805","nonce":3033860799}]},"public_addr":"172.21.15.136:6801/3033860799","cluster_addr":"172.21.15.136:6803/3033860799","heartbeat_back_addr":"172.21.15.136:6807/3033860799","heartbeat_front_addr":"172.21.15.136:6805/3033860799","state":["exists","up"]},{"osd":3,"uuid":"e6da89be-5ba7-48a4-9306-f75498660e0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6811","nonce":2071107815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6813","nonce":2071107815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6817","nonce":2071107815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6815","nonce":2071107815}]},"public_addr":"172.21.15.72:6811/2071107815","cluster_addr":"172.21.15.72:6813/2071107815","heartbeat_back_addr":"172.21.15.72:6817/2071107815","heartbeat_front_addr":"172.21.15.72:6815/2071107815","state":["exists","up"]},{"osd":4,"uuid":"0b17c491-d798-488f-892d-521ef2fbc51e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6809","nonce":147449238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6811","nonce":147449238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6815","nonce":147449238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6813","nonce":147449238}]},"public_addr":"172.21.15.76:6809/147449238","cluster_addr":"172.21.15.76:6811/147449238","heartbeat_back_addr":"172.21.15.76:6815/147449238","heartbeat_front_addr":"172.21.15.76:6813/147449238","state":["exists","up"]},{"osd":5,"uuid":"c7e887b3-ab38-4608-9bfd-1ed49956e8af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6809","nonce":2320149212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6811","nonce":2320149212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6815","nonce":2320149212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6813","nonce":2320149212}]},"public_addr":"172.21.15.136:6809/2320149212","cluster_addr":"172.21.15.136:6811/2320149212","heartbeat_back_addr":"172.21.15.136:6815/2320149212","heartbeat_front_addr":"172.21.15.136:6813/2320149212","state":["exists","up"]},{"osd":6,"uuid":"f0bd604b-0f57-4a38-9295-eb200264f1ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6819","nonce":389499207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6821","nonce":389499207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6825","nonce":389499207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6823","nonce":389499207}]},"public_addr":"172.21.15.72:6819/389499207","cluster_addr":"172.21.15.72:6821/389499207","heartbeat_back_addr":"172.21.15.72:6825/389499207","heartbeat_front_addr":"172.21.15.72:6823/389499207","state":["exists","up"]},{"osd":7,"uuid":"1879d4e4-038e-4cd6-a3ec-3a940bcc3797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6817","nonce":375888575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6819","nonce":375888575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6823","nonce":375888575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6821","nonce":375888575}]},"public_addr":"172.21.15.76:6817/375888575","cluster_addr":"172.21.15.76:6819/375888575","heartbeat_back_addr":"172.21.15.76:6823/375888575","heartbeat_front_addr":"172.21.15.76:6821/375888575","state":["exists","up"]},{"osd":8,"uuid":"6f404062-97e7-4ccb-a582-76dc1ffae548","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6817","nonce":3446774052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6819","nonce":3446774052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6823","nonce":3446774052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6821","nonce":3446774052}]},"public_addr":"172.21.15.136:6817/3446774052","cluster_addr":"172.21.15.136:6819/3446774052","heartbeat_back_addr":"172.21.15.136:6823/3446774052","heartbeat_front_addr":"172.21.15.136:6821/3446774052","state":["exists","up"]},{"osd":9,"uuid":"2989ca68-3164-4667-9bcf-230dcbb7b8f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6827","nonce":3423789152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6829","nonce":3423789152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6832","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6833","nonce":3423789152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6831","nonce":3423789152}]},"public_addr":"172.21.15.72:6827/3423789152","cluster_addr":"172.21.15.72:6829/3423789152","heartbeat_back_addr":"172.21.15.72:6833/3423789152","heartbeat_front_addr":"172.21.15.72:6831/3423789152","state":["exists","up"]},{"osd":10,"uuid":"9ec6d46a-0913-4208-97b5-167188ee8cb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6825","nonce":1120991574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6827","nonce":1120991574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6831","nonce":1120991574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6829","nonce":1120991574}]},"public_addr":"172.21.15.76:6825/1120991574","cluster_addr":"172.21.15.76:6827/1120991574","heartbeat_back_addr":"172.21.15.76:6831/1120991574","heartbeat_front_addr":"172.21.15.76:6829/1120991574","state":["exists","up"]},{"osd":11,"uuid":"bdd0774f-6e58-4918-b4cc-afd4f1118d3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6825","nonce":1135995193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6827","nonce":1135995193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6831","nonce":1135995193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6829","nonce":1135995193}]},"public_addr":"172.21.15.136:6825/1135995193","cluster_addr":"172.21.15.136:6827/1135995193","heartbeat_back_addr":"172.21.15.136:6831/1135995193","heartbeat_front_addr":"172.21.15.136:6829/1135995193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:36:52.128788+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:37:17.523654+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:37:41.634677+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:38:15.060797+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:38:41.234111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:39:06.783560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:39:40.884386+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:40:08.340465+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:40:34.344858+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:41:08.525309+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:41:36.782624+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:42:02.756888+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":6}]},{"pgid":"3.37","mappings":[{"from":5,"to":7}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:0/3625071330":"2024-04-27T23:34:58.266218+0000","172.21.15.72:6801/2903979134":"2024-04-27T23:34:58.266218+0000","172.21.15.72:6800/2903979134":"2024-04-27T23:34:58.266218+0000","172.21.15.72:0/1519142576":"2024-04-27T23:34:58.266218+0000","172.21.15.72:0/3669626460":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/945623297":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/3479150073":"2024-04-27T23:34:24.233319+0000","172.21.15.72:6800/4175896290":"2024-04-27T23:34:24.233319+0000","172.21.15.72:6801/4175896290":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/3783175012":"2024-04-27T23:34:58.266218+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T23:46:05.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.798+0000 7f149f600640 1 -- 172.21.15.72:0/881424407 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f147c0a3cc0 msgr2=0x7f147c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:05.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.798+0000 7f149f600640 1 --2- 172.21.15.72:0/881424407 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f147c0a3cc0 0x7f147c0a6180 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f1488005fd0 tx=0x7f1488009500 comp rx=0 tx=0).stop 2024-04-26T23:46:05.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.798+0000 7f149f600640 1 -- 172.21.15.72:0/881424407 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1498102950 msgr2=0x7f149819adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:05.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.798+0000 7f149f600640 1 --2- 172.21.15.72:0/881424407 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1498102950 0x7f149819adb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f148000e9c0 tx=0x7f148000ee80 comp rx=0 tx=0).stop 2024-04-26T23:46:05.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.798+0000 7f149f600640 1 -- 172.21.15.72:0/881424407 shutdown_connections 2024-04-26T23:46:05.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.798+0000 7f149f600640 1 --2- 172.21.15.72:0/881424407 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f147c0a3cc0 0x7f147c0a6180 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:05.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.798+0000 7f149f600640 1 --2- 172.21.15.72:0/881424407 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1498103b40 0x7f149819b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:05.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.798+0000 7f149f600640 1 --2- 172.21.15.72:0/881424407 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1498104500 0x7f14981a2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:05.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.798+0000 7f149f600640 1 --2- 172.21.15.72:0/881424407 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1498102950 0x7f149819adb0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:05.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.798+0000 7f149f600640 1 -- 172.21.15.72:0/881424407 >> 172.21.15.72:0/881424407 conn(0x7f14980fdee0 msgr2=0x7f1498074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:46:05.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.798+0000 7f149f600640 1 -- 172.21.15.72:0/881424407 shutdown_connections 2024-04-26T23:46:05.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:05.798+0000 7f149f600640 1 -- 172.21.15.72:0/881424407 wait complete. 2024-04-26T23:46:05.822 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-26T23:43:02.928649+0000', 'modified': '2024-04-26T23:43:30.190394+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24590, 'mds_1': 24571, 'mds_2': 24553}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24514': {'gid': 24514, 'name': 'c', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.136:6837/3639308555', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6836', 'nonce': 3639308555}, {'type': 'v1', 'addr': '172.21.15.136:6837', 'nonce': 3639308555}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24533': {'gid': 24533, 'name': 'h', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.76:6837/2506930945', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6836', 'nonce': 2506930945}, {'type': 'v1', 'addr': '172.21.15.76:6837', 'nonce': 2506930945}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24535': {'gid': 24535, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.136:6839/3938677630', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6838', 'nonce': 3938677630}, {'type': 'v1', 'addr': '172.21.15.136:6839', 'nonce': 3938677630}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24553': {'gid': 24553, 'name': 'g', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.72:6839/1403162102', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6838', 'nonce': 1403162102}, {'type': 'v1', 'addr': '172.21.15.72:6839', 'nonce': 1403162102}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24571': {'gid': 24571, 'name': 'a', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.72:6841/1356345077', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6840', 'nonce': 1356345077}, {'type': 'v1', 'addr': '172.21.15.72:6841', 'nonce': 1356345077}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24590': {'gid': 24590, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.76:6839/3405829058', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6838', 'nonce': 3405829058}, {'type': 'v1', 'addr': '172.21.15.76:6839', 'nonce': 3405829058}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24590, 'qdb_cluster': [24553, 24571, 24590]} 2024-04-26T23:46:05.822 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-26T23:46:05.823 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.g status 2024-04-26T23:46:05.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:05 smithi076 ceph-mon[32774]: pgmap v416: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:46:05.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:05 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/2754589002' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T23:46:05.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:05 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/3577055942' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:46:06.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.013+0000 7f2496200640 1 -- 172.21.15.72:0/3539628649 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2490105b80 msgr2=0x7f2490106000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:06.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.013+0000 7f2496200640 1 --2- 172.21.15.72:0/3539628649 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2490105b80 0x7f2490106000 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f2478009900 tx=0x7f247802a3f0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.013+0000 7f2496200640 1 -- 172.21.15.72:0/3539628649 shutdown_connections 2024-04-26T23:46:06.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.013+0000 7f2496200640 1 --2- 172.21.15.72:0/3539628649 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2490106540 0x7f249010cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.013+0000 7f2496200640 1 --2- 172.21.15.72:0/3539628649 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2490105b80 0x7f2490106000 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.013+0000 7f2496200640 1 --2- 172.21.15.72:0/3539628649 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2490104990 0x7f2490104db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.013+0000 7f2496200640 1 -- 172.21.15.72:0/3539628649 >> 172.21.15.72:0/3539628649 conn(0x7f24900fff00 msgr2=0x7f2490102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:46:06.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.013+0000 7f2496200640 1 -- 172.21.15.72:0/3539628649 shutdown_connections 2024-04-26T23:46:06.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.013+0000 7f2496200640 1 -- 172.21.15.72:0/3539628649 wait complete. 2024-04-26T23:46:06.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.014+0000 7f2496200640 1 Processor -- start 2024-04-26T23:46:06.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.014+0000 7f2496200640 1 -- start start 2024-04-26T23:46:06.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.014+0000 7f2496200640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2490104990 0x7f249019ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.014+0000 7f2496200640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2490105b80 0x7f249019b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.014+0000 7f2496200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2490106540 0x7f24901a2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.014+0000 7f2496200640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f24901104f0 con 0x7f2490105b80 2024-04-26T23:46:06.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.014+0000 7f2496200640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2490110370 con 0x7f2490106540 2024-04-26T23:46:06.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.014+0000 7f2496200640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f2490110670 con 0x7f2490104990 2024-04-26T23:46:06.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.015+0000 7f2495800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2490106540 0x7f24901a2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.015+0000 7f2494e00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2490104990 0x7f249019ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.015+0000 7f248fe00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2490105b80 0x7f249019b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.015+0000 7f2495800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2490106540 0x7f24901a2140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:34234/0 (socket says 172.21.15.72:34234) 2024-04-26T23:46:06.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.015+0000 7f2495800640 1 -- 172.21.15.72:0/3722818823 learned_addr learned my addr 172.21.15.72:0/3722818823 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:46:06.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.015+0000 7f2494e00640 1 -- 172.21.15.72:0/3722818823 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2490106540 msgr2=0x7f24901a2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:06.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.015+0000 7f2494e00640 1 --2- 172.21.15.72:0/3722818823 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2490106540 0x7f24901a2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.015+0000 7f2494e00640 1 -- 172.21.15.72:0/3722818823 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2490105b80 msgr2=0x7f249019b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:06.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.015+0000 7f2494e00640 1 --2- 172.21.15.72:0/3722818823 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2490105b80 0x7f249019b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.015+0000 7f2494e00640 1 -- 172.21.15.72:0/3722818823 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24901a27b0 con 0x7f2490104990 2024-04-26T23:46:06.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.015+0000 7f2494e00640 1 --2- 172.21.15.72:0/3722818823 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2490104990 0x7f249019ab90 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2480005fd0 tx=0x7f248000c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:46:06.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.016+0000 7f248d600640 1 -- 172.21.15.72:0/3722818823 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2480010630 con 0x7f2490104990 2024-04-26T23:46:06.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.016+0000 7f2496200640 1 -- 172.21.15.72:0/3722818823 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24901a2aa0 con 0x7f2490104990 2024-04-26T23:46:06.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.016+0000 7f248d600640 1 -- 172.21.15.72:0/3722818823 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2480010c80 con 0x7f2490104990 2024-04-26T23:46:06.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.016+0000 7f248d600640 1 -- 172.21.15.72:0/3722818823 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2480018cc0 con 0x7f2490104990 2024-04-26T23:46:06.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.016+0000 7f2496200640 1 -- 172.21.15.72:0/3722818823 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24901a3010 con 0x7f2490104990 2024-04-26T23:46:06.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.019+0000 7f249a625740 1 Processor -- start 2024-04-26T23:46:06.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.019+0000 7f249a625740 1 -- start start 2024-04-26T23:46:06.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.019+0000 7f249a625740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf58052c0 0x557bf58056e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.019+0000 7f249a625740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557bf5805cb0 0x557bf580a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.019+0000 7f248d600640 1 -- 172.21.15.72:0/3722818823 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f248001f020 con 0x7f2490104990 2024-04-26T23:46:06.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.019+0000 7f249a625740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557bf580ae20 0x557bf580d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.019+0000 7f249a625740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x557bf5769600 con 0x557bf580ae20 2024-04-26T23:46:06.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.019+0000 7f249a625740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x557bf5706100 con 0x557bf58052c0 2024-04-26T23:46:06.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.019+0000 7f249a625740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x557bf57655c0 con 0x557bf5805cb0 2024-04-26T23:46:06.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.019+0000 7f2495800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf58052c0 0x557bf58056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.019+0000 7f2495800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf58052c0 0x557bf58056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:34238/0 (socket says 172.21.15.72:34238) 2024-04-26T23:46:06.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.019+0000 7f2495800640 1 -- 172.21.15.72:0/173726524 learned_addr learned my addr 172.21.15.72:0/173726524 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:46:06.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.019+0000 7f2494e00640 1 --2- 172.21.15.72:0/173726524 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557bf5805cb0 0x557bf580a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.020+0000 7f248fe00640 1 --2- 172.21.15.72:0/173726524 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557bf580ae20 0x557bf580d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.020+0000 7f2495800640 1 -- 172.21.15.72:0/173726524 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557bf5805cb0 msgr2=0x557bf580a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:46:06.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.020+0000 7f2495800640 1 --2- 172.21.15.72:0/173726524 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557bf5805cb0 0x557bf580a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.020+0000 7f2495800640 1 -- 172.21.15.72:0/173726524 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557bf580ae20 msgr2=0x557bf580d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:46:06.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.020+0000 7f2495800640 1 --2- 172.21.15.72:0/173726524 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557bf580ae20 0x557bf580d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.020+0000 7f2495800640 1 -- 172.21.15.72:0/173726524 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557bf547b2d0 con 0x557bf58052c0 2024-04-26T23:46:06.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.020+0000 7f248fe00640 1 --2- 172.21.15.72:0/173726524 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557bf580ae20 0x557bf580d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:46:06.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.020+0000 7f2495800640 1 --2- 172.21.15.72:0/173726524 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf58052c0 0x557bf58056e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2488009d90 tx=0x7f2488011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33ee92da181f4a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:46:06.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.020+0000 7f2486000640 1 -- 172.21.15.72:0/173726524 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f248801e070 con 0x557bf58052c0 2024-04-26T23:46:06.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.021+0000 7f2486000640 1 -- 172.21.15.72:0/173726524 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2488019ae0 con 0x557bf58052c0 2024-04-26T23:46:06.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.021+0000 7f2486000640 1 -- 172.21.15.72:0/173726524 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f248800fb50 con 0x557bf58052c0 2024-04-26T23:46:06.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.021+0000 7f248d600640 1 --2- 172.21.15.72:0/3722818823 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f24740a3c40 0x7f24740a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.021+0000 7f248d600640 1 -- 172.21.15.72:0/3722818823 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f24800d0e00 con 0x7f2490104990 2024-04-26T23:46:06.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.021+0000 7f2494e00640 1 --2- 172.21.15.72:0/3722818823 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f24740a3c40 0x7f24740a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.021+0000 7f249a625740 1 -- 172.21.15.72:0/173726524 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf58052c0 msgr2=0x557bf58056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:06.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.021+0000 7f249a625740 1 --2- 172.21.15.72:0/173726524 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf58052c0 0x557bf58056e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2488009d90 tx=0x7f2488011810 comp rx=0 tx=0).stop 2024-04-26T23:46:06.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.021+0000 7f249a625740 1 -- 172.21.15.72:0/173726524 shutdown_connections 2024-04-26T23:46:06.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.021+0000 7f249a625740 1 --2- 172.21.15.72:0/173726524 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557bf580ae20 0x557bf580d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.021+0000 7f249a625740 1 --2- 172.21.15.72:0/173726524 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557bf5805cb0 0x557bf580a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.021+0000 7f249a625740 1 --2- 172.21.15.72:0/173726524 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf58052c0 0x557bf58056e0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.021+0000 7f249a625740 1 -- 172.21.15.72:0/173726524 >> 172.21.15.72:0/173726524 conn(0x557bf5799d10 msgr2=0x557bf5800d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:46:06.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.021+0000 7f249a625740 1 -- 172.21.15.72:0/173726524 shutdown_connections 2024-04-26T23:46:06.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.022+0000 7f249a625740 1 -- 172.21.15.72:0/173726524 wait complete. 2024-04-26T23:46:06.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.022+0000 7f2494e00640 1 --2- 172.21.15.72:0/3722818823 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f24740a3c40 0x7f24740a6100 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x557bf5806720 tx=0x7f24800d4950 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:46:06.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.022+0000 7f249a625740 1 -- start start 2024-04-26T23:46:06.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.022+0000 7f249a625740 1 Processor -- start 2024-04-26T23:46:06.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.023+0000 7f249a625740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf58052c0 0x557bf580f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.023+0000 7f249a625740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557bf5805cb0 0x557bf580fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.023+0000 7f249a625740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557bf580ae20 0x557bf5816c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.023+0000 7f249a625740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x557bf580ddd0 con 0x557bf5805cb0 2024-04-26T23:46:06.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.023+0000 7f249a625740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x557bf56f0f70 con 0x557bf58052c0 2024-04-26T23:46:06.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.023+0000 7f249a625740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x557bf580df50 con 0x557bf580ae20 2024-04-26T23:46:06.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.023+0000 7f249a625740 10 client.0 _ll_register_callbacks cb 0x557bf575f480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:46:06.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.023+0000 7f248fe00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557bf5805cb0 0x557bf580fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.023+0000 7f2495800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf58052c0 0x557bf580f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.023+0000 7f248fe00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557bf5805cb0 0x557bf580fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58126/0 (socket says 172.21.15.72:58126) 2024-04-26T23:46:06.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.023+0000 7f248fe00640 1 -- 172.21.15.72:0/3702115095 learned_addr learned my addr 172.21.15.72:0/3702115095 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:46:06.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.023+0000 7f2495800640 1 --2- 172.21.15.72:0/3702115095 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557bf580ae20 0x557bf5816c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.023+0000 7f2495800640 1 -- 172.21.15.72:0/3702115095 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557bf580ae20 msgr2=0x557bf5816c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:46:06.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.023+0000 7f2495800640 1 --2- 172.21.15.72:0/3702115095 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557bf580ae20 0x557bf5816c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.023+0000 7f2495800640 1 -- 172.21.15.72:0/3702115095 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557bf5805cb0 msgr2=0x557bf580fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:46:06.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.023+0000 7f2495800640 1 --2- 172.21.15.72:0/3702115095 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557bf5805cb0 0x557bf580fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.023+0000 7f2495800640 1 -- 172.21.15.72:0/3702115095 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557bf55456b0 con 0x557bf58052c0 2024-04-26T23:46:06.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.023+0000 7f248fe00640 1 --2- 172.21.15.72:0/3702115095 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557bf5805cb0 0x557bf580fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:46:06.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.024+0000 7f2495800640 1 --2- 172.21.15.72:0/3702115095 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf58052c0 0x557bf580f710 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f248800f470 tx=0x7f248800f3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3af80d1b7a699c3c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:46:06.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.024+0000 7f247ea00640 10 client.0 ms_handle_connect on v2:172.21.15.76:3300/0 2024-04-26T23:46:06.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.024+0000 7f247ea00640 1 -- 172.21.15.72:0/3702115095 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f248801e040 con 0x557bf58052c0 2024-04-26T23:46:06.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.024+0000 7f247ea00640 1 -- 172.21.15.72:0/3702115095 <== mon.2 v2:172.21.15.76:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f2488005050 con 0x557bf58052c0 2024-04-26T23:46:06.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.024+0000 7f249a625740 1 -- 172.21.15.72:0/3702115095 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2488019ae0 con 0x557bf58052c0 2024-04-26T23:46:06.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.024+0000 7f247ea00640 1 -- 172.21.15.72:0/3702115095 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f2488035050 con 0x557bf58052c0 2024-04-26T23:46:06.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.024+0000 7f249a625740 10 client.25358 fetch_fsmap learned FSMap version 23 2024-04-26T23:46:06.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.024+0000 7f249a625740 1 -- 172.21.15.72:0/3702115095 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x557bf5819c70 con 0x557bf58052c0 2024-04-26T23:46:06.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.025+0000 7f247ea00640 1 -- 172.21.15.72:0/3702115095 <== mon.2 v2:172.21.15.76:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2488019ae0 con 0x557bf58052c0 2024-04-26T23:46:06.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.025+0000 7f247ea00640 1 -- 172.21.15.72:0/3702115095 <== mon.2 v2:172.21.15.76:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7f2488030030 con 0x557bf58052c0 2024-04-26T23:46:06.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.025+0000 7f249a625740 10 client.25358 fetch_fsmap finished waiting for FSMap version 23 2024-04-26T23:46:06.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.025+0000 7f249a625740 10 client.25358 resolve_mds: resolved name 'g' to daemon mds.g 2024-04-26T23:46:06.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.025+0000 7f249a625740 20 client.25358 populate_metadata read hostname 'smithi072' 2024-04-26T23:46:06.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.025+0000 7f249a625740 1 --2- 172.21.15.72:0/3702115095 >> [v2:172.21.15.72:6838/1403162102,v1:172.21.15.72:6839/1403162102] conn(0x557bf581bff0 0x557bf581d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.025+0000 7f249a625740 4 client.25358 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:46:06.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.025+0000 7f249a625740 1 -- 172.21.15.72:0/3702115095 --> [v2:172.21.15.72:6838/1403162102,v1:172.21.15.72:6839/1403162102] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557bf5777210 con 0x557bf581bff0 2024-04-26T23:46:06.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.025+0000 7f248fe00640 1 --2- 172.21.15.72:0/3702115095 >> [v2:172.21.15.72:6838/1403162102,v1:172.21.15.72:6839/1403162102] conn(0x557bf581bff0 0x557bf581d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.026+0000 7f248fe00640 1 --2- 172.21.15.72:0/3702115095 >> [v2:172.21.15.72:6838/1403162102,v1:172.21.15.72:6839/1403162102] conn(0x557bf581bff0 0x557bf581d460 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7bcdb2068b90909d server_cookie=970b604d7b93e561 in_seq=0 out_seq=0 2024-04-26T23:46:06.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.026+0000 7f247ea00640 10 client.25358 ms_handle_connect on v2:172.21.15.72:6838/1403162102 2024-04-26T23:46:06.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.027+0000 7f247ea00640 1 -- 172.21.15.72:0/3702115095 <== mds.2 v2:172.21.15.72:6838/1403162102 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557bf5777210 con 0x557bf581bff0 2024-04-26T23:46:06.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.027+0000 7f247ea00640 10 client.25358 handle_command_reply: tid=0 2024-04-26T23:46:06.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.027+0000 7f249a625740 1 client.25358 shutdown 2024-04-26T23:46:06.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.027+0000 7f249a625740 1 -- 172.21.15.72:0/3702115095 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf58052c0 msgr2=0x557bf580f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:06.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.027+0000 7f249a625740 1 --2- 172.21.15.72:0/3702115095 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf58052c0 0x557bf580f710 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f248800f470 tx=0x7f248800f3c0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.027+0000 7f249a625740 1 -- 172.21.15.72:0/3702115095 shutdown_connections 2024-04-26T23:46:06.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.027+0000 7f249a625740 1 --2- 172.21.15.72:0/3702115095 >> [v2:172.21.15.72:6838/1403162102,v1:172.21.15.72:6839/1403162102] conn(0x557bf581bff0 0x557bf581d460 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.027+0000 7f249a625740 1 --2- 172.21.15.72:0/3702115095 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557bf580ae20 0x557bf5816c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.027+0000 7f249a625740 1 --2- 172.21.15.72:0/3702115095 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557bf5805cb0 0x557bf580fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.027+0000 7f247ea00640 0 client.25358 ms_handle_reset on v2:172.21.15.72:6838/1403162102 2024-04-26T23:46:06.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.027+0000 7f249a625740 1 --2- 172.21.15.72:0/3702115095 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf58052c0 0x557bf580f710 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.027+0000 7f249a625740 1 -- 172.21.15.72:0/3702115095 >> 172.21.15.72:0/3702115095 conn(0x557bf5799d10 msgr2=0x557bf57fff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:46:06.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.027+0000 7f249a625740 1 -- 172.21.15.72:0/3702115095 shutdown_connections 2024-04-26T23:46:06.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.027+0000 7f249a625740 1 -- 172.21.15.72:0/3702115095 wait complete. 2024-04-26T23:46:06.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.027+0000 7f249a625740 20 client.25358 trim_cache size 0 max 16384 2024-04-26T23:46:06.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:05 smithi136 ceph-mon[33050]: pgmap v416: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:46:06.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:05 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/2754589002' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T23:46:06.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:05 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/3577055942' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T23:46:06.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.047+0000 7f249a625740 1 Processor -- start 2024-04-26T23:46:06.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.047+0000 7f249a625740 1 -- start start 2024-04-26T23:46:06.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.047+0000 7f249a625740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557bf58052c0 0x557bf580d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.047+0000 7f249a625740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf5805cb0 0x557bf5816c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.048+0000 7f249a625740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557bf580ae20 0x557bf580f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.048+0000 7f249a625740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x557bf581b7a0 con 0x557bf58052c0 2024-04-26T23:46:06.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.048+0000 7f249a625740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x557bf581be70 con 0x557bf5805cb0 2024-04-26T23:46:06.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.048+0000 7f249a625740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x557bf581c540 con 0x557bf580ae20 2024-04-26T23:46:06.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.048+0000 7f248fe00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf5805cb0 0x557bf5816c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.048+0000 7f2495800640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557bf58052c0 0x557bf580d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.048+0000 7f248fe00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf5805cb0 0x557bf5816c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:34266/0 (socket says 172.21.15.72:34266) 2024-04-26T23:46:06.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.048+0000 7f248fe00640 1 -- 172.21.15.72:0/4047149404 learned_addr learned my addr 172.21.15.72:0/4047149404 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:46:06.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.048+0000 7f248fe00640 1 -- 172.21.15.72:0/4047149404 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557bf580ae20 msgr2=0x557bf580f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:46:06.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.048+0000 7f248fe00640 1 --2- 172.21.15.72:0/4047149404 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557bf580ae20 0x557bf580f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.048+0000 7f248fe00640 1 -- 172.21.15.72:0/4047149404 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557bf58052c0 msgr2=0x557bf580d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:46:06.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.048+0000 7f248fe00640 1 --2- 172.21.15.72:0/4047149404 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557bf58052c0 0x557bf580d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.048+0000 7f248fe00640 1 -- 172.21.15.72:0/4047149404 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557bf58079a0 con 0x557bf5805cb0 2024-04-26T23:46:06.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.049+0000 7f248fe00640 1 --2- 172.21.15.72:0/4047149404 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf5805cb0 0x557bf5816c50 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2478009230 tx=0x7f2478004500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5a06134f21cef67 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:46:06.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.049+0000 7f2485600640 1 -- 172.21.15.72:0/4047149404 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2478035040 con 0x557bf5805cb0 2024-04-26T23:46:06.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.049+0000 7f2485600640 1 -- 172.21.15.72:0/4047149404 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f24780035f0 con 0x557bf5805cb0 2024-04-26T23:46:06.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.049+0000 7f2485600640 1 -- 172.21.15.72:0/4047149404 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2478031800 con 0x557bf5805cb0 2024-04-26T23:46:06.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.049+0000 7f249a625740 1 -- 172.21.15.72:0/4047149404 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf5805cb0 msgr2=0x557bf5816c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:06.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.050+0000 7f249a625740 1 --2- 172.21.15.72:0/4047149404 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf5805cb0 0x557bf5816c50 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2478009230 tx=0x7f2478004500 comp rx=0 tx=0).stop 2024-04-26T23:46:06.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.050+0000 7f249a625740 1 -- 172.21.15.72:0/4047149404 shutdown_connections 2024-04-26T23:46:06.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.050+0000 7f249a625740 1 --2- 172.21.15.72:0/4047149404 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf5805cb0 0x557bf5816c50 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.050+0000 7f249a625740 1 --2- 172.21.15.72:0/4047149404 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557bf580ae20 0x557bf580f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.050+0000 7f249a625740 1 --2- 172.21.15.72:0/4047149404 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557bf58052c0 0x557bf580d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.050+0000 7f249a625740 1 -- 172.21.15.72:0/4047149404 >> 172.21.15.72:0/4047149404 conn(0x557bf5799d10 msgr2=0x557bf57fbdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:46:06.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.050+0000 7f249a625740 1 -- 172.21.15.72:0/4047149404 shutdown_connections 2024-04-26T23:46:06.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.050+0000 7f249a625740 1 -- 172.21.15.72:0/4047149404 wait complete. 2024-04-26T23:46:06.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.050+0000 7f249a625740 1 -- start start 2024-04-26T23:46:06.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.050+0000 7f249a625740 1 Processor -- start 2024-04-26T23:46:06.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.051+0000 7f249a625740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557bf58052c0 0x557bf5812de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.051+0000 7f249a625740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557bf5805cb0 0x557bf5813320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.051+0000 7f249a625740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf580ae20 0x557bf5815f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.051+0000 7f249a625740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x557bf581d2e0 con 0x557bf58052c0 2024-04-26T23:46:06.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.051+0000 7f249a625740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x557bf581cc10 con 0x557bf580ae20 2024-04-26T23:46:06.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.051+0000 7f249a625740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x557bf581d9b0 con 0x557bf5805cb0 2024-04-26T23:46:06.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.051+0000 7f249a625740 10 client.0 _ll_register_callbacks cb 0x7f248801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:46:06.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.051+0000 7f2495800640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557bf58052c0 0x557bf5812de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.051+0000 7f248fe00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557bf5805cb0 0x557bf5813320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.051+0000 7f2495800640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557bf58052c0 0x557bf5812de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58150/0 (socket says 172.21.15.72:58150) 2024-04-26T23:46:06.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.051+0000 7f2495800640 1 -- 172.21.15.72:0/2820919126 learned_addr learned my addr 172.21.15.72:0/2820919126 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:46:06.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.051+0000 7f2495800640 1 --2- 172.21.15.72:0/2820919126 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf580ae20 0x557bf5815f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.052+0000 7f248fe00640 1 -- 172.21.15.72:0/2820919126 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf580ae20 msgr2=0x557bf5815f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:46:06.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.052+0000 7f248fe00640 1 --2- 172.21.15.72:0/2820919126 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf580ae20 0x557bf5815f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.052+0000 7f248fe00640 1 -- 172.21.15.72:0/2820919126 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557bf58052c0 msgr2=0x557bf5812de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:46:06.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.052+0000 7f248fe00640 1 --2- 172.21.15.72:0/2820919126 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557bf58052c0 0x557bf5812de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.052+0000 7f248fe00640 1 -- 172.21.15.72:0/2820919126 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557bf58164f0 con 0x557bf5805cb0 2024-04-26T23:46:06.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.052+0000 7f248fe00640 1 --2- 172.21.15.72:0/2820919126 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557bf5805cb0 0x557bf5813320 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x557bf581d290 tx=0x7f24780042e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5fb07266738cede1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:46:06.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.052+0000 7f247ea00640 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2024-04-26T23:46:06.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.052+0000 7f247ea00640 1 -- 172.21.15.72:0/2820919126 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2478056070 con 0x557bf5805cb0 2024-04-26T23:46:06.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.052+0000 7f249a625740 1 -- 172.21.15.72:0/2820919126 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f24780035f0 con 0x557bf5805cb0 2024-04-26T23:46:06.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.052+0000 7f247ea00640 1 -- 172.21.15.72:0/2820919126 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f2478045400 con 0x557bf5805cb0 2024-04-26T23:46:06.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.053+0000 7f247ea00640 1 -- 172.21.15.72:0/2820919126 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f2478062050 con 0x557bf5805cb0 2024-04-26T23:46:06.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.053+0000 7f249a625740 10 client.25333 fetch_fsmap learned FSMap version 23 2024-04-26T23:46:06.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.053+0000 7f249a625740 1 -- 172.21.15.72:0/2820919126 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x557bf5882600 con 0x557bf5805cb0 2024-04-26T23:46:06.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.053+0000 7f247ea00640 1 -- 172.21.15.72:0/2820919126 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f24780035f0 con 0x557bf5805cb0 2024-04-26T23:46:06.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.053+0000 7f247ea00640 1 -- 172.21.15.72:0/2820919126 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7f2478031be0 con 0x557bf5805cb0 2024-04-26T23:46:06.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.053+0000 7f249a625740 10 client.25333 fetch_fsmap finished waiting for FSMap version 23 2024-04-26T23:46:06.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.053+0000 7f249a625740 10 client.25333 resolve_mds: resolved name 'g' to daemon mds.g 2024-04-26T23:46:06.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.053+0000 7f249a625740 20 client.25333 populate_metadata read hostname 'smithi072' 2024-04-26T23:46:06.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.054+0000 7f249a625740 1 --2- 172.21.15.72:0/2820919126 >> [v2:172.21.15.72:6838/1403162102,v1:172.21.15.72:6839/1403162102] conn(0x557bf5886d00 0x557bf5885a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.054+0000 7f249a625740 4 client.25333 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T23:46:06.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.054+0000 7f249a625740 1 -- 172.21.15.72:0/2820919126 --> [v2:172.21.15.72:6838/1403162102,v1:172.21.15.72:6839/1403162102] -- command(tid 0: {"prefix": "status"}) -- 0x557bf5771080 con 0x557bf5886d00 2024-04-26T23:46:06.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.054+0000 7f2495800640 1 --2- 172.21.15.72:0/2820919126 >> [v2:172.21.15.72:6838/1403162102,v1:172.21.15.72:6839/1403162102] conn(0x557bf5886d00 0x557bf5885a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.054+0000 7f2495800640 1 --2- 172.21.15.72:0/2820919126 >> [v2:172.21.15.72:6838/1403162102,v1:172.21.15.72:6839/1403162102] conn(0x557bf5886d00 0x557bf5885a10 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=31edf835b4cdd67a server_cookie=d963571aac4d6e48 in_seq=0 out_seq=0 2024-04-26T23:46:06.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.054+0000 7f247ea00640 10 client.25333 ms_handle_connect on v2:172.21.15.72:6838/1403162102 2024-04-26T23:46:06.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.055+0000 7f247ea00640 1 -- 172.21.15.72:0/2820919126 <== mds.2 v2:172.21.15.72:6838/1403162102 1 ==== command_reply(tid 0: 0 ) ==== 8+0+321 (crc 0 0 0) 0x557bf5771080 con 0x557bf5886d00 2024-04-26T23:46:06.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.055+0000 7f247ea00640 10 client.25333 handle_command_reply: tid=0 2024-04-26T23:46:06.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.055+0000 7f249a625740 1 client.25333 shutdown 2024-04-26T23:46:06.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.055+0000 7f249a625740 1 -- 172.21.15.72:0/2820919126 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557bf5805cb0 msgr2=0x557bf5813320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:06.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.055+0000 7f249a625740 1 --2- 172.21.15.72:0/2820919126 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557bf5805cb0 0x557bf5813320 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x557bf581d290 tx=0x7f24780042e0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.055+0000 7f249a625740 1 -- 172.21.15.72:0/2820919126 shutdown_connections 2024-04-26T23:46:06.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.055+0000 7f249a625740 1 --2- 172.21.15.72:0/2820919126 >> [v2:172.21.15.72:6838/1403162102,v1:172.21.15.72:6839/1403162102] conn(0x557bf5886d00 0x557bf5885a10 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.055+0000 7f249a625740 1 --2- 172.21.15.72:0/2820919126 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x557bf580ae20 0x557bf5815f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.055+0000 7f247ea00640 0 client.25333 ms_handle_reset on v2:172.21.15.72:6838/1403162102 2024-04-26T23:46:06.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.055+0000 7f249a625740 1 --2- 172.21.15.72:0/2820919126 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x557bf5805cb0 0x557bf5813320 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.055+0000 7f249a625740 1 --2- 172.21.15.72:0/2820919126 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557bf58052c0 0x557bf5812de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.055+0000 7f249a625740 1 -- 172.21.15.72:0/2820919126 >> 172.21.15.72:0/2820919126 conn(0x557bf5799d10 msgr2=0x557bf580d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:46:06.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.055+0000 7f249a625740 1 -- 172.21.15.72:0/2820919126 shutdown_connections 2024-04-26T23:46:06.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.055+0000 7f249a625740 1 -- 172.21.15.72:0/2820919126 wait complete. 2024-04-26T23:46:06.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.055+0000 7f249a625740 20 client.25333 trim_cache size 0 max 16384 2024-04-26T23:46:06.063 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-04-26T23:46:06.063 INFO:teuthology.orchestra.run.smithi072.stdout: "cluster_fsid": "3c8453ae-0425-11ef-bc93-c7b262605968", 2024-04-26T23:46:06.063 INFO:teuthology.orchestra.run.smithi072.stdout: "whoami": 2, 2024-04-26T23:46:06.063 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 24553, 2024-04-26T23:46:06.063 INFO:teuthology.orchestra.run.smithi072.stdout: "want_state": "up:active", 2024-04-26T23:46:06.063 INFO:teuthology.orchestra.run.smithi072.stdout: "state": "up:active", 2024-04-26T23:46:06.063 INFO:teuthology.orchestra.run.smithi072.stdout: "fs_name": "cephfs", 2024-04-26T23:46:06.064 INFO:teuthology.orchestra.run.smithi072.stdout: "rank_uptime": 156.87264273, 2024-04-26T23:46:06.064 INFO:teuthology.orchestra.run.smithi072.stdout: "mdsmap_epoch": 23, 2024-04-26T23:46:06.064 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_epoch": 93, 2024-04-26T23:46:06.064 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_epoch_barrier": 89, 2024-04-26T23:46:06.064 INFO:teuthology.orchestra.run.smithi072.stdout: "uptime": 196.200469336 2024-04-26T23:46:06.064 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-04-26T23:46:06.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.056+0000 7f2496200640 1 -- 172.21.15.72:0/3722818823 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f24740a3c40 msgr2=0x7f24740a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:06.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.056+0000 7f2496200640 1 --2- 172.21.15.72:0/3722818823 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f24740a3c40 0x7f24740a6100 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x557bf5806720 tx=0x7f24800d4950 comp rx=0 tx=0).stop 2024-04-26T23:46:06.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.056+0000 7f2496200640 1 -- 172.21.15.72:0/3722818823 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2490104990 msgr2=0x7f249019ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:06.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.056+0000 7f2496200640 1 --2- 172.21.15.72:0/3722818823 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2490104990 0x7f249019ab90 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2480005fd0 tx=0x7f248000c6a0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.056+0000 7f2496200640 1 -- 172.21.15.72:0/3722818823 shutdown_connections 2024-04-26T23:46:06.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.056+0000 7f2496200640 1 --2- 172.21.15.72:0/3722818823 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f24740a3c40 0x7f24740a6100 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.056+0000 7f2496200640 1 --2- 172.21.15.72:0/3722818823 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2490106540 0x7f24901a2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.056+0000 7f2496200640 1 --2- 172.21.15.72:0/3722818823 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2490105b80 0x7f249019b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.056+0000 7f2496200640 1 --2- 172.21.15.72:0/3722818823 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2490104990 0x7f249019ab90 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.056+0000 7f2496200640 1 -- 172.21.15.72:0/3722818823 >> 172.21.15.72:0/3722818823 conn(0x7f24900fff00 msgr2=0x7f2490108e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:46:06.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.056+0000 7f2496200640 1 -- 172.21.15.72:0/3722818823 shutdown_connections 2024-04-26T23:46:06.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.056+0000 7f2496200640 1 -- 172.21.15.72:0/3722818823 wait complete. 2024-04-26T23:46:06.079 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-26T23:46:06.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.267+0000 7fedeb000640 1 -- 172.21.15.72:0/647887766 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fede4103b70 msgr2=0x7fede4103ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:06.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.267+0000 7fedeb000640 1 --2- 172.21.15.72:0/647887766 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fede4103b70 0x7fede4103ff0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fedcc009be0 tx=0x7fedcc02a6c0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.267+0000 7fedeb000640 1 -- 172.21.15.72:0/647887766 shutdown_connections 2024-04-26T23:46:06.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.267+0000 7fedeb000640 1 --2- 172.21.15.72:0/647887766 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fede4104530 0x7fede410adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.267+0000 7fedeb000640 1 --2- 172.21.15.72:0/647887766 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fede4103b70 0x7fede4103ff0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.267+0000 7fedeb000640 1 --2- 172.21.15.72:0/647887766 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fede4102980 0x7fede4102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.267+0000 7fedeb000640 1 -- 172.21.15.72:0/647887766 >> 172.21.15.72:0/647887766 conn(0x7fede40fdf10 msgr2=0x7fede4100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:46:06.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.267+0000 7fedeb000640 1 -- 172.21.15.72:0/647887766 shutdown_connections 2024-04-26T23:46:06.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.267+0000 7fedeb000640 1 -- 172.21.15.72:0/647887766 wait complete. 2024-04-26T23:46:06.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.268+0000 7fedeb000640 1 Processor -- start 2024-04-26T23:46:06.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.268+0000 7fedeb000640 1 -- start start 2024-04-26T23:46:06.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.268+0000 7fedeb000640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fede4102980 0x7fede419ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.268+0000 7fedeb000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fede4103b70 0x7fede419b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.268+0000 7fedeb000640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fede4104530 0x7fede41a2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.268+0000 7fedeb000640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fede410e4b0 con 0x7fede4102980 2024-04-26T23:46:06.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.268+0000 7fedeb000640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fede410e330 con 0x7fede4103b70 2024-04-26T23:46:06.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.268+0000 7fedeb000640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fede410e630 con 0x7fede4104530 2024-04-26T23:46:06.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.269+0000 7fede3e00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fede4102980 0x7fede419ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.269+0000 7fede3e00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fede4102980 0x7fede419ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58162/0 (socket says 172.21.15.72:58162) 2024-04-26T23:46:06.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.269+0000 7fede3e00640 1 -- 172.21.15.72:0/1328427111 learned_addr learned my addr 172.21.15.72:0/1328427111 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:46:06.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.269+0000 7fede9000640 1 --2- 172.21.15.72:0/1328427111 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fede4104530 0x7fede41a2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.269+0000 7fede3400640 1 --2- 172.21.15.72:0/1328427111 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fede4103b70 0x7fede419b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.269+0000 7fede3e00640 1 -- 172.21.15.72:0/1328427111 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fede4104530 msgr2=0x7fede41a2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:06.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.269+0000 7fede3e00640 1 --2- 172.21.15.72:0/1328427111 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fede4104530 0x7fede41a2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.269+0000 7fede3e00640 1 -- 172.21.15.72:0/1328427111 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fede4103b70 msgr2=0x7fede419b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:06.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.269+0000 7fede3e00640 1 --2- 172.21.15.72:0/1328427111 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fede4103b70 0x7fede419b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.269+0000 7fede3e00640 1 -- 172.21.15.72:0/1328427111 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fede41a2990 con 0x7fede4102980 2024-04-26T23:46:06.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.269+0000 7fede3400640 1 --2- 172.21.15.72:0/1328427111 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fede4103b70 0x7fede419b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:46:06.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.270+0000 7fede3e00640 1 --2- 172.21.15.72:0/1328427111 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fede4102980 0x7fede419ad70 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fedd400e9f0 tx=0x7fedd400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:46:06.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.270+0000 7fede0c00640 1 -- 172.21.15.72:0/1328427111 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd400ccc0 con 0x7fede4102980 2024-04-26T23:46:06.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.270+0000 7fede0c00640 1 -- 172.21.15.72:0/1328427111 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fedd40035f0 con 0x7fede4102980 2024-04-26T23:46:06.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.270+0000 7fedeb000640 1 -- 172.21.15.72:0/1328427111 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fede41a2c80 con 0x7fede4102980 2024-04-26T23:46:06.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.270+0000 7fede0c00640 1 -- 172.21.15.72:0/1328427111 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd400ce60 con 0x7fede4102980 2024-04-26T23:46:06.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.270+0000 7fedeb000640 1 -- 172.21.15.72:0/1328427111 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fede41a31a0 con 0x7fede4102980 2024-04-26T23:46:06.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.274+0000 7fededfb9740 1 Processor -- start 2024-04-26T23:46:06.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.274+0000 7fededfb9740 1 -- start start 2024-04-26T23:46:06.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.274+0000 7fededfb9740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55693de1f0f0 0x55693de1f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.274+0000 7fededfb9740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55693de1fae0 0x55693de245e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.275+0000 7fededfb9740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55693de24c50 0x55693de270e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.275+0000 7fede0c00640 1 -- 172.21.15.72:0/1328427111 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fedd4004000 con 0x7fede4102980 2024-04-26T23:46:06.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.275+0000 7fededfb9740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55693dd83600 con 0x55693de24c50 2024-04-26T23:46:06.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.275+0000 7fededfb9740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55693dd20100 con 0x55693de1f0f0 2024-04-26T23:46:06.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.275+0000 7fededfb9740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55693dd7f5c0 con 0x55693de1fae0 2024-04-26T23:46:06.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.275+0000 7fede9000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55693de1f0f0 0x55693de1f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.275+0000 7fede9000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55693de1f0f0 0x55693de1f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:34308/0 (socket says 172.21.15.72:34308) 2024-04-26T23:46:06.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.275+0000 7fede9000640 1 -- 172.21.15.72:0/3400368721 learned_addr learned my addr 172.21.15.72:0/3400368721 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:46:06.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.275+0000 7fede3400640 1 --2- 172.21.15.72:0/3400368721 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55693de24c50 0x55693de270e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.275+0000 7fede3e00640 1 --2- 172.21.15.72:0/3400368721 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55693de1fae0 0x55693de245e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.275+0000 7fede9000640 1 -- 172.21.15.72:0/3400368721 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55693de1fae0 msgr2=0x55693de245e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:46:06.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.275+0000 7fede9000640 1 --2- 172.21.15.72:0/3400368721 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55693de1fae0 0x55693de245e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.275+0000 7fede9000640 1 -- 172.21.15.72:0/3400368721 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55693de24c50 msgr2=0x55693de270e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:46:06.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.275+0000 7fede9000640 1 --2- 172.21.15.72:0/3400368721 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55693de24c50 0x55693de270e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.275+0000 7fede9000640 1 -- 172.21.15.72:0/3400368721 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55693da952d0 con 0x55693de1f0f0 2024-04-26T23:46:06.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.276+0000 7fede9000640 1 --2- 172.21.15.72:0/3400368721 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55693de1f0f0 0x55693de1f510 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7feddc009ca0 tx=0x7feddc011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15ebf9e61717469f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:46:06.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.276+0000 7fedd9600640 1 -- 172.21.15.72:0/3400368721 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feddc01e070 con 0x55693de1f0f0 2024-04-26T23:46:06.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.276+0000 7fedd9600640 1 -- 172.21.15.72:0/3400368721 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feddc009e50 con 0x55693de1f0f0 2024-04-26T23:46:06.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.276+0000 7fedd9600640 1 -- 172.21.15.72:0/3400368721 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feddc00ea50 con 0x55693de1f0f0 2024-04-26T23:46:06.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.276+0000 7fede0c00640 1 --2- 172.21.15.72:0/1328427111 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fedc40a3cc0 0x7fedc40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.276+0000 7fededfb9740 1 -- 172.21.15.72:0/3400368721 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55693de1f0f0 msgr2=0x55693de1f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:06.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.276+0000 7fededfb9740 1 --2- 172.21.15.72:0/3400368721 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55693de1f0f0 0x55693de1f510 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7feddc009ca0 tx=0x7feddc011810 comp rx=0 tx=0).stop 2024-04-26T23:46:06.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.277+0000 7fede0c00640 1 -- 172.21.15.72:0/1328427111 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7fedd40d0bc0 con 0x7fede4102980 2024-04-26T23:46:06.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.277+0000 7fede3e00640 1 --2- 172.21.15.72:0/1328427111 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fedc40a3cc0 0x7fedc40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.277+0000 7fededfb9740 1 -- 172.21.15.72:0/3400368721 shutdown_connections 2024-04-26T23:46:06.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.277+0000 7fededfb9740 1 --2- 172.21.15.72:0/3400368721 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55693de24c50 0x55693de270e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.277+0000 7fededfb9740 1 --2- 172.21.15.72:0/3400368721 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55693de1fae0 0x55693de245e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.277+0000 7fededfb9740 1 --2- 172.21.15.72:0/3400368721 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55693de1f0f0 0x55693de1f510 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.277+0000 7fededfb9740 1 -- 172.21.15.72:0/3400368721 >> 172.21.15.72:0/3400368721 conn(0x55693ddb3d10 msgr2=0x55693de1ab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:46:06.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.277+0000 7fededfb9740 1 -- 172.21.15.72:0/3400368721 shutdown_connections 2024-04-26T23:46:06.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.277+0000 7fededfb9740 1 -- 172.21.15.72:0/3400368721 wait complete. 2024-04-26T23:46:06.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.277+0000 7fede3e00640 1 --2- 172.21.15.72:0/1328427111 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fedc40a3cc0 0x7fedc40a6180 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x55693de20550 tx=0x7fedd40bd040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:46:06.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.277+0000 7fededfb9740 1 -- start start 2024-04-26T23:46:06.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.278+0000 7fededfb9740 1 Processor -- start 2024-04-26T23:46:06.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.278+0000 7fededfb9740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55693de1f0f0 0x55693de29520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.278+0000 7fededfb9740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55693de1fae0 0x55693de29a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.278+0000 7fededfb9740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55693de24c50 0x55693de30aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.278+0000 7fededfb9740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55693de27c00 con 0x55693de24c50 2024-04-26T23:46:06.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.278+0000 7fededfb9740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55693dd0af70 con 0x55693de1f0f0 2024-04-26T23:46:06.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.278+0000 7fededfb9740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55693de27d80 con 0x55693de1fae0 2024-04-26T23:46:06.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.278+0000 7fededfb9740 10 client.0 _ll_register_callbacks cb 0x55693dd79480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:46:06.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.278+0000 7fede9000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55693de1f0f0 0x55693de29520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.278+0000 7fede9000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55693de1f0f0 0x55693de29520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:34310/0 (socket says 172.21.15.72:34310) 2024-04-26T23:46:06.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.278+0000 7fede3400640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55693de1fae0 0x55693de29a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.278+0000 7fede9000640 1 -- 172.21.15.72:0/3445394267 learned_addr learned my addr 172.21.15.72:0/3445394267 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:46:06.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.278+0000 7fede9000640 1 --2- 172.21.15.72:0/3445394267 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55693de24c50 0x55693de30aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.279+0000 7fede3400640 1 -- 172.21.15.72:0/3445394267 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55693de1f0f0 msgr2=0x55693de29520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:46:06.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.279+0000 7fede3400640 1 --2- 172.21.15.72:0/3445394267 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55693de1f0f0 0x55693de29520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.279+0000 7fede3400640 1 -- 172.21.15.72:0/3445394267 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55693de24c50 msgr2=0x55693de30aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:46:06.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.279+0000 7fede3400640 1 --2- 172.21.15.72:0/3445394267 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55693de24c50 0x55693de30aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.279+0000 7fede3400640 1 -- 172.21.15.72:0/3445394267 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55693db5f6b0 con 0x55693de1fae0 2024-04-26T23:46:06.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.279+0000 7fede3400640 1 --2- 172.21.15.72:0/3445394267 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55693de1fae0 0x55693de29a60 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fedcc02abe0 tx=0x7fedcc030640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94e0915073489706 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:46:06.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.279+0000 7fedd2000640 10 client.0 ms_handle_connect on v2:172.21.15.136:3300/0 2024-04-26T23:46:06.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.279+0000 7fedd2000640 1 -- 172.21.15.72:0/3445394267 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedcc042070 con 0x55693de1fae0 2024-04-26T23:46:06.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.280+0000 7fedd2000640 1 -- 172.21.15.72:0/3445394267 <== mon.1 v2:172.21.15.136:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7fedcc030d30 con 0x55693de1fae0 2024-04-26T23:46:06.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.280+0000 7fededfb9740 1 -- 172.21.15.72:0/3445394267 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feddc009e50 con 0x55693de1fae0 2024-04-26T23:46:06.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.280+0000 7fedd2000640 1 -- 172.21.15.72:0/3445394267 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fedcc04f050 con 0x55693de1fae0 2024-04-26T23:46:06.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.280+0000 7fededfb9740 10 client.25345 fetch_fsmap learned FSMap version 23 2024-04-26T23:46:06.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.280+0000 7fededfb9740 1 -- 172.21.15.72:0/3445394267 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55693de33af0 con 0x55693de1fae0 2024-04-26T23:46:06.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.280+0000 7fedd2000640 1 -- 172.21.15.72:0/3445394267 <== mon.1 v2:172.21.15.136:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feddc009e50 con 0x55693de1fae0 2024-04-26T23:46:06.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.281+0000 7fedd2000640 1 -- 172.21.15.72:0/3445394267 <== mon.1 v2:172.21.15.136:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7fedcc004570 con 0x55693de1fae0 2024-04-26T23:46:06.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.281+0000 7fededfb9740 10 client.25345 fetch_fsmap finished waiting for FSMap version 23 2024-04-26T23:46:06.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.281+0000 7fededfb9740 10 client.25345 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-26T23:46:06.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.281+0000 7fededfb9740 20 client.25345 populate_metadata read hostname 'smithi072' 2024-04-26T23:46:06.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.281+0000 7fededfb9740 1 --2- 172.21.15.72:0/3445394267 >> [v2:172.21.15.72:6840/1356345077,v1:172.21.15.72:6841/1356345077] conn(0x55693de36140 0x55693de375b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.281+0000 7fededfb9740 4 client.25345 mds_command: new command op to 24571 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:46:06.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.281+0000 7fededfb9740 1 -- 172.21.15.72:0/3445394267 --> [v2:172.21.15.72:6840/1356345077,v1:172.21.15.72:6841/1356345077] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55693dd91210 con 0x55693de36140 2024-04-26T23:46:06.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.281+0000 7fede9000640 1 --2- 172.21.15.72:0/3445394267 >> [v2:172.21.15.72:6840/1356345077,v1:172.21.15.72:6841/1356345077] conn(0x55693de36140 0x55693de375b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.282+0000 7fede9000640 1 --2- 172.21.15.72:0/3445394267 >> [v2:172.21.15.72:6840/1356345077,v1:172.21.15.72:6841/1356345077] conn(0x55693de36140 0x55693de375b0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=43536acca5c4d045 server_cookie=bc63ba54fd302a00 in_seq=0 out_seq=0 2024-04-26T23:46:06.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.282+0000 7fedd2000640 10 client.25345 ms_handle_connect on v2:172.21.15.72:6840/1356345077 2024-04-26T23:46:06.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.282+0000 7fedd2000640 1 -- 172.21.15.72:0/3445394267 <== mds.1 v2:172.21.15.72:6840/1356345077 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55693dd91210 con 0x55693de36140 2024-04-26T23:46:06.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.282+0000 7fedd2000640 10 client.25345 handle_command_reply: tid=0 2024-04-26T23:46:06.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.282+0000 7fededfb9740 1 client.25345 shutdown 2024-04-26T23:46:06.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.283+0000 7fededfb9740 1 -- 172.21.15.72:0/3445394267 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55693de1fae0 msgr2=0x55693de29a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:06.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.283+0000 7fededfb9740 1 --2- 172.21.15.72:0/3445394267 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55693de1fae0 0x55693de29a60 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fedcc02abe0 tx=0x7fedcc030640 comp rx=0 tx=0).stop 2024-04-26T23:46:06.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.283+0000 7fededfb9740 1 -- 172.21.15.72:0/3445394267 shutdown_connections 2024-04-26T23:46:06.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.283+0000 7fededfb9740 1 --2- 172.21.15.72:0/3445394267 >> [v2:172.21.15.72:6840/1356345077,v1:172.21.15.72:6841/1356345077] conn(0x55693de36140 0x55693de375b0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.283+0000 7fededfb9740 1 --2- 172.21.15.72:0/3445394267 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55693de24c50 0x55693de30aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.283+0000 7fededfb9740 1 --2- 172.21.15.72:0/3445394267 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55693de1fae0 0x55693de29a60 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.283+0000 7fededfb9740 1 --2- 172.21.15.72:0/3445394267 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55693de1f0f0 0x55693de29520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.283+0000 7fedd2000640 0 client.25345 ms_handle_reset on v2:172.21.15.72:6840/1356345077 2024-04-26T23:46:06.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.283+0000 7fededfb9740 1 -- 172.21.15.72:0/3445394267 >> 172.21.15.72:0/3445394267 conn(0x55693ddb3d10 msgr2=0x55693de19eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:46:06.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.283+0000 7fededfb9740 1 -- 172.21.15.72:0/3445394267 shutdown_connections 2024-04-26T23:46:06.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.283+0000 7fededfb9740 1 -- 172.21.15.72:0/3445394267 wait complete. 2024-04-26T23:46:06.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.283+0000 7fededfb9740 20 client.25345 trim_cache size 0 max 16384 2024-04-26T23:46:06.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.303+0000 7fededfb9740 1 Processor -- start 2024-04-26T23:46:06.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.303+0000 7fededfb9740 1 -- start start 2024-04-26T23:46:06.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.303+0000 7fededfb9740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55693de1f0f0 0x55693de1a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.303+0000 7fededfb9740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55693de1fae0 0x55693de30a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.303+0000 7fededfb9740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55693de24c50 0x55693de19d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.303+0000 7fededfb9740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55693de358f0 con 0x55693de24c50 2024-04-26T23:46:06.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.303+0000 7fededfb9740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55693de35fc0 con 0x55693de1fae0 2024-04-26T23:46:06.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.303+0000 7fededfb9740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55693de36690 con 0x55693de1f0f0 2024-04-26T23:46:06.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.304+0000 7fede9000640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55693de24c50 0x55693de19d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.304+0000 7fede9000640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55693de24c50 0x55693de19d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58200/0 (socket says 172.21.15.72:58200) 2024-04-26T23:46:06.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.304+0000 7fede9000640 1 -- 172.21.15.72:0/3590213436 learned_addr learned my addr 172.21.15.72:0/3590213436 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:46:06.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.304+0000 7fede3400640 1 --2- 172.21.15.72:0/3590213436 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55693de1fae0 0x55693de30a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.304+0000 7fede9000640 1 --2- 172.21.15.72:0/3590213436 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55693de1f0f0 0x55693de1a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.304+0000 7fede9000640 1 -- 172.21.15.72:0/3590213436 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55693de1f0f0 msgr2=0x55693de1a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:46:06.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.304+0000 7fede9000640 1 --2- 172.21.15.72:0/3590213436 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55693de1f0f0 0x55693de1a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.304+0000 7fede9000640 1 -- 172.21.15.72:0/3590213436 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55693de1fae0 msgr2=0x55693de30a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:46:06.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.304+0000 7fede9000640 1 --2- 172.21.15.72:0/3590213436 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55693de1fae0 0x55693de30a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.304+0000 7fede9000640 1 -- 172.21.15.72:0/3590213436 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55693de202d0 con 0x55693de24c50 2024-04-26T23:46:06.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.304+0000 7fede3400640 1 --2- 172.21.15.72:0/3590213436 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55693de1fae0 0x55693de30a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:46:06.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.304+0000 7fede9000640 1 --2- 172.21.15.72:0/3590213436 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55693de24c50 0x55693de19d30 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7feddc014040 tx=0x7feddc01b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e30528474330f03 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:46:06.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.304+0000 7fedd8c00640 1 -- 172.21.15.72:0/3590213436 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feddc01e040 con 0x55693de24c50 2024-04-26T23:46:06.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.305+0000 7fedd8c00640 1 -- 172.21.15.72:0/3590213436 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feddc005560 con 0x55693de24c50 2024-04-26T23:46:06.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.305+0000 7fedd8c00640 1 -- 172.21.15.72:0/3590213436 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feddc003a50 con 0x55693de24c50 2024-04-26T23:46:06.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.305+0000 7fededfb9740 1 -- 172.21.15.72:0/3590213436 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55693de24c50 msgr2=0x55693de19d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:06.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.305+0000 7fededfb9740 1 --2- 172.21.15.72:0/3590213436 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55693de24c50 0x55693de19d30 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7feddc014040 tx=0x7feddc01b040 comp rx=0 tx=0).stop 2024-04-26T23:46:06.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.305+0000 7fededfb9740 1 -- 172.21.15.72:0/3590213436 shutdown_connections 2024-04-26T23:46:06.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.305+0000 7fededfb9740 1 --2- 172.21.15.72:0/3590213436 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55693de1fae0 0x55693de30a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.305+0000 7fededfb9740 1 --2- 172.21.15.72:0/3590213436 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55693de24c50 0x55693de19d30 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.305+0000 7fededfb9740 1 --2- 172.21.15.72:0/3590213436 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55693de1f0f0 0x55693de1a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.305+0000 7fededfb9740 1 -- 172.21.15.72:0/3590213436 >> 172.21.15.72:0/3590213436 conn(0x55693ddb3d10 msgr2=0x55693de19620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:46:06.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.305+0000 7fededfb9740 1 -- 172.21.15.72:0/3590213436 shutdown_connections 2024-04-26T23:46:06.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.305+0000 7fededfb9740 1 -- 172.21.15.72:0/3590213436 wait complete. 2024-04-26T23:46:06.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.306+0000 7fededfb9740 1 -- start start 2024-04-26T23:46:06.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.306+0000 7fededfb9740 1 Processor -- start 2024-04-26T23:46:06.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.306+0000 7fededfb9740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55693de1f0f0 0x55693de30310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.306+0000 7fededfb9740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55693de1fae0 0x55693de30850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.306+0000 7fededfb9740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55693de24c50 0x55693de2cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.306+0000 7fededfb9740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55693de37430 con 0x55693de1fae0 2024-04-26T23:46:06.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.306+0000 7fededfb9740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55693de36d60 con 0x55693de1f0f0 2024-04-26T23:46:06.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.306+0000 7fededfb9740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55693de37b00 con 0x55693de24c50 2024-04-26T23:46:06.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.306+0000 7fededfb9740 10 client.0 _ll_register_callbacks cb 0x7fedd4005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:46:06.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.307+0000 7fede9000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55693de1f0f0 0x55693de30310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.307+0000 7fede9000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55693de1f0f0 0x55693de30310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:34332/0 (socket says 172.21.15.72:34332) 2024-04-26T23:46:06.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.307+0000 7fede9000640 1 -- 172.21.15.72:0/4226316866 learned_addr learned my addr 172.21.15.72:0/4226316866 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:46:06.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.307+0000 7fede3400640 1 --2- 172.21.15.72:0/4226316866 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55693de1fae0 0x55693de30850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.307+0000 7fede9000640 1 --2- 172.21.15.72:0/4226316866 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55693de24c50 0x55693de2cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.307+0000 7fede3400640 1 -- 172.21.15.72:0/4226316866 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55693de24c50 msgr2=0x55693de2cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:46:06.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.307+0000 7fede3400640 1 --2- 172.21.15.72:0/4226316866 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55693de24c50 0x55693de2cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.307+0000 7fede3400640 1 -- 172.21.15.72:0/4226316866 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55693de1f0f0 msgr2=0x55693de30310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:46:06.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.307+0000 7fede3400640 1 --2- 172.21.15.72:0/4226316866 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55693de1f0f0 0x55693de30310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.307+0000 7fede3400640 1 -- 172.21.15.72:0/4226316866 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55693de2d4c0 con 0x55693de1fae0 2024-04-26T23:46:06.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.307+0000 7fede9000640 1 --2- 172.21.15.72:0/4226316866 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55693de1f0f0 0x55693de30310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:46:06.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.308+0000 7fede3400640 1 --2- 172.21.15.72:0/4226316866 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55693de1fae0 0x55693de30850 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fedcc02a700 tx=0x7fedcc030de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4419b004d4b27cc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:46:06.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.308+0000 7fedd2000640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-04-26T23:46:06.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.308+0000 7fedd2000640 1 -- 172.21.15.72:0/4226316866 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedcc042030 con 0x55693de1fae0 2024-04-26T23:46:06.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.308+0000 7fededfb9740 1 -- 172.21.15.72:0/4226316866 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feddc005560 con 0x55693de1fae0 2024-04-26T23:46:06.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.308+0000 7fedd2000640 1 -- 172.21.15.72:0/4226316866 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7fedcc003b80 con 0x55693de1fae0 2024-04-26T23:46:06.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.308+0000 7fedd2000640 1 -- 172.21.15.72:0/4226316866 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fedcc063050 con 0x55693de1fae0 2024-04-26T23:46:06.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.308+0000 7fededfb9740 10 client.15279 fetch_fsmap learned FSMap version 23 2024-04-26T23:46:06.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.308+0000 7fededfb9740 1 -- 172.21.15.72:0/4226316866 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55693dea55d0 con 0x55693de1fae0 2024-04-26T23:46:06.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.309+0000 7fedd2000640 1 -- 172.21.15.72:0/4226316866 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feddc005560 con 0x55693de1fae0 2024-04-26T23:46:06.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.309+0000 7fedd2000640 1 -- 172.21.15.72:0/4226316866 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7fedcc004000 con 0x55693de1fae0 2024-04-26T23:46:06.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.309+0000 7fededfb9740 10 client.15279 fetch_fsmap finished waiting for FSMap version 23 2024-04-26T23:46:06.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.309+0000 7fededfb9740 10 client.15279 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-26T23:46:06.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.309+0000 7fededfb9740 20 client.15279 populate_metadata read hostname 'smithi072' 2024-04-26T23:46:06.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.309+0000 7fededfb9740 1 --2- 172.21.15.72:0/4226316866 >> [v2:172.21.15.72:6840/1356345077,v1:172.21.15.72:6841/1356345077] conn(0x55693deaa190 0x55693dea8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.309+0000 7fededfb9740 4 client.15279 mds_command: new command op to 24571 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T23:46:06.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.309+0000 7fededfb9740 1 -- 172.21.15.72:0/4226316866 --> [v2:172.21.15.72:6840/1356345077,v1:172.21.15.72:6841/1356345077] -- command(tid 0: {"prefix": "status"}) -- 0x55693dd8b080 con 0x55693deaa190 2024-04-26T23:46:06.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.309+0000 7fede9000640 1 --2- 172.21.15.72:0/4226316866 >> [v2:172.21.15.72:6840/1356345077,v1:172.21.15.72:6841/1356345077] conn(0x55693deaa190 0x55693dea8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.310+0000 7fede9000640 1 --2- 172.21.15.72:0/4226316866 >> [v2:172.21.15.72:6840/1356345077,v1:172.21.15.72:6841/1356345077] conn(0x55693deaa190 0x55693dea8ea0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ca7334a3794041ea server_cookie=95b4e229d607433f in_seq=0 out_seq=0 2024-04-26T23:46:06.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.310+0000 7fedd2000640 10 client.15279 ms_handle_connect on v2:172.21.15.72:6840/1356345077 2024-04-26T23:46:06.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.310+0000 7fedd2000640 1 -- 172.21.15.72:0/4226316866 <== mds.1 v2:172.21.15.72:6840/1356345077 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55693dd8b080 con 0x55693deaa190 2024-04-26T23:46:06.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.310+0000 7fedd2000640 10 client.15279 handle_command_reply: tid=0 2024-04-26T23:46:06.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.310+0000 7fededfb9740 1 client.15279 shutdown 2024-04-26T23:46:06.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.310+0000 7fededfb9740 1 -- 172.21.15.72:0/4226316866 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55693de1fae0 msgr2=0x55693de30850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:06.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.310+0000 7fededfb9740 1 --2- 172.21.15.72:0/4226316866 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55693de1fae0 0x55693de30850 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fedcc02a700 tx=0x7fedcc030de0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.310+0000 7fededfb9740 1 -- 172.21.15.72:0/4226316866 shutdown_connections 2024-04-26T23:46:06.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.310+0000 7fededfb9740 1 --2- 172.21.15.72:0/4226316866 >> [v2:172.21.15.72:6840/1356345077,v1:172.21.15.72:6841/1356345077] conn(0x55693deaa190 0x55693dea8ea0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.310+0000 7fededfb9740 1 --2- 172.21.15.72:0/4226316866 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55693de24c50 0x55693de2cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.310+0000 7fededfb9740 1 --2- 172.21.15.72:0/4226316866 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55693de1fae0 0x55693de30850 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.310+0000 7fedd2000640 0 client.15279 ms_handle_reset on v2:172.21.15.72:6840/1356345077 2024-04-26T23:46:06.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.310+0000 7fededfb9740 1 --2- 172.21.15.72:0/4226316866 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55693de1f0f0 0x55693de30310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.310+0000 7fededfb9740 1 -- 172.21.15.72:0/4226316866 >> 172.21.15.72:0/4226316866 conn(0x55693ddb3d10 msgr2=0x55693de19eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:46:06.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.310+0000 7fededfb9740 1 -- 172.21.15.72:0/4226316866 shutdown_connections 2024-04-26T23:46:06.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.310+0000 7fededfb9740 1 -- 172.21.15.72:0/4226316866 wait complete. 2024-04-26T23:46:06.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.310+0000 7fededfb9740 20 client.15279 trim_cache size 0 max 16384 2024-04-26T23:46:06.318 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-04-26T23:46:06.318 INFO:teuthology.orchestra.run.smithi072.stdout: "cluster_fsid": "3c8453ae-0425-11ef-bc93-c7b262605968", 2024-04-26T23:46:06.318 INFO:teuthology.orchestra.run.smithi072.stdout: "whoami": 1, 2024-04-26T23:46:06.318 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 24571, 2024-04-26T23:46:06.319 INFO:teuthology.orchestra.run.smithi072.stdout: "want_state": "up:active", 2024-04-26T23:46:06.319 INFO:teuthology.orchestra.run.smithi072.stdout: "state": "up:active", 2024-04-26T23:46:06.319 INFO:teuthology.orchestra.run.smithi072.stdout: "fs_name": "cephfs", 2024-04-26T23:46:06.319 INFO:teuthology.orchestra.run.smithi072.stdout: "rank_uptime": 158.14103388800001, 2024-04-26T23:46:06.319 INFO:teuthology.orchestra.run.smithi072.stdout: "mdsmap_epoch": 23, 2024-04-26T23:46:06.319 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_epoch": 93, 2024-04-26T23:46:06.319 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_epoch_barrier": 89, 2024-04-26T23:46:06.319 INFO:teuthology.orchestra.run.smithi072.stdout: "uptime": 192.16300544399999 2024-04-26T23:46:06.319 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-04-26T23:46:06.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.311+0000 7fedeb000640 1 -- 172.21.15.72:0/1328427111 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fedc40a3cc0 msgr2=0x7fedc40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:06.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.311+0000 7fedeb000640 1 --2- 172.21.15.72:0/1328427111 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fedc40a3cc0 0x7fedc40a6180 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x55693de20550 tx=0x7fedd40bd040 comp rx=0 tx=0).stop 2024-04-26T23:46:06.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.311+0000 7fedeb000640 1 -- 172.21.15.72:0/1328427111 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fede4102980 msgr2=0x7fede419ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:06.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.311+0000 7fedeb000640 1 --2- 172.21.15.72:0/1328427111 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fede4102980 0x7fede419ad70 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fedd400e9f0 tx=0x7fedd400eeb0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.311+0000 7fedeb000640 1 -- 172.21.15.72:0/1328427111 shutdown_connections 2024-04-26T23:46:06.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.311+0000 7fedeb000640 1 --2- 172.21.15.72:0/1328427111 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fedc40a3cc0 0x7fedc40a6180 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.311+0000 7fedeb000640 1 --2- 172.21.15.72:0/1328427111 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fede4103b70 0x7fede419b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.311+0000 7fedeb000640 1 --2- 172.21.15.72:0/1328427111 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fede4104530 0x7fede41a2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.311+0000 7fedeb000640 1 --2- 172.21.15.72:0/1328427111 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fede4102980 0x7fede419ad70 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.311+0000 7fedeb000640 1 -- 172.21.15.72:0/1328427111 >> 172.21.15.72:0/1328427111 conn(0x7fede40fdf10 msgr2=0x7fede4074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:46:06.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.311+0000 7fedeb000640 1 -- 172.21.15.72:0/1328427111 shutdown_connections 2024-04-26T23:46:06.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.311+0000 7fedeb000640 1 -- 172.21.15.72:0/1328427111 wait complete. 2024-04-26T23:46:06.332 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-26T23:46:06.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.526+0000 7ffaf9400640 1 -- 172.21.15.72:0/3274534671 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffaf4108d90 msgr2=0x7ffaf41091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:06.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.526+0000 7ffaf9400640 1 --2- 172.21.15.72:0/3274534671 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffaf4108d90 0x7ffaf41091b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ffae4009900 tx=0x7ffae402a400 comp rx=0 tx=0).stop 2024-04-26T23:46:06.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.526+0000 7ffaf9400640 1 -- 172.21.15.72:0/3274534671 shutdown_connections 2024-04-26T23:46:06.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.526+0000 7ffaf9400640 1 --2- 172.21.15.72:0/3274534671 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffaf4109f80 0x7ffaf410a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.526+0000 7ffaf9400640 1 --2- 172.21.15.72:0/3274534671 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffaf410a940 0x7ffaf41111d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.526+0000 7ffaf9400640 1 --2- 172.21.15.72:0/3274534671 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffaf4108d90 0x7ffaf41091b0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.526+0000 7ffaf9400640 1 -- 172.21.15.72:0/3274534671 >> 172.21.15.72:0/3274534671 conn(0x7ffaf4077d30 msgr2=0x7ffaf407a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:46:06.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.526+0000 7ffaf9400640 1 -- 172.21.15.72:0/3274534671 shutdown_connections 2024-04-26T23:46:06.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.526+0000 7ffaf9400640 1 -- 172.21.15.72:0/3274534671 wait complete. 2024-04-26T23:46:06.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.527+0000 7ffaf9400640 1 Processor -- start 2024-04-26T23:46:06.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.527+0000 7ffaf9400640 1 -- start start 2024-04-26T23:46:06.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.527+0000 7ffaf9400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffaf4108d90 0x7ffaf419efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.527+0000 7ffaf9400640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffaf4109f80 0x7ffaf419f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.527+0000 7ffaf9400640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffaf410a940 0x7ffaf41a6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.527+0000 7ffaf9400640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7ffaf4114900 con 0x7ffaf410a940 2024-04-26T23:46:06.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.527+0000 7ffaf9400640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7ffaf4114780 con 0x7ffaf4108d90 2024-04-26T23:46:06.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.527+0000 7ffaf9400640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7ffaf4114a80 con 0x7ffaf4109f80 2024-04-26T23:46:06.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.528+0000 7ffaf3e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffaf4108d90 0x7ffaf419efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.528+0000 7ffaf3e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffaf4108d90 0x7ffaf419efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:34350/0 (socket says 172.21.15.72:34350) 2024-04-26T23:46:06.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.528+0000 7ffaf3e00640 1 -- 172.21.15.72:0/1831687933 learned_addr learned my addr 172.21.15.72:0/1831687933 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:46:06.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.528+0000 7ffaeb400640 1 --2- 172.21.15.72:0/1831687933 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffaf4109f80 0x7ffaf419f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.528+0000 7ffaf8a00640 1 --2- 172.21.15.72:0/1831687933 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffaf410a940 0x7ffaf41a6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.528+0000 7ffaeb400640 1 -- 172.21.15.72:0/1831687933 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffaf4108d90 msgr2=0x7ffaf419efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:06.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.528+0000 7ffaeb400640 1 --2- 172.21.15.72:0/1831687933 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffaf4108d90 0x7ffaf419efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.528+0000 7ffaeb400640 1 -- 172.21.15.72:0/1831687933 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffaf410a940 msgr2=0x7ffaf41a6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:06.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.528+0000 7ffaeb400640 1 --2- 172.21.15.72:0/1831687933 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffaf410a940 0x7ffaf41a6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.528+0000 7ffaeb400640 1 -- 172.21.15.72:0/1831687933 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaf41a6be0 con 0x7ffaf4109f80 2024-04-26T23:46:06.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.528+0000 7ffaf3e00640 1 --2- 172.21.15.72:0/1831687933 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffaf4108d90 0x7ffaf419efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:46:06.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.528+0000 7ffaeb400640 1 --2- 172.21.15.72:0/1831687933 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffaf4109f80 0x7ffaf419f500 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ffadc00e9f0 tx=0x7ffadc00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:46:06.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.529+0000 7ffaf1600640 1 -- 172.21.15.72:0/1831687933 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffadc00ccc0 con 0x7ffaf4109f80 2024-04-26T23:46:06.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.529+0000 7ffaf1600640 1 -- 172.21.15.72:0/1831687933 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffadc005020 con 0x7ffaf4109f80 2024-04-26T23:46:06.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.529+0000 7ffaf1600640 1 -- 172.21.15.72:0/1831687933 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffadc004a00 con 0x7ffaf4109f80 2024-04-26T23:46:06.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.529+0000 7ffaf9400640 1 -- 172.21.15.72:0/1831687933 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffaf41a6ed0 con 0x7ffaf4109f80 2024-04-26T23:46:06.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.529+0000 7ffaf9400640 1 -- 172.21.15.72:0/1831687933 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffaf41a7340 con 0x7ffaf4109f80 2024-04-26T23:46:06.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.532+0000 7ffafd862740 1 Processor -- start 2024-04-26T23:46:06.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.532+0000 7ffafd862740 1 -- start start 2024-04-26T23:46:06.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.532+0000 7ffaf1600640 1 -- 172.21.15.72:0/1831687933 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ffadc004ba0 con 0x7ffaf4109f80 2024-04-26T23:46:06.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.532+0000 7ffafd862740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b0b12d32c0 0x55b0b12d36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.532+0000 7ffafd862740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b0b12d3cb0 0x55b0b12d87b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.532+0000 7ffafd862740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b0b12d8e20 0x55b0b12db2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.532+0000 7ffafd862740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55b0b1237600 con 0x55b0b12d32c0 2024-04-26T23:46:06.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.532+0000 7ffafd862740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55b0b11d4100 con 0x55b0b12d3cb0 2024-04-26T23:46:06.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.532+0000 7ffafd862740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55b0b12335c0 con 0x55b0b12d8e20 2024-04-26T23:46:06.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.533+0000 7ffaf8a00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b0b12d32c0 0x55b0b12d36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.533+0000 7ffaf8a00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b0b12d32c0 0x55b0b12d36e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58230/0 (socket says 172.21.15.72:58230) 2024-04-26T23:46:06.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.533+0000 7ffaf8a00640 1 -- 172.21.15.72:0/1123552252 learned_addr learned my addr 172.21.15.72:0/1123552252 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:46:06.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.533+0000 7ffaf3e00640 1 --2- 172.21.15.72:0/1123552252 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b0b12d3cb0 0x55b0b12d87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.533+0000 7ffaeb400640 1 --2- 172.21.15.72:0/1123552252 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b0b12d8e20 0x55b0b12db2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.533+0000 7ffaf8a00640 1 -- 172.21.15.72:0/1123552252 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b0b12d8e20 msgr2=0x55b0b12db2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:46:06.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.533+0000 7ffaf8a00640 1 --2- 172.21.15.72:0/1123552252 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b0b12d8e20 0x55b0b12db2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.533+0000 7ffaf8a00640 1 -- 172.21.15.72:0/1123552252 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b0b12d3cb0 msgr2=0x55b0b12d87b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:46:06.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.533+0000 7ffaf8a00640 1 --2- 172.21.15.72:0/1123552252 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b0b12d3cb0 0x55b0b12d87b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.533+0000 7ffaf8a00640 1 -- 172.21.15.72:0/1123552252 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0b0f492d0 con 0x55b0b12d32c0 2024-04-26T23:46:06.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.533+0000 7ffaf8a00640 1 --2- 172.21.15.72:0/1123552252 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b0b12d32c0 0x55b0b12d36e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ffaec009fd0 tx=0x7ffaec0118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a55be3e4d230845b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:46:06.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.534+0000 7ffae9200640 1 -- 172.21.15.72:0/1123552252 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaec004d20 con 0x55b0b12d32c0 2024-04-26T23:46:06.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.534+0000 7ffae9200640 1 -- 172.21.15.72:0/1123552252 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffaec004050 con 0x55b0b12d32c0 2024-04-26T23:46:06.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.534+0000 7ffae9200640 1 -- 172.21.15.72:0/1123552252 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaec005280 con 0x55b0b12d32c0 2024-04-26T23:46:06.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.534+0000 7ffaf1600640 1 --2- 172.21.15.72:0/1831687933 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ffad80a3c40 0x7ffad80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.534+0000 7ffaf1600640 1 -- 172.21.15.72:0/1831687933 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7ffadc0d0900 con 0x7ffaf4109f80 2024-04-26T23:46:06.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.534+0000 7ffaf3e00640 1 --2- 172.21.15.72:0/1831687933 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ffad80a3c40 0x7ffad80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.535+0000 7ffafd862740 1 -- 172.21.15.72:0/1123552252 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b0b12d32c0 msgr2=0x55b0b12d36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:06.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.535+0000 7ffafd862740 1 --2- 172.21.15.72:0/1123552252 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b0b12d32c0 0x55b0b12d36e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ffaec009fd0 tx=0x7ffaec0118d0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.535+0000 7ffafd862740 1 -- 172.21.15.72:0/1123552252 shutdown_connections 2024-04-26T23:46:06.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.535+0000 7ffafd862740 1 --2- 172.21.15.72:0/1123552252 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b0b12d3cb0 0x55b0b12d87b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.535+0000 7ffafd862740 1 --2- 172.21.15.72:0/1123552252 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b0b12d8e20 0x55b0b12db2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.535+0000 7ffafd862740 1 --2- 172.21.15.72:0/1123552252 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b0b12d32c0 0x55b0b12d36e0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.535+0000 7ffafd862740 1 -- 172.21.15.72:0/1123552252 >> 172.21.15.72:0/1123552252 conn(0x55b0b1267d10 msgr2=0x55b0b12ced40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:46:06.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.535+0000 7ffafd862740 1 -- 172.21.15.72:0/1123552252 shutdown_connections 2024-04-26T23:46:06.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.535+0000 7ffafd862740 1 -- 172.21.15.72:0/1123552252 wait complete. 2024-04-26T23:46:06.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.535+0000 7ffaf3e00640 1 --2- 172.21.15.72:0/1831687933 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ffad80a3c40 0x7ffad80a6100 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ffae4000c00 tx=0x7ffae4005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:46:06.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.535+0000 7ffafd862740 1 -- start start 2024-04-26T23:46:06.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.536+0000 7ffafd862740 1 Processor -- start 2024-04-26T23:46:06.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.536+0000 7ffafd862740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b0b12d32c0 0x55b0b12dd660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.536+0000 7ffafd862740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b0b12d3cb0 0x55b0b12ddba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.536+0000 7ffafd862740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b0b12d8e20 0x55b0b12e4be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.536+0000 7ffafd862740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55b0b12dbdd0 con 0x55b0b12d32c0 2024-04-26T23:46:06.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.536+0000 7ffafd862740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55b0b11bef70 con 0x55b0b12d3cb0 2024-04-26T23:46:06.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.536+0000 7ffafd862740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55b0b12dbf50 con 0x55b0b12d8e20 2024-04-26T23:46:06.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.536+0000 7ffafd862740 10 client.0 _ll_register_callbacks cb 0x55b0b122d480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:46:06.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.536+0000 7ffaf3e00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b0b12d32c0 0x55b0b12dd660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.536+0000 7ffaf3e00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b0b12d32c0 0x55b0b12dd660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58236/0 (socket says 172.21.15.72:58236) 2024-04-26T23:46:06.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.536+0000 7ffaf3e00640 1 -- 172.21.15.72:0/3417411520 learned_addr learned my addr 172.21.15.72:0/3417411520 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:46:06.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.536+0000 7ffaeb400640 1 --2- 172.21.15.72:0/3417411520 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b0b12d3cb0 0x55b0b12ddba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.536+0000 7ffaf8a00640 1 --2- 172.21.15.72:0/3417411520 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b0b12d8e20 0x55b0b12e4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.537+0000 7ffaf3e00640 1 -- 172.21.15.72:0/3417411520 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b0b12d8e20 msgr2=0x55b0b12e4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:46:06.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.537+0000 7ffaf3e00640 1 --2- 172.21.15.72:0/3417411520 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b0b12d8e20 0x55b0b12e4be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.537+0000 7ffaf3e00640 1 -- 172.21.15.72:0/3417411520 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b0b12d3cb0 msgr2=0x55b0b12ddba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:46:06.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.537+0000 7ffaf3e00640 1 --2- 172.21.15.72:0/3417411520 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b0b12d3cb0 0x55b0b12ddba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.537+0000 7ffaf3e00640 1 -- 172.21.15.72:0/3417411520 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0b10136b0 con 0x55b0b12d32c0 2024-04-26T23:46:06.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.537+0000 7ffaf8a00640 1 --2- 172.21.15.72:0/3417411520 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b0b12d8e20 0x55b0b12e4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:46:06.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.537+0000 7ffaeb400640 1 --2- 172.21.15.72:0/3417411520 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b0b12d3cb0 0x55b0b12ddba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:46:06.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.537+0000 7ffaf3e00640 1 --2- 172.21.15.72:0/3417411520 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b0b12d32c0 0x55b0b12dd660 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ffae402a9f0 tx=0x7ffae402a400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e48dcb56a0203dd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:46:06.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.537+0000 7ffae1600640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-04-26T23:46:06.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.537+0000 7ffae1600640 1 -- 172.21.15.72:0/3417411520 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffae4035050 con 0x55b0b12d32c0 2024-04-26T23:46:06.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.537+0000 7ffae1600640 1 -- 172.21.15.72:0/3417411520 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7ffae40317b0 con 0x55b0b12d32c0 2024-04-26T23:46:06.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.537+0000 7ffafd862740 1 -- 172.21.15.72:0/3417411520 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffaec004050 con 0x55b0b12d32c0 2024-04-26T23:46:06.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.538+0000 7ffae1600640 1 -- 172.21.15.72:0/3417411520 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7ffae40319b0 con 0x55b0b12d32c0 2024-04-26T23:46:06.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.538+0000 7ffafd862740 10 client.15291 fetch_fsmap learned FSMap version 23 2024-04-26T23:46:06.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.538+0000 7ffafd862740 1 -- 172.21.15.72:0/3417411520 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55b0b12e7c30 con 0x55b0b12d32c0 2024-04-26T23:46:06.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.538+0000 7ffae1600640 1 -- 172.21.15.72:0/3417411520 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffaec004050 con 0x55b0b12d32c0 2024-04-26T23:46:06.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.538+0000 7ffae1600640 1 -- 172.21.15.72:0/3417411520 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7ffae4053ce0 con 0x55b0b12d32c0 2024-04-26T23:46:06.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.538+0000 7ffafd862740 10 client.15291 fetch_fsmap finished waiting for FSMap version 23 2024-04-26T23:46:06.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.538+0000 7ffafd862740 10 client.15291 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T23:46:06.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.538+0000 7ffafd862740 20 client.15291 populate_metadata read hostname 'smithi072' 2024-04-26T23:46:06.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.538+0000 7ffafd862740 1 --2- 172.21.15.72:0/3417411520 >> [v2:172.21.15.76:6838/3405829058,v1:172.21.15.76:6839/3405829058] conn(0x55b0b12ea310 0x55b0b12eb780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.538+0000 7ffafd862740 4 client.15291 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T23:46:06.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.538+0000 7ffafd862740 1 -- 172.21.15.72:0/3417411520 --> [v2:172.21.15.76:6838/3405829058,v1:172.21.15.76:6839/3405829058] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b0b1245210 con 0x55b0b12ea310 2024-04-26T23:46:06.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.539+0000 7ffaeb400640 1 --2- 172.21.15.72:0/3417411520 >> [v2:172.21.15.76:6838/3405829058,v1:172.21.15.76:6839/3405829058] conn(0x55b0b12ea310 0x55b0b12eb780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.539+0000 7ffaeb400640 1 --2- 172.21.15.72:0/3417411520 >> [v2:172.21.15.76:6838/3405829058,v1:172.21.15.76:6839/3405829058] conn(0x55b0b12ea310 0x55b0b12eb780 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5a7d77265c6a61f server_cookie=635aae731f7cf466 in_seq=0 out_seq=0 2024-04-26T23:46:06.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.539+0000 7ffae1600640 10 client.15291 ms_handle_connect on v2:172.21.15.76:6838/3405829058 2024-04-26T23:46:06.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.540+0000 7ffae1600640 1 -- 172.21.15.72:0/3417411520 <== mds.0 v2:172.21.15.76:6838/3405829058 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b0b1245210 con 0x55b0b12ea310 2024-04-26T23:46:06.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.540+0000 7ffae1600640 10 client.15291 handle_command_reply: tid=0 2024-04-26T23:46:06.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.540+0000 7ffafd862740 1 client.15291 shutdown 2024-04-26T23:46:06.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.540+0000 7ffafd862740 1 -- 172.21.15.72:0/3417411520 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b0b12d32c0 msgr2=0x55b0b12dd660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:06.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.540+0000 7ffafd862740 1 --2- 172.21.15.72:0/3417411520 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b0b12d32c0 0x55b0b12dd660 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ffae402a9f0 tx=0x7ffae402a400 comp rx=0 tx=0).stop 2024-04-26T23:46:06.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.540+0000 7ffafd862740 1 -- 172.21.15.72:0/3417411520 shutdown_connections 2024-04-26T23:46:06.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.540+0000 7ffafd862740 1 --2- 172.21.15.72:0/3417411520 >> [v2:172.21.15.76:6838/3405829058,v1:172.21.15.76:6839/3405829058] conn(0x55b0b12ea310 0x55b0b12eb780 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.540+0000 7ffafd862740 1 --2- 172.21.15.72:0/3417411520 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b0b12d3cb0 0x55b0b12ddba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.540+0000 7ffafd862740 1 --2- 172.21.15.72:0/3417411520 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b0b12d8e20 0x55b0b12e4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.540+0000 7ffafd862740 1 --2- 172.21.15.72:0/3417411520 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b0b12d32c0 0x55b0b12dd660 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.540+0000 7ffae1600640 0 client.15291 ms_handle_reset on v2:172.21.15.76:6838/3405829058 2024-04-26T23:46:06.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.540+0000 7ffafd862740 1 -- 172.21.15.72:0/3417411520 >> 172.21.15.72:0/3417411520 conn(0x55b0b1267d10 msgr2=0x55b0b12cdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:46:06.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.540+0000 7ffafd862740 1 -- 172.21.15.72:0/3417411520 shutdown_connections 2024-04-26T23:46:06.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.540+0000 7ffafd862740 1 -- 172.21.15.72:0/3417411520 wait complete. 2024-04-26T23:46:06.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.541+0000 7ffafd862740 20 client.15291 trim_cache size 0 max 16384 2024-04-26T23:46:06.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.559+0000 7ffafd862740 1 Processor -- start 2024-04-26T23:46:06.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.559+0000 7ffafd862740 1 -- start start 2024-04-26T23:46:06.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.559+0000 7ffafd862740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b0b12d32c0 0x55b0b12e90f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.559+0000 7ffafd862740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b0b12d3cb0 0x55b0b12e4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.559+0000 7ffafd862740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b0b12d8e20 0x55b0b12d5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.559+0000 7ffafd862740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55b0b12e9ac0 con 0x55b0b12d8e20 2024-04-26T23:46:06.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.559+0000 7ffafd862740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55b0b12ea190 con 0x55b0b12d3cb0 2024-04-26T23:46:06.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.559+0000 7ffafd862740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55b0b12ea860 con 0x55b0b12d32c0 2024-04-26T23:46:06.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.560+0000 7ffaf8a00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b0b12d8e20 0x55b0b12d5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.560+0000 7ffaeb400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b0b12d3cb0 0x55b0b12e4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.560+0000 7ffaeb400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b0b12d3cb0 0x55b0b12e4ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:34370/0 (socket says 172.21.15.72:34370) 2024-04-26T23:46:06.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.560+0000 7ffaeb400640 1 -- 172.21.15.72:0/2858214191 learned_addr learned my addr 172.21.15.72:0/2858214191 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:46:06.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.560+0000 7ffaf3e00640 1 --2- 172.21.15.72:0/2858214191 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b0b12d32c0 0x55b0b12e90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.560+0000 7ffaf8a00640 1 -- 172.21.15.72:0/2858214191 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b0b12d32c0 msgr2=0x55b0b12e90f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:46:06.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.560+0000 7ffaf8a00640 1 --2- 172.21.15.72:0/2858214191 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b0b12d32c0 0x55b0b12e90f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.560+0000 7ffaf8a00640 1 -- 172.21.15.72:0/2858214191 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b0b12d3cb0 msgr2=0x55b0b12e4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:46:06.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.560+0000 7ffaf8a00640 1 --2- 172.21.15.72:0/2858214191 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b0b12d3cb0 0x55b0b12e4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.560+0000 7ffaf8a00640 1 -- 172.21.15.72:0/2858214191 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0b12eb3e0 con 0x55b0b12d8e20 2024-04-26T23:46:06.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.560+0000 7ffaf3e00640 1 --2- 172.21.15.72:0/2858214191 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b0b12d32c0 0x55b0b12e90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:46:06.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.561+0000 7ffaf8a00640 1 --2- 172.21.15.72:0/2858214191 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b0b12d8e20 0x55b0b12d5040 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ffaec01b040 tx=0x7ffaec003d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93c6bc3369c32483 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:46:06.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.561+0000 7ffae3e00640 1 -- 172.21.15.72:0/2858214191 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaec004a00 con 0x55b0b12d8e20 2024-04-26T23:46:06.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.561+0000 7ffae3e00640 1 -- 172.21.15.72:0/2858214191 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffaec004ba0 con 0x55b0b12d8e20 2024-04-26T23:46:06.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.561+0000 7ffae3e00640 1 -- 172.21.15.72:0/2858214191 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaec0052b0 con 0x55b0b12d8e20 2024-04-26T23:46:06.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.562+0000 7ffafd862740 1 -- 172.21.15.72:0/2858214191 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b0b12d8e20 msgr2=0x55b0b12d5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:06.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.562+0000 7ffafd862740 1 --2- 172.21.15.72:0/2858214191 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b0b12d8e20 0x55b0b12d5040 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ffaec01b040 tx=0x7ffaec003d40 comp rx=0 tx=0).stop 2024-04-26T23:46:06.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.562+0000 7ffafd862740 1 -- 172.21.15.72:0/2858214191 shutdown_connections 2024-04-26T23:46:06.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.562+0000 7ffafd862740 1 --2- 172.21.15.72:0/2858214191 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b0b12d3cb0 0x55b0b12e4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.562+0000 7ffafd862740 1 --2- 172.21.15.72:0/2858214191 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b0b12d8e20 0x55b0b12d5040 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.562+0000 7ffafd862740 1 --2- 172.21.15.72:0/2858214191 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b0b12d32c0 0x55b0b12e90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.562+0000 7ffafd862740 1 -- 172.21.15.72:0/2858214191 >> 172.21.15.72:0/2858214191 conn(0x55b0b1267d10 msgr2=0x55b0b12c9df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:46:06.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.562+0000 7ffafd862740 1 -- 172.21.15.72:0/2858214191 shutdown_connections 2024-04-26T23:46:06.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.562+0000 7ffafd862740 1 -- 172.21.15.72:0/2858214191 wait complete. 2024-04-26T23:46:06.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.562+0000 7ffafd862740 1 -- start start 2024-04-26T23:46:06.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.562+0000 7ffafd862740 1 Processor -- start 2024-04-26T23:46:06.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.563+0000 7ffafd862740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b0b12d32c0 0x55b0b12e03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.563+0000 7ffafd862740 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b0b12d3cb0 0x55b0b12e0920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.563+0000 7ffafd862740 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b0b12d8e20 0x55b0b12e38d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.563+0000 7ffafd862740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55b0b12eb600 con 0x55b0b12d32c0 2024-04-26T23:46:06.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.563+0000 7ffafd862740 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x55b0b12eaf30 con 0x55b0b12d3cb0 2024-04-26T23:46:06.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.563+0000 7ffafd862740 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x55b0b12ebcd0 con 0x55b0b12d8e20 2024-04-26T23:46:06.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.563+0000 7ffafd862740 10 client.0 _ll_register_callbacks cb 0x7ffae4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T23:46:06.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.563+0000 7ffaf3e00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b0b12d32c0 0x55b0b12e03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.563+0000 7ffaf3e00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b0b12d32c0 0x55b0b12e03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58262/0 (socket says 172.21.15.72:58262) 2024-04-26T23:46:06.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.563+0000 7ffaf3e00640 1 -- 172.21.15.72:0/770713438 learned_addr learned my addr 172.21.15.72:0/770713438 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:46:06.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.563+0000 7ffaeb400640 1 --2- 172.21.15.72:0/770713438 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b0b12d3cb0 0x55b0b12e0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.563+0000 7ffaf3e00640 1 -- 172.21.15.72:0/770713438 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b0b12d8e20 msgr2=0x55b0b12e38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:46:06.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.563+0000 7ffaf3e00640 1 --2- 172.21.15.72:0/770713438 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b0b12d8e20 0x55b0b12e38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.563+0000 7ffaf3e00640 1 -- 172.21.15.72:0/770713438 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b0b12d3cb0 msgr2=0x55b0b12e0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T23:46:06.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.563+0000 7ffaf3e00640 1 --2- 172.21.15.72:0/770713438 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b0b12d3cb0 0x55b0b12e0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.563+0000 7ffaf3e00640 1 -- 172.21.15.72:0/770713438 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0b12e3ec0 con 0x55b0b12d32c0 2024-04-26T23:46:06.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.563+0000 7ffaeb400640 1 --2- 172.21.15.72:0/770713438 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b0b12d3cb0 0x55b0b12e0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T23:46:06.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.564+0000 7ffaf3e00640 1 --2- 172.21.15.72:0/770713438 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b0b12d32c0 0x55b0b12e03e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ffae4006010 tx=0x7ffae4049e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4657fa51da875ecf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:46:06.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.564+0000 7ffae1600640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-04-26T23:46:06.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.564+0000 7ffae1600640 1 -- 172.21.15.72:0/770713438 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffae4065060 con 0x55b0b12d32c0 2024-04-26T23:46:06.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.564+0000 7ffae1600640 1 -- 172.21.15.72:0/770713438 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7ffae4060480 con 0x55b0b12d32c0 2024-04-26T23:46:06.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.564+0000 7ffafd862740 1 -- 172.21.15.72:0/770713438 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffaec004ba0 con 0x55b0b12d32c0 2024-04-26T23:46:06.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.564+0000 7ffae1600640 1 -- 172.21.15.72:0/770713438 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7ffae406c050 con 0x55b0b12d32c0 2024-04-26T23:46:06.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.564+0000 7ffafd862740 10 client.15297 fetch_fsmap learned FSMap version 23 2024-04-26T23:46:06.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.564+0000 7ffafd862740 1 -- 172.21.15.72:0/770713438 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55b0b1350330 con 0x55b0b12d32c0 2024-04-26T23:46:06.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.564+0000 7ffae1600640 1 -- 172.21.15.72:0/770713438 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffaec004ba0 con 0x55b0b12d32c0 2024-04-26T23:46:06.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.565+0000 7ffae1600640 1 -- 172.21.15.72:0/770713438 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7ffae4053600 con 0x55b0b12d32c0 2024-04-26T23:46:06.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.565+0000 7ffafd862740 10 client.15297 fetch_fsmap finished waiting for FSMap version 23 2024-04-26T23:46:06.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.565+0000 7ffafd862740 10 client.15297 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T23:46:06.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.565+0000 7ffafd862740 20 client.15297 populate_metadata read hostname 'smithi072' 2024-04-26T23:46:06.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.565+0000 7ffafd862740 1 --2- 172.21.15.72:0/770713438 >> [v2:172.21.15.76:6838/3405829058,v1:172.21.15.76:6839/3405829058] conn(0x55b0b1354fe0 0x55b0b1353cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:06.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.565+0000 7ffafd862740 4 client.15297 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T23:46:06.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.565+0000 7ffafd862740 1 -- 172.21.15.72:0/770713438 --> [v2:172.21.15.76:6838/3405829058,v1:172.21.15.76:6839/3405829058] -- command(tid 0: {"prefix": "status"}) -- 0x55b0b123f080 con 0x55b0b1354fe0 2024-04-26T23:46:06.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.565+0000 7ffaeb400640 1 --2- 172.21.15.72:0/770713438 >> [v2:172.21.15.76:6838/3405829058,v1:172.21.15.76:6839/3405829058] conn(0x55b0b1354fe0 0x55b0b1353cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:06.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.566+0000 7ffaeb400640 1 --2- 172.21.15.72:0/770713438 >> [v2:172.21.15.76:6838/3405829058,v1:172.21.15.76:6839/3405829058] conn(0x55b0b1354fe0 0x55b0b1353cf0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4328af0fbb58d477 server_cookie=7625e83e8ccedf00 in_seq=0 out_seq=0 2024-04-26T23:46:06.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.566+0000 7ffae1600640 10 client.15297 ms_handle_connect on v2:172.21.15.76:6838/3405829058 2024-04-26T23:46:06.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.566+0000 7ffae1600640 1 -- 172.21.15.72:0/770713438 <== mds.0 v2:172.21.15.76:6838/3405829058 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55b0b123f080 con 0x55b0b1354fe0 2024-04-26T23:46:06.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.566+0000 7ffae1600640 10 client.15297 handle_command_reply: tid=0 2024-04-26T23:46:06.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.566+0000 7ffafd862740 1 client.15297 shutdown 2024-04-26T23:46:06.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.566+0000 7ffafd862740 1 -- 172.21.15.72:0/770713438 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b0b12d32c0 msgr2=0x55b0b12e03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:06.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.566+0000 7ffafd862740 1 --2- 172.21.15.72:0/770713438 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b0b12d32c0 0x55b0b12e03e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ffae4006010 tx=0x7ffae4049e70 comp rx=0 tx=0).stop 2024-04-26T23:46:06.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.566+0000 7ffafd862740 1 -- 172.21.15.72:0/770713438 shutdown_connections 2024-04-26T23:46:06.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.566+0000 7ffafd862740 1 --2- 172.21.15.72:0/770713438 >> [v2:172.21.15.76:6838/3405829058,v1:172.21.15.76:6839/3405829058] conn(0x55b0b1354fe0 0x55b0b1353cf0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.566+0000 7ffafd862740 1 --2- 172.21.15.72:0/770713438 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x55b0b12d3cb0 0x55b0b12e0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.566+0000 7ffafd862740 1 --2- 172.21.15.72:0/770713438 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55b0b12d8e20 0x55b0b12e38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.566+0000 7ffae1600640 0 client.15297 ms_handle_reset on v2:172.21.15.76:6838/3405829058 2024-04-26T23:46:06.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.566+0000 7ffafd862740 1 --2- 172.21.15.72:0/770713438 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b0b12d32c0 0x55b0b12e03e0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.566+0000 7ffafd862740 1 -- 172.21.15.72:0/770713438 >> 172.21.15.72:0/770713438 conn(0x55b0b1267d10 msgr2=0x55b0b12e90f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:46:06.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.566+0000 7ffafd862740 1 -- 172.21.15.72:0/770713438 shutdown_connections 2024-04-26T23:46:06.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.566+0000 7ffafd862740 1 -- 172.21.15.72:0/770713438 wait complete. 2024-04-26T23:46:06.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.566+0000 7ffafd862740 20 client.15297 trim_cache size 0 max 16384 2024-04-26T23:46:06.575 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-04-26T23:46:06.575 INFO:teuthology.orchestra.run.smithi072.stdout: "cluster_fsid": "3c8453ae-0425-11ef-bc93-c7b262605968", 2024-04-26T23:46:06.575 INFO:teuthology.orchestra.run.smithi072.stdout: "whoami": 0, 2024-04-26T23:46:06.575 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 24590, 2024-04-26T23:46:06.575 INFO:teuthology.orchestra.run.smithi072.stdout: "want_state": "up:active", 2024-04-26T23:46:06.576 INFO:teuthology.orchestra.run.smithi072.stdout: "state": "up:active", 2024-04-26T23:46:06.576 INFO:teuthology.orchestra.run.smithi072.stdout: "fs_name": "cephfs", 2024-04-26T23:46:06.576 INFO:teuthology.orchestra.run.smithi072.stdout: "rank_uptime": 183.10649785699999, 2024-04-26T23:46:06.576 INFO:teuthology.orchestra.run.smithi072.stdout: "mdsmap_epoch": 23, 2024-04-26T23:46:06.576 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_epoch": 93, 2024-04-26T23:46:06.576 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_epoch_barrier": 89, 2024-04-26T23:46:06.576 INFO:teuthology.orchestra.run.smithi072.stdout: "uptime": 189.605889117 2024-04-26T23:46:06.576 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-04-26T23:46:06.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.567+0000 7ffaf9400640 1 -- 172.21.15.72:0/1831687933 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ffad80a3c40 msgr2=0x7ffad80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:06.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.567+0000 7ffaf9400640 1 --2- 172.21.15.72:0/1831687933 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ffad80a3c40 0x7ffad80a6100 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ffae4000c00 tx=0x7ffae4005a20 comp rx=0 tx=0).stop 2024-04-26T23:46:06.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.567+0000 7ffaf9400640 1 -- 172.21.15.72:0/1831687933 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffaf4109f80 msgr2=0x7ffaf419f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:06.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.567+0000 7ffaf9400640 1 --2- 172.21.15.72:0/1831687933 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffaf4109f80 0x7ffaf419f500 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ffadc00e9f0 tx=0x7ffadc00eeb0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.567+0000 7ffaf9400640 1 -- 172.21.15.72:0/1831687933 shutdown_connections 2024-04-26T23:46:06.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.567+0000 7ffaf9400640 1 --2- 172.21.15.72:0/1831687933 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7ffad80a3c40 0x7ffad80a6100 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.567+0000 7ffaf9400640 1 --2- 172.21.15.72:0/1831687933 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffaf410a940 0x7ffaf41a6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.567+0000 7ffaf9400640 1 --2- 172.21.15.72:0/1831687933 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffaf4109f80 0x7ffaf419f500 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.567+0000 7ffaf9400640 1 --2- 172.21.15.72:0/1831687933 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffaf4108d90 0x7ffaf419efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:06.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.567+0000 7ffaf9400640 1 -- 172.21.15.72:0/1831687933 >> 172.21.15.72:0/1831687933 conn(0x7ffaf4077d30 msgr2=0x7ffaf410d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:46:06.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.567+0000 7ffaf9400640 1 -- 172.21.15.72:0/1831687933 shutdown_connections 2024-04-26T23:46:06.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:06.567+0000 7ffaf9400640 1 -- 172.21.15.72:0/1831687933 wait complete. 2024-04-26T23:46:06.587 INFO:teuthology.run_tasks:Running task sequential... 2024-04-26T23:46:06.602 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-26T23:46:06.602 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-26T23:46:06.603 INFO:teuthology.task.print:Enabling mgr modules 2024-04-26T23:46:06.603 INFO:teuthology.run_tasks:Running task kclient... 2024-04-26T23:46:06.673 INFO:tasks.kclient:Mounting kernel clients... 2024-04-26T23:46:06.673 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-26T23:46:06.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:46:06.674 DEBUG:teuthology.orchestra.run.smithi072:> ip netns list 2024-04-26T23:46:06.725 INFO:teuthology.orchestra.run.smithi072.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-04-26T23:46:06.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:46:06.726 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-26T23:46:06.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:46:06.798 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link delete ceph-brx 2024-04-26T23:46:06.854 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:06 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/881424407' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:46:06.911 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'wsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-04-26T23:46:06.911 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-26T23:46:06.911 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-26T23:46:06.911 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-26T23:46:06.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:06 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/881424407' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:46:07.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:06 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/881424407' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T23:46:07.332 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T23:46:07.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:07 smithi072 ceph-mon[28184]: pgmap v417: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:46:07.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:07 smithi076 ceph-mon[32774]: pgmap v417: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:46:08.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:07 smithi136 ceph-mon[33050]: pgmap v417: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:46:09.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:09.758+0000 7f771fc00700 1 -- 172.21.15.72:0/510503958 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7718108370 msgr2=0x7f7718108750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:09.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:09.758+0000 7f771fc00700 1 --2- 172.21.15.72:0/510503958 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7718108370 0x7f7718108750 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f770c009990 tx=0x7f770c009ca0 comp rx=0 tx=0).stop 2024-04-26T23:46:09.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:09.758+0000 7f771fc00700 1 -- 172.21.15.72:0/510503958 shutdown_connections 2024-04-26T23:46:09.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:09.758+0000 7f771fc00700 1 --2- 172.21.15.72:0/510503958 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f771806c680 0x7f7718105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:09.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:09.758+0000 7f771fc00700 1 --2- 172.21.15.72:0/510503958 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7718108370 0x7f7718108750 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:09.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:09.758+0000 7f771fc00700 1 --2- 172.21.15.72:0/510503958 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f771806baf0 0x7f771806bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:09.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:09.758+0000 7f771fc00700 1 -- 172.21.15.72:0/510503958 >> 172.21.15.72:0/510503958 conn(0x7f77180792d0 msgr2=0x7f77180796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:46:09.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:09.758+0000 7f771fc00700 1 -- 172.21.15.72:0/510503958 shutdown_connections 2024-04-26T23:46:09.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:09.758+0000 7f771fc00700 1 -- 172.21.15.72:0/510503958 wait complete. 2024-04-26T23:46:09.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:09.759+0000 7f771fc00700 1 Processor -- start 2024-04-26T23:46:09.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:09.759+0000 7f771fc00700 1 -- start start 2024-04-26T23:46:09.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:09.760+0000 7f771fc00700 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f771806baf0 0x7f77181a01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:09.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:09.760+0000 7f771fc00700 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f771806c680 0x7f77181a0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:09.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:09.760+0000 7f771fc00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7718108370 0x7f77181a4a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:09.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:09.760+0000 7f771fc00700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f7718119c70 con 0x7f7718108370 2024-04-26T23:46:09.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:09.760+0000 7f771fc00700 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f7718119af0 con 0x7f771806c680 2024-04-26T23:46:09.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:09.760+0000 7f771fc00700 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f7718119df0 con 0x7f771806baf0 2024-04-26T23:46:09.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:09.760+0000 7f771dc00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7718108370 0x7f77181a4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:09.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:09.760+0000 7f771dc00700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7718108370 0x7f77181a4a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:49180/0 (socket says 172.21.15.72:49180) 2024-04-26T23:46:09.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:09.760+0000 7f771dc00700 1 -- 172.21.15.72:0/4103464133 learned_addr learned my addr 172.21.15.72:0/4103464133 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:46:09.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:09.760+0000 7f771d200700 1 --2- 172.21.15.72:0/4103464133 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f771806baf0 0x7f77181a01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:09.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:09.760+0000 7f7717e00700 1 --2- 172.21.15.72:0/4103464133 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f771806c680 0x7f77181a0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:09.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:09.761+0000 7f771d200700 1 -- 172.21.15.72:0/4103464133 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f771806c680 msgr2=0x7f77181a0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:09.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:09.761+0000 7f771d200700 1 --2- 172.21.15.72:0/4103464133 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f771806c680 0x7f77181a0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:09.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:09.761+0000 7f771d200700 1 -- 172.21.15.72:0/4103464133 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7718108370 msgr2=0x7f77181a4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:09.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:09.761+0000 7f771d200700 1 --2- 172.21.15.72:0/4103464133 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7718108370 0x7f77181a4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:09.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:09.761+0000 7f771d200700 1 -- 172.21.15.72:0/4103464133 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77181a51d0 con 0x7f771806baf0 2024-04-26T23:46:09.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:09.761+0000 7f771d200700 1 --2- 172.21.15.72:0/4103464133 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f771806baf0 0x7f77181a01c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f770c000c00 tx=0x7f770c0047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:46:09.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:09.761+0000 7f7715600700 1 -- 172.21.15.72:0/4103464133 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f770c026070 con 0x7f771806baf0 2024-04-26T23:46:09.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:09.761+0000 7f7715600700 1 -- 172.21.15.72:0/4103464133 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f770c021410 con 0x7f771806baf0 2024-04-26T23:46:09.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:09.761+0000 7f771fc00700 1 -- 172.21.15.72:0/4103464133 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77181a5460 con 0x7f771806baf0 2024-04-26T23:46:09.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:09.761+0000 7f7715600700 1 -- 172.21.15.72:0/4103464133 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f770c017650 con 0x7f771806baf0 2024-04-26T23:46:09.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:09.762+0000 7f771fc00700 1 -- 172.21.15.72:0/4103464133 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77181acd40 con 0x7f771806baf0 2024-04-26T23:46:09.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:09.763+0000 7f771fc00700 1 -- 172.21.15.72:0/4103464133 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7718102960 con 0x7f771806baf0 2024-04-26T23:46:09.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:09.764+0000 7f7715600700 1 -- 172.21.15.72:0/4103464133 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f770c029070 con 0x7f771806baf0 2024-04-26T23:46:09.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:09.766+0000 7f7715600700 1 --2- 172.21.15.72:0/4103464133 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f77000a3d50 0x7f77000a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:09.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:09.766+0000 7f7715600700 1 -- 172.21.15.72:0/4103464133 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f770c0d1600 con 0x7f771806baf0 2024-04-26T23:46:09.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:09.766+0000 7f7717e00700 1 --2- 172.21.15.72:0/4103464133 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f77000a3d50 0x7f77000a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:09.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:09.767+0000 7f7717e00700 1 --2- 172.21.15.72:0/4103464133 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f77000a3d50 0x7f77000a6210 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f7718079a10 tx=0x7f7704009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:46:09.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:09.770+0000 7f7715600700 1 -- 172.21.15.72:0/4103464133 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f770c05e060 con 0x7f771806baf0 2024-04-26T23:46:09.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:09 smithi076 ceph-mon[32774]: pgmap v418: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T23:46:10.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:09 smithi072 ceph-mon[28184]: pgmap v418: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T23:46:10.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:10.043+0000 7f771fc00700 1 -- 172.21.15.72:0/4103464133 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f77181acf10 con 0x7f771806baf0 2024-04-26T23:46:10.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:10.044+0000 7f7715600700 1 -- 172.21.15.72:0/4103464133 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v23) ==== 53+0+88 (secure 0 0 0) 0x7f770c013070 con 0x7f771806baf0 2024-04-26T23:46:10.049 INFO:teuthology.orchestra.run.smithi072.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-26T23:46:10.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:09 smithi136 ceph-mon[33050]: pgmap v418: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T23:46:10.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:10.047+0000 7f771fc00700 1 -- 172.21.15.72:0/4103464133 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f77000a3d50 msgr2=0x7f77000a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:10.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:10.047+0000 7f771fc00700 1 --2- 172.21.15.72:0/4103464133 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f77000a3d50 0x7f77000a6210 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f7718079a10 tx=0x7f7704009260 comp rx=0 tx=0).stop 2024-04-26T23:46:10.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:10.047+0000 7f771fc00700 1 -- 172.21.15.72:0/4103464133 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f771806baf0 msgr2=0x7f77181a01c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:10.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:10.047+0000 7f771fc00700 1 --2- 172.21.15.72:0/4103464133 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f771806baf0 0x7f77181a01c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f770c000c00 tx=0x7f770c0047b0 comp rx=0 tx=0).stop 2024-04-26T23:46:10.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:10.047+0000 7f771fc00700 1 -- 172.21.15.72:0/4103464133 shutdown_connections 2024-04-26T23:46:10.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:10.047+0000 7f771fc00700 1 --2- 172.21.15.72:0/4103464133 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f77000a3d50 0x7f77000a6210 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:10.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:10.047+0000 7f771fc00700 1 --2- 172.21.15.72:0/4103464133 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7718108370 0x7f77181a4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:10.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:10.047+0000 7f771fc00700 1 --2- 172.21.15.72:0/4103464133 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f771806baf0 0x7f77181a01c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:10.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:10.047+0000 7f771fc00700 1 --2- 172.21.15.72:0/4103464133 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f771806c680 0x7f77181a0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:10.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:10.047+0000 7f771fc00700 1 -- 172.21.15.72:0/4103464133 >> 172.21.15.72:0/4103464133 conn(0x7f77180792d0 msgr2=0x7f77181016f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:46:10.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:10.047+0000 7f771fc00700 1 -- 172.21.15.72:0/4103464133 shutdown_connections 2024-04-26T23:46:10.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:10.047+0000 7f771fc00700 1 -- 172.21.15.72:0/4103464133 wait complete. 2024-04-26T23:46:10.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:10 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/4103464133' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-26T23:46:10.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:10 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/4103464133' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-26T23:46:10.999 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-26T23:46:10.999 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-26T23:46:10.999 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi072.front.sepia.ceph.com 2024-04-26T23:46:10.999 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-26T23:46:10.999 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-26T23:46:10.999 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-26T23:46:10.999 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-26T23:46:10.999 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-26T23:46:10.999 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-26T23:46:10.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:46:10.999 DEBUG:teuthology.orchestra.run.smithi072:> ip addr 2024-04-26T23:46:11.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:10 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/4103464133' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-26T23:46:11.057 INFO:teuthology.orchestra.run.smithi072.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-26T23:46:11.057 INFO:teuthology.orchestra.run.smithi072.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-26T23:46:11.057 INFO:teuthology.orchestra.run.smithi072.stdout: inet 127.0.0.1/8 scope host lo 2024-04-26T23:46:11.057 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:46:11.057 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 ::1/128 scope host 2024-04-26T23:46:11.057 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:46:11.057 INFO:teuthology.orchestra.run.smithi072.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T23:46:11.057 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:d8:da:f4 brd ff:ff:ff:ff:ff:ff 2024-04-26T23:46:11.057 INFO:teuthology.orchestra.run.smithi072.stdout: altname enp7s0f0 2024-04-26T23:46:11.057 INFO:teuthology.orchestra.run.smithi072.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T23:46:11.058 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:d8:da:f5 brd ff:ff:ff:ff:ff:ff 2024-04-26T23:46:11.058 INFO:teuthology.orchestra.run.smithi072.stdout: altname enp7s0f1 2024-04-26T23:46:11.058 INFO:teuthology.orchestra.run.smithi072.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T23:46:11.058 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:8f:b7:64 brd ff:ff:ff:ff:ff:ff 2024-04-26T23:46:11.058 INFO:teuthology.orchestra.run.smithi072.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-26T23:46:11.058 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:8f:b7:65 brd ff:ff:ff:ff:ff:ff 2024-04-26T23:46:11.058 INFO:teuthology.orchestra.run.smithi072.stdout: inet 172.21.15.72/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-26T23:46:11.058 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft 41821sec preferred_lft 41821sec 2024-04-26T23:46:11.058 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::ec4:7aff:fe8f:b765/64 scope link 2024-04-26T23:46:11.058 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-04-26T23:46:11.059 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-26T23:46:11.059 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T23:46:11.059 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-04-26T23:46:11.059 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link add name ceph-brx type bridge 2024-04-26T23:46:11.059 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip addr flush dev ceph-brx 2024-04-26T23:46:11.060 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link set ceph-brx up 2024-04-26T23:46:11.060 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-26T23:46:11.060 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-04-26T23:46:11.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:11 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T23:46:11.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:46:11.298 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:46:11.298 DEBUG:teuthology.orchestra.run.smithi072:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-26T23:46:11.364 INFO:teuthology.orchestra.run.smithi072.stdout:1 2024-04-26T23:46:11.367 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:46:11.368 DEBUG:teuthology.orchestra.run.smithi072:> route 2024-04-26T23:46:11.417 INFO:teuthology.orchestra.run.smithi072.stdout:Kernel IP routing table 2024-04-26T23:46:11.417 INFO:teuthology.orchestra.run.smithi072.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-26T23:46:11.417 INFO:teuthology.orchestra.run.smithi072.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-26T23:46:11.417 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-26T23:46:11.417 INFO:teuthology.orchestra.run.smithi072.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-26T23:46:11.418 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T23:46:11.418 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-04-26T23:46:11.418 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-26T23:46:11.418 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-26T23:46:11.418 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-26T23:46:11.418 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-04-26T23:46:11.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:11 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T23:46:11.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:46:11.625 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:46:11.625 DEBUG:teuthology.orchestra.run.smithi072:> ip netns list 2024-04-26T23:46:11.670 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:46:11.670 DEBUG:teuthology.orchestra.run.smithi072:> ip netns list-id 2024-04-26T23:46:11.765 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T23:46:11.766 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-04-26T23:46:11.766 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-26T23:46:11.766 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-26T23:46:11.766 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-04-26T23:46:11.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:11 smithi072 ceph-mon[28184]: pgmap v419: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:46:11.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:11 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T23:46:11.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:11 smithi076 ceph-mon[32774]: pgmap v419: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:46:11.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:46:11.954 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-26T23:46:11.954 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T23:46:11.954 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-04-26T23:46:11.954 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-26T23:46:11.955 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-26T23:46:11.955 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-26T23:46:11.955 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-26T23:46:11.955 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-26T23:46:11.955 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-04-26T23:46:12.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:11 smithi136 ceph-mon[33050]: pgmap v419: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:46:12.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:12 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T23:46:12.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:46:12.271 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T23:46:12.271 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-04-26T23:46:12.271 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link set brx.0 up 2024-04-26T23:46:12.272 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link set dev brx.0 master ceph-brx 2024-04-26T23:46:12.272 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-04-26T23:46:12.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:12 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T23:46:12.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:46:12.472 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T23:46:12.472 DEBUG:teuthology.orchestra.run.smithi072:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-26T23:46:12.518 INFO:teuthology.orchestra.run.smithi072.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-26T23:46:12.519 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T23:46:12.519 DEBUG:teuthology.orchestra.run.smithi072:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-26T23:46:12.616 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-04-26T23:46:12.616 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-04-26T23:46:12.616 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/b5db3a0e-4c33-4081-af16-2a62694d4438 2024-04-26T23:46:12.616 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:46:12.616 DEBUG:teuthology.orchestra.run.smithi072:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/b5db3a0e-4c33-4081-af16-2a62694d4438 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync 2024-04-26T23:46:12.739 INFO:teuthology.orchestra.run.smithi072.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync 2024-04-26T23:46:12.739 INFO:teuthology.orchestra.run.smithi072.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync". 2024-04-26T23:46:12.739 INFO:teuthology.orchestra.run.smithi072.stdout:invalid new device string format 2024-04-26T23:46:13.681 INFO:teuthology.orchestra.run.smithi072.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync 2024-04-26T23:46:13.681 INFO:teuthology.orchestra.run.smithi072.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync". 2024-04-26T23:46:13.681 INFO:teuthology.orchestra.run.smithi072.stdout:invalid new device string format 2024-04-26T23:46:13.681 INFO:teuthology.orchestra.run.smithi072.stdout:mount.ceph: resolved to: "172.21.15.72:3300,172.21.15.76:3300,172.21.15.136:3300" 2024-04-26T23:46:13.681 INFO:teuthology.orchestra.run.smithi072.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.72:3300,172.21.15.76:3300,172.21.15.136:3300:/volumes/qa/sv_1/b5db3a0e-4c33-4081-af16-2a62694d4438 2024-04-26T23:46:13.681 INFO:teuthology.orchestra.run.smithi072.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync,key=0,fsid=3c8453ae-0425-11ef-bc93-c7b262605968" will pass to kernel 2024-04-26T23:46:13.686 INFO:teuthology.orchestra.run.smithi072.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-26T23:46:13.686 INFO:teuthology.orchestra.run.smithi072.stdout: You just mounted a file system that supports labels which does not 2024-04-26T23:46:13.686 INFO:teuthology.orchestra.run.smithi072.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-26T23:46:13.686 INFO:teuthology.orchestra.run.smithi072.stdout: applications will generate AVC messages and not be allowed access to 2024-04-26T23:46:13.686 INFO:teuthology.orchestra.run.smithi072.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-26T23:46:13.690 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-26T23:46:13.690 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T23:46:13.691 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-26T23:46:13.780 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T23:46:13.867 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:13 smithi136 ceph-mon[33050]: pgmap v420: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:46:13.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:13 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T23:46:13.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:46:13.902 INFO:teuthology.orchestra.run.smithi072.stdout:client25409 2024-04-26T23:46:13.902 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-26T23:46:13.902 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T23:46:13.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:13 smithi076 ceph-mon[32774]: pgmap v420: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:46:13.952 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T23:46:14.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:13 smithi072 ceph-mon[28184]: pgmap v420: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:46:14.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:14 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T23:46:14.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:46:14.120 INFO:teuthology.orchestra.run.smithi072.stdout:client25409 2024-04-26T23:46:14.120 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T23:46:14.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:14 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T23:46:14.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:46:14.240 INFO:teuthology.orchestra.run.smithi072.stdout:client25409 2024-04-26T23:46:14.241 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/3c8453ae-0425-11ef-bc93-c7b262605968.client25409/status') 2024-04-26T23:46:14.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:14 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T23:46:14.372 INFO:teuthology.orchestra.run.smithi072.stdout:instance: client.25409 (3)172.21.15.72:0/3294152383 2024-04-26T23:46:14.373 INFO:teuthology.orchestra.run.smithi072.stdout:blocklisted: false 2024-04-26T23:46:14.373 INFO:teuthology.orchestra.run.smithi072.stderr:0+1 records in 2024-04-26T23:46:14.373 INFO:teuthology.orchestra.run.smithi072.stderr:0+1 records out 2024-04-26T23:46:14.373 INFO:teuthology.orchestra.run.smithi072.stderr:71 bytes copied, 7.9929e-05 s, 888 kB/s 2024-04-26T23:46:14.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T23:46:14.385 INFO:teuthology.run_tasks:Running task exec... 2024-04-26T23:46:14.396 INFO:teuthology.task.exec:Executing custom commands... 2024-04-26T23:46:14.396 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi072.front.sepia.ceph.com 2024-04-26T23:46:14.396 DEBUG:teuthology.orchestra.run.smithi072:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-04-26T23:46:14.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:14.593+0000 7f8056800640 1 -- 172.21.15.72:0/3164814978 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f805010a1b0 msgr2=0x7f805010a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:14.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:14.593+0000 7f8056800640 1 --2- 172.21.15.72:0/3164814978 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f805010a1b0 0x7f805010a630 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8038009900 tx=0x7f803802a400 comp rx=0 tx=0).stop 2024-04-26T23:46:14.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:14.593+0000 7f8056800640 1 -- 172.21.15.72:0/3164814978 shutdown_connections 2024-04-26T23:46:14.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:14.593+0000 7f8056800640 1 --2- 172.21.15.72:0/3164814978 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f805010a1b0 0x7f805010a630 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:14.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:14.593+0000 7f8056800640 1 --2- 172.21.15.72:0/3164814978 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f805010ab70 0x7f8050111400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:14.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:14.593+0000 7f8056800640 1 --2- 172.21.15.72:0/3164814978 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8050108fc0 0x7f80501093e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:14.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:14.593+0000 7f8056800640 1 -- 172.21.15.72:0/3164814978 >> 172.21.15.72:0/3164814978 conn(0x7f8050077f80 msgr2=0x7f805007a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:46:14.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:14.593+0000 7f8056800640 1 -- 172.21.15.72:0/3164814978 shutdown_connections 2024-04-26T23:46:14.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:14.593+0000 7f8056800640 1 -- 172.21.15.72:0/3164814978 wait complete. 2024-04-26T23:46:14.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:14.594+0000 7f8056800640 1 Processor -- start 2024-04-26T23:46:14.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:14.595+0000 7f8056800640 1 -- start start 2024-04-26T23:46:14.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:14.595+0000 7f8056800640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8050108fc0 0x7f805019f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:14.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:14.595+0000 7f8056800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f805010a1b0 0x7f805019f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:14.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:14.595+0000 7f8056800640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f805010ab70 0x7f80501a6700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:14.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:14.595+0000 7f8056800640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f8050114b00 con 0x7f8050108fc0 2024-04-26T23:46:14.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:14.595+0000 7f8056800640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f8050114980 con 0x7f805010a1b0 2024-04-26T23:46:14.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:14.595+0000 7f8056800640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f8050114c80 con 0x7f805010ab70 2024-04-26T23:46:14.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:14.595+0000 7f804f400640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8050108fc0 0x7f805019f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:14.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:14.595+0000 7f804f400640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8050108fc0 0x7f805019f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:49192/0 (socket says 172.21.15.72:49192) 2024-04-26T23:46:14.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:14.595+0000 7f804f400640 1 -- 172.21.15.72:0/3973101896 learned_addr learned my addr 172.21.15.72:0/3973101896 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:46:14.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:14.595+0000 7f804fe00640 1 --2- 172.21.15.72:0/3973101896 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f805010ab70 0x7f80501a6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:14.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:14.595+0000 7f804ea00640 1 --2- 172.21.15.72:0/3973101896 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f805010a1b0 0x7f805019f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:14.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:14.596+0000 7f804f400640 1 -- 172.21.15.72:0/3973101896 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f805010ab70 msgr2=0x7f80501a6700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:14.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:14.596+0000 7f804f400640 1 --2- 172.21.15.72:0/3973101896 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f805010ab70 0x7f80501a6700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:14.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:14.596+0000 7f804f400640 1 -- 172.21.15.72:0/3973101896 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f805010a1b0 msgr2=0x7f805019f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:14.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:14.596+0000 7f804f400640 1 --2- 172.21.15.72:0/3973101896 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f805010a1b0 0x7f805019f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:14.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:14.596+0000 7f804f400640 1 -- 172.21.15.72:0/3973101896 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80501a6d70 con 0x7f8050108fc0 2024-04-26T23:46:14.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:14.596+0000 7f804fe00640 1 --2- 172.21.15.72:0/3973101896 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f805010ab70 0x7f80501a6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:46:14.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:14.596+0000 7f804f400640 1 --2- 172.21.15.72:0/3973101896 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8050108fc0 0x7f805019f150 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f804000e9c0 tx=0x7f804000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:46:14.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:14.596+0000 7f8047e00640 1 -- 172.21.15.72:0/3973101896 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f804000cd80 con 0x7f8050108fc0 2024-04-26T23:46:14.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:14.596+0000 7f8047e00640 1 -- 172.21.15.72:0/3973101896 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80400035f0 con 0x7f8050108fc0 2024-04-26T23:46:14.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:14.596+0000 7f8056800640 1 -- 172.21.15.72:0/3973101896 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80501a7060 con 0x7f8050108fc0 2024-04-26T23:46:14.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:14.596+0000 7f8047e00640 1 -- 172.21.15.72:0/3973101896 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8040010690 con 0x7f8050108fc0 2024-04-26T23:46:14.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:14.597+0000 7f8056800640 1 -- 172.21.15.72:0/3973101896 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80501a7530 con 0x7f8050108fc0 2024-04-26T23:46:14.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:14.598+0000 7f8056800640 1 -- 172.21.15.72:0/3973101896 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80501102e0 con 0x7f8050108fc0 2024-04-26T23:46:14.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:14.600+0000 7f8047e00640 1 -- 172.21.15.72:0/3973101896 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f8040004000 con 0x7f8050108fc0 2024-04-26T23:46:14.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:14.601+0000 7f8047e00640 1 --2- 172.21.15.72:0/3973101896 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f80340a3cf0 0x7f80340a61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:14.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:14.601+0000 7f8047e00640 1 -- 172.21.15.72:0/3973101896 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f80400d0ed0 con 0x7f8050108fc0 2024-04-26T23:46:14.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:14.601+0000 7f804ea00640 1 --2- 172.21.15.72:0/3973101896 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f80340a3cf0 0x7f80340a61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:14.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:14.602+0000 7f804ea00640 1 --2- 172.21.15.72:0/3973101896 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f80340a3cf0 0x7f80340a61b0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8038000c00 tx=0x7f80380314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:46:14.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:14.606+0000 7f8047e00640 1 -- 172.21.15.72:0/3973101896 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f804009d300 con 0x7f8050108fc0 2024-04-26T23:46:14.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:14.852+0000 7f8056800640 1 -- 172.21.15.72:0/3973101896 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7f805002dad0 con 0x7f8050108fc0 2024-04-26T23:46:15.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.626+0000 7f8047e00640 1 -- 172.21.15.72:0/3973101896 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v24) ==== 118+0+0 (secure 0 0 0) 0x7f804009cc00 con 0x7f8050108fc0 2024-04-26T23:46:15.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.629+0000 7f8056800640 1 -- 172.21.15.72:0/3973101896 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f80340a3cf0 msgr2=0x7f80340a61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:15.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.629+0000 7f8056800640 1 --2- 172.21.15.72:0/3973101896 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f80340a3cf0 0x7f80340a61b0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8038000c00 tx=0x7f80380314a0 comp rx=0 tx=0).stop 2024-04-26T23:46:15.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.629+0000 7f8056800640 1 -- 172.21.15.72:0/3973101896 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8050108fc0 msgr2=0x7f805019f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:15.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.629+0000 7f8056800640 1 --2- 172.21.15.72:0/3973101896 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8050108fc0 0x7f805019f150 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f804000e9c0 tx=0x7f804000ee80 comp rx=0 tx=0).stop 2024-04-26T23:46:15.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.629+0000 7f8056800640 1 -- 172.21.15.72:0/3973101896 shutdown_connections 2024-04-26T23:46:15.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.629+0000 7f8056800640 1 --2- 172.21.15.72:0/3973101896 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f80340a3cf0 0x7f80340a61b0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:15.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.631+0000 7f8056800640 1 --2- 172.21.15.72:0/3973101896 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f805010a1b0 0x7f805019f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:15.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.631+0000 7f8056800640 1 --2- 172.21.15.72:0/3973101896 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f805010ab70 0x7f80501a6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:15.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.631+0000 7f8056800640 1 --2- 172.21.15.72:0/3973101896 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8050108fc0 0x7f805019f150 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:15.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.631+0000 7f8056800640 1 -- 172.21.15.72:0/3973101896 >> 172.21.15.72:0/3973101896 conn(0x7f8050077f80 msgr2=0x7f805010d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:46:15.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.631+0000 7f8056800640 1 -- 172.21.15.72:0/3973101896 shutdown_connections 2024-04-26T23:46:15.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.631+0000 7f8056800640 1 -- 172.21.15.72:0/3973101896 wait complete. 2024-04-26T23:46:15.657 DEBUG:teuthology.orchestra.run.smithi072:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-04-26T23:46:15.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.854+0000 7f46bfe00640 1 -- 172.21.15.72:0/91501324 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f46b8103b70 msgr2=0x7f46b8105fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:15.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.854+0000 7f46bfe00640 1 --2- 172.21.15.72:0/91501324 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f46b8103b70 0x7f46b8105fe0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f46a4009860 tx=0x7f46a402a340 comp rx=0 tx=0).stop 2024-04-26T23:46:15.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.854+0000 7f46bfe00640 1 -- 172.21.15.72:0/91501324 shutdown_connections 2024-04-26T23:46:15.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.854+0000 7f46bfe00640 1 --2- 172.21.15.72:0/91501324 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f46b8106520 0x7f46b81069a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:15.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.854+0000 7f46bfe00640 1 --2- 172.21.15.72:0/91501324 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f46b8103b70 0x7f46b8105fe0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:15.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.854+0000 7f46bfe00640 1 --2- 172.21.15.72:0/91501324 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46b8101200 0x7f46b8103630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:15.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.854+0000 7f46bfe00640 1 -- 172.21.15.72:0/91501324 >> 172.21.15.72:0/91501324 conn(0x7f46b80fce20 msgr2=0x7f46b80ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T23:46:15.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.855+0000 7f46bfe00640 1 -- 172.21.15.72:0/91501324 shutdown_connections 2024-04-26T23:46:15.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.855+0000 7f46bfe00640 1 -- 172.21.15.72:0/91501324 wait complete. 2024-04-26T23:46:15.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.855+0000 7f46bfe00640 1 Processor -- start 2024-04-26T23:46:15.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.856+0000 7f46bfe00640 1 -- start start 2024-04-26T23:46:15.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.856+0000 7f46bfe00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f46b8101200 0x7f46b819f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:15.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.856+0000 7f46bfe00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f46b8103b70 0x7f46b819f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:15.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.856+0000 7f46bfe00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46b8106520 0x7f46b81a66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:15.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.856+0000 7f46bfe00640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f46b8114ae0 con 0x7f46b8101200 2024-04-26T23:46:15.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.856+0000 7f46bfe00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f46b8114960 con 0x7f46b8106520 2024-04-26T23:46:15.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.856+0000 7f46bfe00640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f46b8114c60 con 0x7f46b8103b70 2024-04-26T23:46:15.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.856+0000 7f46bd400640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f46b8101200 0x7f46b819f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:15.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.856+0000 7f46bca00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f46b8103b70 0x7f46b819f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:15.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.856+0000 7f46bde00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46b8106520 0x7f46b81a66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:15.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.856+0000 7f46bca00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f46b8103b70 0x7f46b819f640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.72:43282/0 (socket says 172.21.15.72:43282) 2024-04-26T23:46:15.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.856+0000 7f46bca00640 1 -- 172.21.15.72:0/1244301295 learned_addr learned my addr 172.21.15.72:0/1244301295 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-26T23:46:15.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.857+0000 7f46bca00640 1 -- 172.21.15.72:0/1244301295 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46b8106520 msgr2=0x7f46b81a66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:15.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.857+0000 7f46bca00640 1 --2- 172.21.15.72:0/1244301295 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46b8106520 0x7f46b81a66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:15.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.857+0000 7f46bca00640 1 -- 172.21.15.72:0/1244301295 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f46b8101200 msgr2=0x7f46b819f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T23:46:15.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.857+0000 7f46bca00640 1 --2- 172.21.15.72:0/1244301295 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f46b8101200 0x7f46b819f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T23:46:15.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.857+0000 7f46bca00640 1 -- 172.21.15.72:0/1244301295 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46b81a6d20 con 0x7f46b8103b70 2024-04-26T23:46:15.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.857+0000 7f46bde00640 1 --2- 172.21.15.72:0/1244301295 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46b8106520 0x7f46b81a66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T23:46:15.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.857+0000 7f46bca00640 1 --2- 172.21.15.72:0/1244301295 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f46b8103b70 0x7f46b819f640 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f46a40057c0 tx=0x7f46a4032da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:46:15.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.857+0000 7f46b6000640 1 -- 172.21.15.72:0/1244301295 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46a4037040 con 0x7f46b8103b70 2024-04-26T23:46:15.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.857+0000 7f46b6000640 1 -- 172.21.15.72:0/1244301295 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46a4030bd0 con 0x7f46b8103b70 2024-04-26T23:46:15.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.857+0000 7f46bfe00640 1 -- 172.21.15.72:0/1244301295 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46b81a6fb0 con 0x7f46b8103b70 2024-04-26T23:46:15.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.857+0000 7f46b6000640 1 -- 172.21.15.72:0/1244301295 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46a4005280 con 0x7f46b8103b70 2024-04-26T23:46:15.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.858+0000 7f46bfe00640 1 -- 172.21.15.72:0/1244301295 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46b81a73b0 con 0x7f46b8103b70 2024-04-26T23:46:15.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.859+0000 7f46bfe00640 1 -- 172.21.15.72:0/1244301295 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46b80fe910 con 0x7f46b8103b70 2024-04-26T23:46:15.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.860+0000 7f46b6000640 1 -- 172.21.15.72:0/1244301295 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f46a4043020 con 0x7f46b8103b70 2024-04-26T23:46:15.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.862+0000 7f46b6000640 1 --2- 172.21.15.72:0/1244301295 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f46a00a3cf0 0x7f46a00a61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T23:46:15.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.862+0000 7f46b6000640 1 -- 172.21.15.72:0/1244301295 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f46a40efec0 con 0x7f46b8103b70 2024-04-26T23:46:15.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.862+0000 7f46bd400640 1 --2- 172.21.15.72:0/1244301295 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f46a00a3cf0 0x7f46a00a61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T23:46:15.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.863+0000 7f46bd400640 1 --2- 172.21.15.72:0/1244301295 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f46a00a3cf0 0x7f46a00a61b0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f46ac009bd0 tx=0x7f46ac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T23:46:15.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:15.866+0000 7f46b6000640 1 -- 172.21.15.72:0/1244301295 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f46a40bc400 con 0x7f46b8103b70 2024-04-26T23:46:15.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:15 smithi076 ceph-mon[32774]: pgmap v421: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-26T23:46:15.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:15 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/3973101896' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-26T23:46:16.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:15 smithi072 ceph-mon[28184]: pgmap v421: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-26T23:46:16.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:15 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/3973101896' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-26T23:46:16.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:15 smithi136 ceph-mon[33050]: pgmap v421: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-26T23:46:16.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:15 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/3973101896' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-26T23:46:16.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:46:16.101+0000 7f46bfe00640 1 -- 172.21.15.72:0/1244301295 --> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f46b8062ed0 con 0x7f46a00a3cf0 2024-04-26T23:46:16.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:16 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/3973101896' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-26T23:46:16.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:16 smithi076 ceph-mon[32774]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T23:46:17.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:16 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/3973101896' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-26T23:46:17.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:16 smithi072 ceph-mon[28184]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T23:46:17.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:16 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/3973101896' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-26T23:46:17.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:16 smithi136 ceph-mon[33050]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=g=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T23:46:17.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:17 smithi076 ceph-mon[32774]: from='client.25384 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:46:17.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:17 smithi076 ceph-mon[32774]: pgmap v422: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-26T23:46:18.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:17 smithi072 ceph-mon[28184]: from='client.25384 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:46:18.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:17 smithi072 ceph-mon[28184]: pgmap v422: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-26T23:46:18.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:17 smithi136 ceph-mon[33050]: from='client.25384 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T23:46:18.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:17 smithi136 ceph-mon[33050]: pgmap v422: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-26T23:46:18.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:18 smithi076 ceph-mon[32774]: pgmap v423: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-26T23:46:19.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:18 smithi072 ceph-mon[28184]: pgmap v423: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-26T23:46:19.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:18 smithi136 ceph-mon[33050]: pgmap v423: 129 pgs: 129 active+clean; 635 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-26T23:46:21.748 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:21 smithi072 ceph-mon[28184]: pgmap v424: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 852 B/s wr, 7 op/s 2024-04-26T23:46:21.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:21 smithi076 ceph-mon[32774]: pgmap v424: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 852 B/s wr, 7 op/s 2024-04-26T23:46:22.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:21 smithi136 ceph-mon[33050]: pgmap v424: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 852 B/s wr, 7 op/s 2024-04-26T23:46:23.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:23 smithi072 ceph-mon[28184]: pgmap v425: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 852 B/s wr, 5 op/s 2024-04-26T23:46:23.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:23 smithi076 ceph-mon[32774]: pgmap v425: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 852 B/s wr, 5 op/s 2024-04-26T23:46:24.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:23 smithi136 ceph-mon[33050]: pgmap v425: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 852 B/s wr, 5 op/s 2024-04-26T23:46:25.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:25 smithi076 ceph-mon[32774]: pgmap v426: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 852 B/s wr, 8 op/s 2024-04-26T23:46:26.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:25 smithi072 ceph-mon[28184]: pgmap v426: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 852 B/s wr, 8 op/s 2024-04-26T23:46:26.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:25 smithi136 ceph-mon[33050]: pgmap v426: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 852 B/s wr, 8 op/s 2024-04-26T23:46:27.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:27 smithi076 ceph-mon[32774]: pgmap v427: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 767 B/s wr, 5 op/s 2024-04-26T23:46:28.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:27 smithi072 ceph-mon[28184]: pgmap v427: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 767 B/s wr, 5 op/s 2024-04-26T23:46:28.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:27 smithi136 ceph-mon[33050]: pgmap v427: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 767 B/s wr, 5 op/s 2024-04-26T23:46:29.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:29 smithi076 ceph-mon[32774]: pgmap v428: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 682 B/s wr, 5 op/s 2024-04-26T23:46:30.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:29 smithi072 ceph-mon[28184]: pgmap v428: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 682 B/s wr, 5 op/s 2024-04-26T23:46:30.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:29 smithi136 ceph-mon[33050]: pgmap v428: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 682 B/s wr, 5 op/s 2024-04-26T23:46:31.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:31 smithi076 ceph-mon[32774]: pgmap v429: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 682 B/s wr, 7 op/s 2024-04-26T23:46:32.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:31 smithi072 ceph-mon[28184]: pgmap v429: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 682 B/s wr, 7 op/s 2024-04-26T23:46:32.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:31 smithi136 ceph-mon[33050]: pgmap v429: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 682 B/s wr, 7 op/s 2024-04-26T23:46:33.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:33 smithi076 ceph-mon[32774]: pgmap v430: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:46:34.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:33 smithi072 ceph-mon[28184]: pgmap v430: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:46:34.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:33 smithi136 ceph-mon[33050]: pgmap v430: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:46:35.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:35 smithi076 ceph-mon[32774]: pgmap v431: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:46:36.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:35 smithi072 ceph-mon[28184]: pgmap v431: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:46:36.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:35 smithi136 ceph-mon[33050]: pgmap v431: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:46:37.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:37 smithi076 ceph-mon[32774]: pgmap v432: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:46:38.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:37 smithi072 ceph-mon[28184]: pgmap v432: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:46:38.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:37 smithi136 ceph-mon[33050]: pgmap v432: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:46:39.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:39 smithi076 ceph-mon[32774]: pgmap v433: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T23:46:40.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:39 smithi072 ceph-mon[28184]: pgmap v433: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T23:46:40.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:39 smithi136 ceph-mon[33050]: pgmap v433: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T23:46:41.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:41 smithi076 ceph-mon[32774]: pgmap v434: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:46:42.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:41 smithi072 ceph-mon[28184]: pgmap v434: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:46:42.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:41 smithi136 ceph-mon[33050]: pgmap v434: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:46:43.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:43 smithi076 ceph-mon[32774]: pgmap v435: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:46:44.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:43 smithi072 ceph-mon[28184]: pgmap v435: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:46:44.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:43 smithi136 ceph-mon[33050]: pgmap v435: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:46:45.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:45 smithi076 ceph-mon[32774]: pgmap v436: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:46:46.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:45 smithi072 ceph-mon[28184]: pgmap v436: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:46:46.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:45 smithi136 ceph-mon[33050]: pgmap v436: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:46:47.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:47 smithi076 ceph-mon[32774]: pgmap v437: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:46:48.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:47 smithi072 ceph-mon[28184]: pgmap v437: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:46:48.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:47 smithi136 ceph-mon[33050]: pgmap v437: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:46:49.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:49 smithi076 ceph-mon[32774]: pgmap v438: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T23:46:50.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:49 smithi072 ceph-mon[28184]: pgmap v438: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T23:46:50.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:49 smithi136 ceph-mon[33050]: pgmap v438: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T23:46:51.931 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:51 smithi076 ceph-mon[32774]: pgmap v439: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T23:46:52.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:51 smithi072 ceph-mon[28184]: pgmap v439: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T23:46:52.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:51 smithi136 ceph-mon[33050]: pgmap v439: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T23:46:53.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:53 smithi076 ceph-mon[32774]: pgmap v440: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:46:54.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:53 smithi072 ceph-mon[28184]: pgmap v440: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:46:54.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:53 smithi136 ceph-mon[33050]: pgmap v440: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:46:55.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:55 smithi076 ceph-mon[32774]: pgmap v441: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:46:56.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:55 smithi072 ceph-mon[28184]: pgmap v441: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:46:56.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:55 smithi136 ceph-mon[33050]: pgmap v441: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:46:56.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:56 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:46:57.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:56 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:46:57.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:56 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:46:57.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:57 smithi076 ceph-mon[32774]: pgmap v442: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:46:58.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:57 smithi072 ceph-mon[28184]: pgmap v442: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:46:58.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:57 smithi136 ceph-mon[33050]: pgmap v442: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:46:58.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:58 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:46:58.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:58 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:46:58.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:58 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:46:58.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:46:58 smithi076 ceph-mon[32774]: pgmap v443: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T23:46:59.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:58 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:46:59.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:58 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:46:59.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:58 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:46:59.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:46:58 smithi072 ceph-mon[28184]: pgmap v443: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T23:46:59.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:58 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:46:59.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:58 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:46:59.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:58 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:46:59.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:46:58 smithi136 ceph-mon[33050]: pgmap v443: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T23:47:01.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:47:01 smithi076 ceph-mon[32774]: pgmap v444: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:47:02.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:47:01 smithi072 ceph-mon[28184]: pgmap v444: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:47:02.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:47:01 smithi136 ceph-mon[33050]: pgmap v444: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:47:03.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:47:03 smithi076 ceph-mon[32774]: pgmap v445: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:47:04.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:47:03 smithi072 ceph-mon[28184]: pgmap v445: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:47:04.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:47:03 smithi136 ceph-mon[33050]: pgmap v445: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:47:05.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:47:05 smithi076 ceph-mon[32774]: pgmap v446: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:47:06.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:47:05 smithi072 ceph-mon[28184]: pgmap v446: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:47:06.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:47:05 smithi136 ceph-mon[33050]: pgmap v446: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:47:07.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:47:07 smithi076 ceph-mon[32774]: pgmap v447: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T23:47:08.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:47:07 smithi072 ceph-mon[28184]: pgmap v447: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T23:47:08.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:47:07 smithi136 ceph-mon[33050]: pgmap v447: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T23:47:09.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:47:09 smithi076 ceph-mon[32774]: pgmap v448: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-04-26T23:47:10.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:47:09 smithi072 ceph-mon[28184]: pgmap v448: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-04-26T23:47:10.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:47:09 smithi136 ceph-mon[33050]: pgmap v448: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-04-26T23:47:11.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:47:11 smithi076 ceph-mon[32774]: pgmap v449: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-04-26T23:47:12.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:47:11 smithi072 ceph-mon[28184]: pgmap v449: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-04-26T23:47:12.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:47:11 smithi136 ceph-mon[33050]: pgmap v449: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-04-26T23:47:13.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:47:13 smithi076 ceph-mon[32774]: pgmap v450: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T23:47:14.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:47:13 smithi072 ceph-mon[28184]: pgmap v450: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T23:47:14.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:47:13 smithi136 ceph-mon[33050]: pgmap v450: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T23:47:15.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:47:15 smithi076 ceph-mon[32774]: pgmap v451: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-04-26T23:47:16.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:47:15 smithi072 ceph-mon[28184]: pgmap v451: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-04-26T23:47:16.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:47:15 smithi136 ceph-mon[33050]: pgmap v451: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-04-26T23:47:17.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:47:17 smithi076 ceph-mon[32774]: pgmap v452: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T23:47:18.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:47:17 smithi072 ceph-mon[28184]: pgmap v452: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T23:47:18.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:47:17 smithi136 ceph-mon[33050]: pgmap v452: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T23:47:19.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:47:19 smithi076 ceph-mon[32774]: pgmap v453: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T23:47:20.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:47:19 smithi072 ceph-mon[28184]: pgmap v453: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T23:47:20.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:47:19 smithi136 ceph-mon[33050]: pgmap v453: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T23:47:21.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:47:21 smithi076 ceph-mon[32774]: pgmap v454: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-26T23:47:22.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:47:21 smithi072 ceph-mon[28184]: pgmap v454: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-26T23:47:22.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:47:21 smithi136 ceph-mon[33050]: pgmap v454: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-26T23:47:23.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:47:23 smithi076 ceph-mon[32774]: pgmap v455: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T23:47:24.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:47:23 smithi072 ceph-mon[28184]: pgmap v455: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T23:47:24.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:47:23 smithi136 ceph-mon[33050]: pgmap v455: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T23:47:25.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:47:25 smithi076 ceph-mon[32774]: pgmap v456: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-26T23:47:26.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:47:25 smithi072 ceph-mon[28184]: pgmap v456: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-26T23:47:26.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:47:25 smithi136 ceph-mon[33050]: pgmap v456: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-26T23:47:27.930 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:47:27 smithi076 ceph-mon[32774]: pgmap v457: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T23:47:28.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:47:27 smithi072 ceph-mon[28184]: pgmap v457: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T23:47:28.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:47:27 smithi136 ceph-mon[33050]: pgmap v457: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T23:47:29.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:47:29 smithi076 ceph-mon[32774]: pgmap v458: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T23:47:30.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:47:29 smithi072 ceph-mon[28184]: pgmap v458: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T23:47:30.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:47:29 smithi136 ceph-mon[33050]: pgmap v458: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T23:47:31.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:47:31 smithi076 ceph-mon[32774]: pgmap v459: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:47:32.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:47:31 smithi072 ceph-mon[28184]: pgmap v459: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:47:32.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:47:31 smithi136 ceph-mon[33050]: pgmap v459: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:47:33.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:47:32 smithi072 ceph-mon[28184]: pgmap v460: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:47:33.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:47:32 smithi136 ceph-mon[33050]: pgmap v460: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:47:33.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:47:32 smithi076 ceph-mon[32774]: pgmap v460: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:47:35.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:47:35 smithi076 ceph-mon[32774]: pgmap v461: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:47:36.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:47:35 smithi072 ceph-mon[28184]: pgmap v461: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:47:36.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:47:35 smithi136 ceph-mon[33050]: pgmap v461: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:47:37.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:47:37 smithi076 ceph-mon[32774]: pgmap v462: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:47:38.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:47:37 smithi072 ceph-mon[28184]: pgmap v462: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:47:38.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:47:37 smithi136 ceph-mon[33050]: pgmap v462: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:47:39.930 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:47:39 smithi076 ceph-mon[32774]: pgmap v463: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T23:47:40.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:47:39 smithi072 ceph-mon[28184]: pgmap v463: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T23:47:40.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:47:39 smithi136 ceph-mon[33050]: pgmap v463: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T23:47:41.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:47:41 smithi076 ceph-mon[32774]: pgmap v464: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:47:42.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:47:41 smithi072 ceph-mon[28184]: pgmap v464: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:47:42.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:47:41 smithi136 ceph-mon[33050]: pgmap v464: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:47:43.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:47:43 smithi076 ceph-mon[32774]: pgmap v465: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:47:44.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:47:43 smithi072 ceph-mon[28184]: pgmap v465: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:47:44.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:47:43 smithi136 ceph-mon[33050]: pgmap v465: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:47:45.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:47:45 smithi076 ceph-mon[32774]: pgmap v466: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:47:46.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:47:45 smithi072 ceph-mon[28184]: pgmap v466: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:47:46.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:47:45 smithi136 ceph-mon[33050]: pgmap v466: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:47:47.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:47:47 smithi076 ceph-mon[32774]: pgmap v467: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:47:48.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:47:47 smithi072 ceph-mon[28184]: pgmap v467: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:47:48.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:47:47 smithi136 ceph-mon[33050]: pgmap v467: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:47:49.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:47:49 smithi076 ceph-mon[32774]: pgmap v468: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T23:47:50.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:47:49 smithi072 ceph-mon[28184]: pgmap v468: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T23:47:50.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:47:49 smithi136 ceph-mon[33050]: pgmap v468: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T23:47:51.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:47:51 smithi076 ceph-mon[32774]: pgmap v469: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:47:52.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:47:51 smithi072 ceph-mon[28184]: pgmap v469: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:47:52.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:47:51 smithi136 ceph-mon[33050]: pgmap v469: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:47:53.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:47:53 smithi076 ceph-mon[32774]: pgmap v470: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:47:54.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:47:53 smithi072 ceph-mon[28184]: pgmap v470: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:47:54.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:47:53 smithi136 ceph-mon[33050]: pgmap v470: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:47:55.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:47:55 smithi076 ceph-mon[32774]: pgmap v471: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:47:56.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:47:55 smithi072 ceph-mon[28184]: pgmap v471: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:47:56.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:47:55 smithi136 ceph-mon[33050]: pgmap v471: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:47:57.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:47:57 smithi076 ceph-mon[32774]: pgmap v472: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:47:58.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:47:57 smithi072 ceph-mon[28184]: pgmap v472: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:47:58.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:47:57 smithi136 ceph-mon[33050]: pgmap v472: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:47:58.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:47:58 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:47:59.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:47:58 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:47:59.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:47:58 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:47:59.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:47:59 smithi076 ceph-mon[32774]: pgmap v473: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T23:47:59.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:47:59 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:47:59.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:47:59 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:47:59.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:47:59 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:48:00.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:47:59 smithi072 ceph-mon[28184]: pgmap v473: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T23:48:00.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:47:59 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:48:00.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:47:59 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:48:00.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:47:59 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:48:00.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:47:59 smithi136 ceph-mon[33050]: pgmap v473: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T23:48:00.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:47:59 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:48:00.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:47:59 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:48:00.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:47:59 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:48:01.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:48:00 smithi072 ceph-mon[28184]: pgmap v474: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:48:01.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:48:00 smithi136 ceph-mon[33050]: pgmap v474: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:48:01.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:48:00 smithi076 ceph-mon[32774]: pgmap v474: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:48:03.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:48:03 smithi076 ceph-mon[32774]: pgmap v475: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:48:04.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:48:03 smithi072 ceph-mon[28184]: pgmap v475: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:48:04.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:48:03 smithi136 ceph-mon[33050]: pgmap v475: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:48:05.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:48:05 smithi076 ceph-mon[32774]: pgmap v476: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:48:06.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:48:05 smithi072 ceph-mon[28184]: pgmap v476: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:48:06.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:48:05 smithi136 ceph-mon[33050]: pgmap v476: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:48:07.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:48:07 smithi076 ceph-mon[32774]: pgmap v477: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T23:48:08.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:48:07 smithi072 ceph-mon[28184]: pgmap v477: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T23:48:08.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:48:07 smithi136 ceph-mon[33050]: pgmap v477: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T23:48:09.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:48:09 smithi076 ceph-mon[32774]: pgmap v478: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-26T23:48:10.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:48:09 smithi072 ceph-mon[28184]: pgmap v478: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-26T23:48:10.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:48:09 smithi136 ceph-mon[33050]: pgmap v478: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-26T23:48:11.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:48:11 smithi076 ceph-mon[32774]: pgmap v479: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T23:48:12.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:48:11 smithi072 ceph-mon[28184]: pgmap v479: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T23:48:12.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:48:11 smithi136 ceph-mon[33050]: pgmap v479: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T23:48:13.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:48:13 smithi076 ceph-mon[32774]: pgmap v480: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T23:48:14.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:48:13 smithi072 ceph-mon[28184]: pgmap v480: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T23:48:14.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:48:13 smithi136 ceph-mon[33050]: pgmap v480: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T23:48:15.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:48:15 smithi076 ceph-mon[32774]: pgmap v481: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:48:16.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:48:15 smithi072 ceph-mon[28184]: pgmap v481: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:48:16.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:48:15 smithi136 ceph-mon[33050]: pgmap v481: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:48:17.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:48:17 smithi076 ceph-mon[32774]: pgmap v482: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T23:48:18.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:48:17 smithi072 ceph-mon[28184]: pgmap v482: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T23:48:18.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:48:17 smithi136 ceph-mon[33050]: pgmap v482: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T23:48:19.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:48:19 smithi076 ceph-mon[32774]: pgmap v483: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-26T23:48:20.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:48:19 smithi072 ceph-mon[28184]: pgmap v483: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-26T23:48:20.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:48:19 smithi136 ceph-mon[33050]: pgmap v483: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-26T23:48:21.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:48:21 smithi076 ceph-mon[32774]: pgmap v484: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T23:48:22.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:48:21 smithi072 ceph-mon[28184]: pgmap v484: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T23:48:22.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:48:21 smithi136 ceph-mon[33050]: pgmap v484: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T23:48:23.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:48:23 smithi076 ceph-mon[32774]: pgmap v485: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T23:48:24.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:48:23 smithi072 ceph-mon[28184]: pgmap v485: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T23:48:24.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:48:23 smithi136 ceph-mon[33050]: pgmap v485: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T23:48:25.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:48:25 smithi076 ceph-mon[32774]: pgmap v486: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-26T23:48:26.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:48:25 smithi072 ceph-mon[28184]: pgmap v486: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-26T23:48:26.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:48:25 smithi136 ceph-mon[33050]: pgmap v486: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-26T23:48:26.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:48:26 smithi076 ceph-mon[32774]: pgmap v487: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T23:48:27.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:48:26 smithi072 ceph-mon[28184]: pgmap v487: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T23:48:27.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:48:26 smithi136 ceph-mon[33050]: pgmap v487: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T23:48:29.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:48:29 smithi076 ceph-mon[32774]: pgmap v488: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T23:48:30.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:48:29 smithi072 ceph-mon[28184]: pgmap v488: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T23:48:30.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:48:29 smithi136 ceph-mon[33050]: pgmap v488: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T23:48:31.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:48:31 smithi076 ceph-mon[32774]: pgmap v489: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:48:32.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:48:31 smithi072 ceph-mon[28184]: pgmap v489: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:48:32.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:48:31 smithi136 ceph-mon[33050]: pgmap v489: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:48:33.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:48:33 smithi076 ceph-mon[32774]: pgmap v490: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:48:34.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:48:33 smithi072 ceph-mon[28184]: pgmap v490: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:48:34.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:48:33 smithi136 ceph-mon[33050]: pgmap v490: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:48:35.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:48:35 smithi076 ceph-mon[32774]: pgmap v491: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:48:36.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:48:35 smithi072 ceph-mon[28184]: pgmap v491: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:48:36.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:48:35 smithi136 ceph-mon[33050]: pgmap v491: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:48:37.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:48:37 smithi076 ceph-mon[32774]: pgmap v492: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:48:38.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:48:37 smithi072 ceph-mon[28184]: pgmap v492: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:48:38.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:48:37 smithi136 ceph-mon[33050]: pgmap v492: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:48:39.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:48:39 smithi076 ceph-mon[32774]: pgmap v493: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T23:48:40.052 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:48:39 smithi072 ceph-mon[28184]: pgmap v493: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T23:48:40.052 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:48:39 smithi136 ceph-mon[33050]: pgmap v493: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T23:48:41.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:48:41 smithi076 ceph-mon[32774]: pgmap v494: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:48:42.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:48:41 smithi072 ceph-mon[28184]: pgmap v494: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:48:42.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:48:41 smithi136 ceph-mon[33050]: pgmap v494: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:48:43.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:48:43 smithi076 ceph-mon[32774]: pgmap v495: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:48:44.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:48:43 smithi072 ceph-mon[28184]: pgmap v495: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:48:44.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:48:43 smithi136 ceph-mon[33050]: pgmap v495: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:48:45.930 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:48:45 smithi076 ceph-mon[32774]: pgmap v496: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:48:46.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:48:45 smithi072 ceph-mon[28184]: pgmap v496: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:48:46.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:48:45 smithi136 ceph-mon[33050]: pgmap v496: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:48:47.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:48:47 smithi076 ceph-mon[32774]: pgmap v497: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:48:48.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:48:47 smithi072 ceph-mon[28184]: pgmap v497: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:48:48.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:48:47 smithi136 ceph-mon[33050]: pgmap v497: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:48:49.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:48:48 smithi072 ceph-mon[28184]: pgmap v498: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T23:48:49.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:48:48 smithi136 ceph-mon[33050]: pgmap v498: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T23:48:49.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:48:48 smithi076 ceph-mon[32774]: pgmap v498: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T23:48:51.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:48:51 smithi076 ceph-mon[32774]: pgmap v499: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:48:52.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:48:51 smithi072 ceph-mon[28184]: pgmap v499: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:48:52.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:48:51 smithi136 ceph-mon[33050]: pgmap v499: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:48:53.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:48:53 smithi072 ceph-mon[28184]: pgmap v500: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:48:53.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:48:53 smithi076 ceph-mon[32774]: pgmap v500: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:48:54.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:48:53 smithi136 ceph-mon[33050]: pgmap v500: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:48:55.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:48:55 smithi076 ceph-mon[32774]: pgmap v501: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:48:56.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:48:55 smithi072 ceph-mon[28184]: pgmap v501: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:48:56.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:48:55 smithi136 ceph-mon[33050]: pgmap v501: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:48:57.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:48:57 smithi076 ceph-mon[32774]: pgmap v502: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:48:58.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:48:57 smithi072 ceph-mon[28184]: pgmap v502: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:48:58.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:48:57 smithi136 ceph-mon[33050]: pgmap v502: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:48:59.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:48:59 smithi076 ceph-mon[32774]: pgmap v503: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:48:59.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:48:59 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:49:00.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:48:59 smithi072 ceph-mon[28184]: pgmap v503: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:49:00.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:48:59 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:49:00.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:48:59 smithi136 ceph-mon[33050]: pgmap v503: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:49:00.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:48:59 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:49:01.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:49:01 smithi076 ceph-mon[32774]: pgmap v504: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:49:01.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:49:01 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:49:01.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:49:01 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:49:01.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:49:01 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:49:02.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:49:01 smithi072 ceph-mon[28184]: pgmap v504: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:49:02.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:49:01 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:49:02.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:49:01 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:49:02.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:49:01 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:49:02.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:49:01 smithi136 ceph-mon[33050]: pgmap v504: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:49:02.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:49:01 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:49:02.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:49:01 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:49:02.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:49:01 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:49:03.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:49:03 smithi076 ceph-mon[32774]: pgmap v505: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:49:04.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:49:03 smithi072 ceph-mon[28184]: pgmap v505: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:49:04.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:49:03 smithi136 ceph-mon[33050]: pgmap v505: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:49:05.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:49:05 smithi076 ceph-mon[32774]: pgmap v506: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:49:06.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:49:05 smithi072 ceph-mon[28184]: pgmap v506: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:49:06.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:49:05 smithi136 ceph-mon[33050]: pgmap v506: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:49:07.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:49:07 smithi076 ceph-mon[32774]: pgmap v507: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:49:08.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:49:07 smithi072 ceph-mon[28184]: pgmap v507: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:49:08.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:49:07 smithi136 ceph-mon[33050]: pgmap v507: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:49:09.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:49:08 smithi072 ceph-mon[28184]: pgmap v508: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:49:09.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:49:08 smithi136 ceph-mon[33050]: pgmap v508: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:49:09.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:49:08 smithi076 ceph-mon[32774]: pgmap v508: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:49:11.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:49:11 smithi076 ceph-mon[32774]: pgmap v509: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:49:12.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:49:11 smithi072 ceph-mon[28184]: pgmap v509: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:49:12.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:49:11 smithi136 ceph-mon[33050]: pgmap v509: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:49:13.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:49:13 smithi076 ceph-mon[32774]: pgmap v510: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:49:14.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:49:13 smithi072 ceph-mon[28184]: pgmap v510: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:49:14.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:49:13 smithi136 ceph-mon[33050]: pgmap v510: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:49:15.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:49:15.884+0000 7f46b4c00640 1 -- 172.21.15.72:0/1244301295 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f46940034c0 con 0x7f46b8103b70 2024-04-26T23:49:15.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:49:15.885+0000 7f46b6000640 1 -- 172.21.15.72:0/1244301295 <== mon.1 v2:172.21.15.136:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f46940034c0 con 0x7f46b8103b70 2024-04-26T23:49:15.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:49:15 smithi076 ceph-mon[32774]: pgmap v511: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:49:16.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:49:15 smithi072 ceph-mon[28184]: pgmap v511: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:49:16.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:49:15 smithi136 ceph-mon[33050]: pgmap v511: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:49:17.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:49:17 smithi076 ceph-mon[32774]: pgmap v512: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:49:18.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:49:17 smithi072 ceph-mon[28184]: pgmap v512: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:49:18.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:49:17 smithi136 ceph-mon[33050]: pgmap v512: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:49:19.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:49:19 smithi076 ceph-mon[32774]: pgmap v513: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:49:20.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:49:19 smithi072 ceph-mon[28184]: pgmap v513: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:49:20.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:49:19 smithi136 ceph-mon[33050]: pgmap v513: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:49:21.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:49:21 smithi076 ceph-mon[32774]: pgmap v514: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:49:22.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:49:21 smithi072 ceph-mon[28184]: pgmap v514: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:49:22.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:49:21 smithi136 ceph-mon[33050]: pgmap v514: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:49:23.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:49:23 smithi076 ceph-mon[32774]: pgmap v515: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:49:24.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:49:23 smithi072 ceph-mon[28184]: pgmap v515: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:49:24.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:49:23 smithi136 ceph-mon[33050]: pgmap v515: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:49:25.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:49:24 smithi072 ceph-mon[28184]: pgmap v516: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:49:25.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:49:24 smithi136 ceph-mon[33050]: pgmap v516: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:49:25.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:49:24 smithi076 ceph-mon[32774]: pgmap v516: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:49:27.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:49:27 smithi076 ceph-mon[32774]: pgmap v517: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:49:28.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:49:27 smithi072 ceph-mon[28184]: pgmap v517: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:49:28.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:49:27 smithi136 ceph-mon[33050]: pgmap v517: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:49:29.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:49:29 smithi076 ceph-mon[32774]: pgmap v518: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:49:30.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:49:29 smithi072 ceph-mon[28184]: pgmap v518: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:49:30.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:49:29 smithi136 ceph-mon[33050]: pgmap v518: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:49:31.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:49:31 smithi076 ceph-mon[32774]: pgmap v519: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:49:32.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:49:31 smithi072 ceph-mon[28184]: pgmap v519: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:49:32.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:49:31 smithi136 ceph-mon[33050]: pgmap v519: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:49:33.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:49:33 smithi076 ceph-mon[32774]: pgmap v520: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:49:34.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:49:33 smithi072 ceph-mon[28184]: pgmap v520: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:49:34.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:49:33 smithi136 ceph-mon[33050]: pgmap v520: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:49:35.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:49:35 smithi076 ceph-mon[32774]: pgmap v521: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:49:36.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:49:35 smithi072 ceph-mon[28184]: pgmap v521: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:49:36.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:49:35 smithi136 ceph-mon[33050]: pgmap v521: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:49:38.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:49:37 smithi072 ceph-mon[28184]: pgmap v522: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:49:38.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:49:37 smithi136 ceph-mon[33050]: pgmap v522: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:49:38.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:49:37 smithi076 ceph-mon[32774]: pgmap v522: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:49:39.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:49:38 smithi072 ceph-mon[28184]: pgmap v523: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:49:39.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:49:38 smithi136 ceph-mon[33050]: pgmap v523: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:49:39.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:49:38 smithi076 ceph-mon[32774]: pgmap v523: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:49:41.930 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:49:41 smithi076 ceph-mon[32774]: pgmap v524: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:49:42.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:49:41 smithi072 ceph-mon[28184]: pgmap v524: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:49:42.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:49:41 smithi136 ceph-mon[33050]: pgmap v524: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:49:43.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:49:43 smithi076 ceph-mon[32774]: pgmap v525: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:49:44.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:49:43 smithi072 ceph-mon[28184]: pgmap v525: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:49:44.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:49:43 smithi136 ceph-mon[33050]: pgmap v525: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:49:45.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:49:45 smithi076 ceph-mon[32774]: pgmap v526: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:49:46.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:49:45 smithi072 ceph-mon[28184]: pgmap v526: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:49:46.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:49:45 smithi136 ceph-mon[33050]: pgmap v526: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:49:47.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:49:47 smithi076 ceph-mon[32774]: pgmap v527: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:49:48.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:49:47 smithi072 ceph-mon[28184]: pgmap v527: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:49:48.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:49:47 smithi136 ceph-mon[33050]: pgmap v527: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:49:49.679 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:49:49 smithi076 ceph-mon[32774]: pgmap v528: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:49:49.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:49:49 smithi072 ceph-mon[28184]: pgmap v528: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:49:49.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:49:49 smithi136 ceph-mon[33050]: pgmap v528: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:49:51.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:49:51 smithi076 ceph-mon[32774]: pgmap v529: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:49:52.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:49:51 smithi072 ceph-mon[28184]: pgmap v529: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:49:52.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:49:51 smithi136 ceph-mon[33050]: pgmap v529: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:49:53.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:49:53 smithi076 ceph-mon[32774]: pgmap v530: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:49:54.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:49:53 smithi072 ceph-mon[28184]: pgmap v530: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:49:54.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:49:53 smithi136 ceph-mon[33050]: pgmap v530: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:49:56.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:49:55 smithi072 ceph-mon[28184]: pgmap v531: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:49:56.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:49:55 smithi136 ceph-mon[33050]: pgmap v531: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:49:56.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:49:55 smithi076 ceph-mon[32774]: pgmap v531: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:49:58.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:49:57 smithi072 ceph-mon[28184]: pgmap v532: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:49:58.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:49:57 smithi136 ceph-mon[33050]: pgmap v532: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:49:58.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:49:57 smithi076 ceph-mon[32774]: pgmap v532: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:50:00.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:49:59 smithi072 ceph-mon[28184]: pgmap v533: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:50:00.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:49:59 smithi136 ceph-mon[33050]: pgmap v533: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:50:00.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:49:59 smithi076 ceph-mon[32774]: pgmap v533: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:50:01.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:50:00 smithi072 ceph-mon[28184]: overall HEALTH_OK 2024-04-26T23:50:01.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:50:00 smithi072 ceph-mon[28184]: pgmap v534: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:50:01.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:50:00 smithi136 ceph-mon[33050]: overall HEALTH_OK 2024-04-26T23:50:01.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:50:00 smithi136 ceph-mon[33050]: pgmap v534: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:50:01.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:50:00 smithi076 ceph-mon[32774]: overall HEALTH_OK 2024-04-26T23:50:01.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:50:00 smithi076 ceph-mon[32774]: pgmap v534: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:50:02.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:50:01 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:50:02.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:50:01 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:50:02.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:50:01 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:50:03.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:50:02 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:50:03.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:50:02 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:50:03.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:50:02 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:50:03.045 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:50:02 smithi072 ceph-mon[28184]: pgmap v535: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:50:03.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:50:02 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:50:03.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:50:02 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:50:03.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:50:02 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:50:03.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:50:02 smithi136 ceph-mon[33050]: pgmap v535: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:50:03.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:50:02 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:50:03.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:50:02 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:50:03.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:50:02 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:50:03.180 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:50:02 smithi076 ceph-mon[32774]: pgmap v535: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:50:05.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:50:05 smithi076 ceph-mon[32774]: pgmap v536: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:50:06.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:50:05 smithi072 ceph-mon[28184]: pgmap v536: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:50:06.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:50:05 smithi136 ceph-mon[33050]: pgmap v536: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:50:08.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:50:07 smithi072 ceph-mon[28184]: pgmap v537: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:50:08.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:50:07 smithi136 ceph-mon[33050]: pgmap v537: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:50:08.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:50:07 smithi076 ceph-mon[32774]: pgmap v537: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:50:10.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:50:09 smithi072 ceph-mon[28184]: pgmap v538: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:50:10.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:50:09 smithi136 ceph-mon[33050]: pgmap v538: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:50:10.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:50:09 smithi076 ceph-mon[32774]: pgmap v538: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:50:12.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:50:11 smithi072 ceph-mon[28184]: pgmap v539: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:50:12.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:50:11 smithi136 ceph-mon[33050]: pgmap v539: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:50:12.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:50:11 smithi076 ceph-mon[32774]: pgmap v539: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:50:13.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:50:12 smithi072 ceph-mon[28184]: pgmap v540: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:50:13.078 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:50:12 smithi136 ceph-mon[33050]: pgmap v540: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:50:13.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:50:12 smithi076 ceph-mon[32774]: pgmap v540: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:50:16.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:50:15 smithi072 ceph-mon[28184]: pgmap v541: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:50:16.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:50:15 smithi136 ceph-mon[33050]: pgmap v541: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:50:16.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:50:15 smithi076 ceph-mon[32774]: pgmap v541: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:50:18.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:50:17 smithi072 ceph-mon[28184]: pgmap v542: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:50:18.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:50:17 smithi136 ceph-mon[33050]: pgmap v542: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:50:18.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:50:17 smithi076 ceph-mon[32774]: pgmap v542: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:50:20.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:50:19 smithi072 ceph-mon[28184]: pgmap v543: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:50:20.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:50:19 smithi136 ceph-mon[33050]: pgmap v543: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:50:20.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:50:19 smithi076 ceph-mon[32774]: pgmap v543: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:50:21.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:50:20 smithi072 ceph-mon[28184]: pgmap v544: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:50:21.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:50:20 smithi136 ceph-mon[33050]: pgmap v544: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:50:21.080 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:50:20 smithi076 ceph-mon[32774]: pgmap v544: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:50:24.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:50:23 smithi072 ceph-mon[28184]: pgmap v545: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:50:24.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:50:23 smithi136 ceph-mon[33050]: pgmap v545: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:50:24.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:50:23 smithi076 ceph-mon[32774]: pgmap v545: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:50:26.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:50:25 smithi072 ceph-mon[28184]: pgmap v546: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:50:26.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:50:25 smithi136 ceph-mon[33050]: pgmap v546: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:50:26.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:50:25 smithi076 ceph-mon[32774]: pgmap v546: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:50:27.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:50:26 smithi072 ceph-mon[28184]: pgmap v547: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:50:27.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:50:26 smithi136 ceph-mon[33050]: pgmap v547: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:50:27.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:50:26 smithi076 ceph-mon[32774]: pgmap v547: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:50:30.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:50:29 smithi072 ceph-mon[28184]: pgmap v548: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:50:30.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:50:29 smithi136 ceph-mon[33050]: pgmap v548: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:50:30.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:50:29 smithi076 ceph-mon[32774]: pgmap v548: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:50:32.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:50:31 smithi072 ceph-mon[28184]: pgmap v549: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:50:32.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:50:31 smithi136 ceph-mon[33050]: pgmap v549: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:50:32.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:50:31 smithi076 ceph-mon[32774]: pgmap v549: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:50:33.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:50:32 smithi072 ceph-mon[28184]: pgmap v550: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:50:33.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:50:32 smithi136 ceph-mon[33050]: pgmap v550: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:50:33.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:50:32 smithi076 ceph-mon[32774]: pgmap v550: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:50:36.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:50:35 smithi072 ceph-mon[28184]: pgmap v551: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:50:36.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:50:35 smithi136 ceph-mon[33050]: pgmap v551: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:50:36.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:50:35 smithi076 ceph-mon[32774]: pgmap v551: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:50:38.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:50:37 smithi072 ceph-mon[28184]: pgmap v552: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:50:38.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:50:37 smithi136 ceph-mon[33050]: pgmap v552: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:50:38.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:50:37 smithi076 ceph-mon[32774]: pgmap v552: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:50:39.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:50:38 smithi072 ceph-mon[28184]: pgmap v553: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:50:39.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:50:38 smithi136 ceph-mon[33050]: pgmap v553: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:50:39.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:50:38 smithi076 ceph-mon[32774]: pgmap v553: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:50:42.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:50:41 smithi072 ceph-mon[28184]: pgmap v554: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:50:42.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:50:41 smithi136 ceph-mon[33050]: pgmap v554: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:50:42.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:50:41 smithi076 ceph-mon[32774]: pgmap v554: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:50:43.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:50:42 smithi072 ceph-mon[28184]: pgmap v555: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:50:43.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:50:42 smithi136 ceph-mon[33050]: pgmap v555: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:50:43.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:50:42 smithi076 ceph-mon[32774]: pgmap v555: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:50:46.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:50:45 smithi072 ceph-mon[28184]: pgmap v556: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:50:46.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:50:45 smithi136 ceph-mon[33050]: pgmap v556: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:50:46.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:50:45 smithi076 ceph-mon[32774]: pgmap v556: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:50:47.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:50:46 smithi072 ceph-mon[28184]: pgmap v557: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:50:47.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:50:46 smithi136 ceph-mon[33050]: pgmap v557: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:50:47.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:50:46 smithi076 ceph-mon[32774]: pgmap v557: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:50:50.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:50:49 smithi072 ceph-mon[28184]: pgmap v558: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:50:50.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:50:49 smithi136 ceph-mon[33050]: pgmap v558: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:50:50.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:50:49 smithi076 ceph-mon[32774]: pgmap v558: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:50:51.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:50:50 smithi072 ceph-mon[28184]: pgmap v559: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:50:51.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:50:50 smithi136 ceph-mon[33050]: pgmap v559: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:50:51.080 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:50:50 smithi076 ceph-mon[32774]: pgmap v559: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:50:54.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:50:53 smithi072 ceph-mon[28184]: pgmap v560: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:50:54.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:50:53 smithi136 ceph-mon[33050]: pgmap v560: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:50:54.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:50:53 smithi076 ceph-mon[32774]: pgmap v560: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:50:55.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:50:54 smithi072 ceph-mon[28184]: pgmap v561: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:50:55.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:50:54 smithi136 ceph-mon[33050]: pgmap v561: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:50:55.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:50:54 smithi076 ceph-mon[32774]: pgmap v561: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:50:57.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:50:56 smithi072 ceph-mon[28184]: pgmap v562: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:50:57.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:50:56 smithi136 ceph-mon[33050]: pgmap v562: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:50:57.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:50:56 smithi076 ceph-mon[32774]: pgmap v562: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:50:59.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:50:58 smithi072 ceph-mon[28184]: pgmap v563: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:50:59.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:50:58 smithi136 ceph-mon[33050]: pgmap v563: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:50:59.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:50:58 smithi076 ceph-mon[32774]: pgmap v563: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:51:01.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:51:00 smithi072 ceph-mon[28184]: pgmap v564: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:51:01.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:51:00 smithi136 ceph-mon[33050]: pgmap v564: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:51:01.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:51:00 smithi076 ceph-mon[32774]: pgmap v564: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:51:03.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:51:02 smithi072 ceph-mon[28184]: pgmap v565: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:51:03.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:51:02 smithi136 ceph-mon[33050]: pgmap v565: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:51:03.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:51:02 smithi076 ceph-mon[32774]: pgmap v565: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:51:04.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:51:03 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:51:04.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:51:03 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:51:04.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:51:03 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:51:05.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:51:04 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:51:05.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:51:04 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:51:05.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:51:04 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:51:05.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:51:04 smithi072 ceph-mon[28184]: pgmap v566: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:51:05.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:51:04 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:51:05.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:51:04 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:51:05.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:51:04 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:51:05.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:51:04 smithi136 ceph-mon[33050]: pgmap v566: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:51:05.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:51:04 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:51:05.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:51:04 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:51:05.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:51:04 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:51:05.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:51:04 smithi076 ceph-mon[32774]: pgmap v566: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:51:07.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:51:06 smithi072 ceph-mon[28184]: pgmap v567: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:51:07.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:51:06 smithi136 ceph-mon[33050]: pgmap v567: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:51:07.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:51:06 smithi076 ceph-mon[32774]: pgmap v567: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:51:09.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:51:08 smithi072 ceph-mon[28184]: pgmap v568: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:51:09.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:51:08 smithi136 ceph-mon[33050]: pgmap v568: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:51:09.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:51:08 smithi076 ceph-mon[32774]: pgmap v568: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:51:11.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:51:10 smithi072 ceph-mon[28184]: pgmap v569: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:51:11.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:51:10 smithi136 ceph-mon[33050]: pgmap v569: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:51:11.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:51:10 smithi076 ceph-mon[32774]: pgmap v569: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:51:13.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:51:12 smithi072 ceph-mon[28184]: pgmap v570: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:51:13.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:51:12 smithi136 ceph-mon[33050]: pgmap v570: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:51:13.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:51:12 smithi076 ceph-mon[32774]: pgmap v570: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:51:15.045 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:51:14 smithi072 ceph-mon[28184]: pgmap v571: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:51:15.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:51:14 smithi136 ceph-mon[33050]: pgmap v571: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:51:15.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:51:14 smithi076 ceph-mon[32774]: pgmap v571: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:51:17.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:51:16 smithi072 ceph-mon[28184]: pgmap v572: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:51:17.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:51:16 smithi136 ceph-mon[33050]: pgmap v572: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:51:17.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:51:16 smithi076 ceph-mon[32774]: pgmap v572: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:51:19.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:51:18 smithi072 ceph-mon[28184]: pgmap v573: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:51:19.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:51:18 smithi136 ceph-mon[33050]: pgmap v573: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:51:19.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:51:18 smithi076 ceph-mon[32774]: pgmap v573: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:51:21.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:51:20 smithi072 ceph-mon[28184]: pgmap v574: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:51:21.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:51:20 smithi136 ceph-mon[33050]: pgmap v574: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:51:21.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:51:20 smithi076 ceph-mon[32774]: pgmap v574: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:51:23.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:51:22 smithi072 ceph-mon[28184]: pgmap v575: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:51:23.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:51:22 smithi136 ceph-mon[33050]: pgmap v575: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:51:23.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:51:22 smithi076 ceph-mon[32774]: pgmap v575: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:51:25.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:51:24 smithi072 ceph-mon[28184]: pgmap v576: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:51:25.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:51:24 smithi136 ceph-mon[33050]: pgmap v576: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:51:25.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:51:24 smithi076 ceph-mon[32774]: pgmap v576: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:51:27.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:51:26 smithi072 ceph-mon[28184]: pgmap v577: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:51:27.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:51:26 smithi136 ceph-mon[33050]: pgmap v577: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:51:27.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:51:26 smithi076 ceph-mon[32774]: pgmap v577: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:51:29.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:51:28 smithi072 ceph-mon[28184]: pgmap v578: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:51:29.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:51:28 smithi136 ceph-mon[33050]: pgmap v578: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:51:29.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:51:28 smithi076 ceph-mon[32774]: pgmap v578: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:51:31.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:51:30 smithi072 ceph-mon[28184]: pgmap v579: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:51:31.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:51:30 smithi136 ceph-mon[33050]: pgmap v579: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:51:31.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:51:30 smithi076 ceph-mon[32774]: pgmap v579: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:51:33.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:51:32 smithi072 ceph-mon[28184]: pgmap v580: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:51:33.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:51:32 smithi136 ceph-mon[33050]: pgmap v580: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:51:33.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:51:32 smithi076 ceph-mon[32774]: pgmap v580: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:51:35.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:51:34 smithi072 ceph-mon[28184]: pgmap v581: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:51:35.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:51:34 smithi136 ceph-mon[33050]: pgmap v581: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:51:35.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:51:34 smithi076 ceph-mon[32774]: pgmap v581: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:51:37.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:51:36 smithi072 ceph-mon[28184]: pgmap v582: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:51:37.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:51:36 smithi136 ceph-mon[33050]: pgmap v582: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:51:37.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:51:36 smithi076 ceph-mon[32774]: pgmap v582: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:51:39.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:51:38 smithi072 ceph-mon[28184]: pgmap v583: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:51:39.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:51:38 smithi136 ceph-mon[33050]: pgmap v583: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:51:39.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:51:38 smithi076 ceph-mon[32774]: pgmap v583: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:51:41.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:51:40 smithi072 ceph-mon[28184]: pgmap v584: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:51:41.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:51:40 smithi136 ceph-mon[33050]: pgmap v584: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:51:41.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:51:40 smithi076 ceph-mon[32774]: pgmap v584: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:51:43.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:51:42 smithi072 ceph-mon[28184]: pgmap v585: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:51:43.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:51:42 smithi136 ceph-mon[33050]: pgmap v585: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:51:43.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:51:42 smithi076 ceph-mon[32774]: pgmap v585: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:51:45.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:51:44 smithi072 ceph-mon[28184]: pgmap v586: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:51:45.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:51:44 smithi136 ceph-mon[33050]: pgmap v586: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:51:45.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:51:44 smithi076 ceph-mon[32774]: pgmap v586: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:51:47.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:51:46 smithi072 ceph-mon[28184]: pgmap v587: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:51:47.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:51:46 smithi136 ceph-mon[33050]: pgmap v587: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:51:47.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:51:46 smithi076 ceph-mon[32774]: pgmap v587: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:51:49.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:51:48 smithi072 ceph-mon[28184]: pgmap v588: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:51:49.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:51:48 smithi136 ceph-mon[33050]: pgmap v588: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:51:49.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:51:48 smithi076 ceph-mon[32774]: pgmap v588: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:51:51.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:51:50 smithi072 ceph-mon[28184]: pgmap v589: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:51:51.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:51:50 smithi136 ceph-mon[33050]: pgmap v589: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:51:51.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:51:50 smithi076 ceph-mon[32774]: pgmap v589: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:51:53.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:51:52 smithi072 ceph-mon[28184]: pgmap v590: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:51:53.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:51:52 smithi136 ceph-mon[33050]: pgmap v590: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:51:53.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:51:52 smithi076 ceph-mon[32774]: pgmap v590: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:51:55.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:51:54 smithi072 ceph-mon[28184]: pgmap v591: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:51:55.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:51:54 smithi136 ceph-mon[33050]: pgmap v591: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:51:55.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:51:54 smithi076 ceph-mon[32774]: pgmap v591: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:51:57.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:51:56 smithi072 ceph-mon[28184]: pgmap v592: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:51:57.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:51:56 smithi136 ceph-mon[33050]: pgmap v592: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:51:57.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:51:56 smithi076 ceph-mon[32774]: pgmap v592: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:51:59.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:51:58 smithi072 ceph-mon[28184]: pgmap v593: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:51:59.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:51:58 smithi136 ceph-mon[33050]: pgmap v593: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:51:59.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:51:58 smithi076 ceph-mon[32774]: pgmap v593: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:52:01.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:52:00 smithi072 ceph-mon[28184]: pgmap v594: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:52:01.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:52:00 smithi136 ceph-mon[33050]: pgmap v594: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:52:01.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:52:00 smithi076 ceph-mon[32774]: pgmap v594: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:52:03.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:52:02 smithi072 ceph-mon[28184]: pgmap v595: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:52:03.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:52:02 smithi136 ceph-mon[33050]: pgmap v595: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:52:03.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:52:02 smithi076 ceph-mon[32774]: pgmap v595: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:52:04.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:52:04 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:52:04.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:52:04 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:52:04.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:52:04 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:52:05.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:52:05 smithi072 ceph-mon[28184]: pgmap v596: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:52:05.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:52:05 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:52:05.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:52:05 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:52:05.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:52:05 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:52:05.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:52:05 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:52:05.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:52:05 smithi136 ceph-mon[33050]: pgmap v596: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:52:05.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:52:05 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:52:05.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:52:05 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:52:05.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:52:05 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:52:05.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:52:05 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:52:05.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:52:05 smithi076 ceph-mon[32774]: pgmap v596: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:52:05.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:52:05 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:52:05.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:52:05 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:52:05.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:52:05 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:52:05.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:52:05 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:52:06.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:52:06 smithi072 ceph-mon[28184]: Adjusting osd_memory_target on smithi076 to 176.2M 2024-04-26T23:52:06.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:52:06 smithi072 ceph-mon[28184]: Unable to set osd_memory_target on smithi076 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-04-26T23:52:06.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:52:06 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:52:06.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:52:06 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:52:06.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:52:06 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:52:06.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:52:06 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:52:06.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:52:06 smithi072 ceph-mon[28184]: Adjusting osd_memory_target on smithi072 to 176.4M 2024-04-26T23:52:06.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:52:06 smithi072 ceph-mon[28184]: Unable to set osd_memory_target on smithi072 to 185003929: error parsing value: Value '185003929' is below minimum 939524096 2024-04-26T23:52:06.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:52:06 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:52:06.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:52:06 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:52:06.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:52:06 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:52:06.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:52:06 smithi136 ceph-mon[33050]: Adjusting osd_memory_target on smithi076 to 176.2M 2024-04-26T23:52:06.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:52:06 smithi136 ceph-mon[33050]: Unable to set osd_memory_target on smithi076 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-04-26T23:52:06.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:52:06 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:52:06.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:52:06 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:52:06.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:52:06 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:52:06.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:52:06 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:52:06.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:52:06 smithi136 ceph-mon[33050]: Adjusting osd_memory_target on smithi072 to 176.4M 2024-04-26T23:52:06.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:52:06 smithi136 ceph-mon[33050]: Unable to set osd_memory_target on smithi072 to 185003929: error parsing value: Value '185003929' is below minimum 939524096 2024-04-26T23:52:06.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:52:06 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:52:06.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:52:06 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:52:06.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:52:06 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:52:06.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:52:06 smithi076 ceph-mon[32774]: Adjusting osd_memory_target on smithi076 to 176.2M 2024-04-26T23:52:06.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:52:06 smithi076 ceph-mon[32774]: Unable to set osd_memory_target on smithi076 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-04-26T23:52:06.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:52:06 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:52:06.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:52:06 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:52:06.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:52:06 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:52:06.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:52:06 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:52:06.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:52:06 smithi076 ceph-mon[32774]: Adjusting osd_memory_target on smithi072 to 176.4M 2024-04-26T23:52:06.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:52:06 smithi076 ceph-mon[32774]: Unable to set osd_memory_target on smithi072 to 185003929: error parsing value: Value '185003929' is below minimum 939524096 2024-04-26T23:52:06.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:52:06 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:52:06.930 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:52:06 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:52:06.930 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:52:06 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:52:07.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:52:07 smithi072 ceph-mon[28184]: pgmap v597: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:52:07.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:52:07 smithi136 ceph-mon[33050]: pgmap v597: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:52:07.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:52:07 smithi076 ceph-mon[32774]: pgmap v597: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:52:09.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:52:08 smithi072 ceph-mon[28184]: pgmap v598: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:52:09.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:52:08 smithi136 ceph-mon[33050]: pgmap v598: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:52:09.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:52:08 smithi076 ceph-mon[32774]: pgmap v598: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:52:11.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:52:10 smithi072 ceph-mon[28184]: pgmap v599: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:52:11.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:52:10 smithi136 ceph-mon[33050]: pgmap v599: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:52:11.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:52:10 smithi076 ceph-mon[32774]: pgmap v599: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:52:13.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:52:12 smithi072 ceph-mon[28184]: pgmap v600: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:52:13.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:52:12 smithi136 ceph-mon[33050]: pgmap v600: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:52:13.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:52:12 smithi076 ceph-mon[32774]: pgmap v600: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:52:15.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:52:14 smithi072 ceph-mon[28184]: pgmap v601: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:52:15.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:52:14 smithi136 ceph-mon[33050]: pgmap v601: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:52:15.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:52:14 smithi076 ceph-mon[32774]: pgmap v601: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:52:15.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:52:15.914+0000 7f46b4c00640 1 -- 172.21.15.72:0/1244301295 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f4694005850 con 0x7f46b8103b70 2024-04-26T23:52:15.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:52:15.915+0000 7f46b6000640 1 -- 172.21.15.72:0/1244301295 <== mon.1 v2:172.21.15.136:3300/0 8 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f4694005850 con 0x7f46b8103b70 2024-04-26T23:52:17.045 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:52:16 smithi072 ceph-mon[28184]: pgmap v602: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:52:17.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:52:16 smithi136 ceph-mon[33050]: pgmap v602: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:52:17.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:52:16 smithi076 ceph-mon[32774]: pgmap v602: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:52:19.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:52:18 smithi072 ceph-mon[28184]: pgmap v603: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:52:19.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:52:18 smithi136 ceph-mon[33050]: pgmap v603: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:52:19.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:52:18 smithi076 ceph-mon[32774]: pgmap v603: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:52:21.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:52:20 smithi072 ceph-mon[28184]: pgmap v604: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:52:21.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:52:20 smithi136 ceph-mon[33050]: pgmap v604: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:52:21.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:52:20 smithi076 ceph-mon[32774]: pgmap v604: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:52:23.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:52:22 smithi072 ceph-mon[28184]: pgmap v605: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:52:23.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:52:22 smithi136 ceph-mon[33050]: pgmap v605: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:52:23.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:52:22 smithi076 ceph-mon[32774]: pgmap v605: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:52:25.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:52:24 smithi072 ceph-mon[28184]: pgmap v606: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:52:25.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:52:24 smithi136 ceph-mon[33050]: pgmap v606: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:52:25.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:52:24 smithi076 ceph-mon[32774]: pgmap v606: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:52:27.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:52:26 smithi072 ceph-mon[28184]: pgmap v607: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:52:27.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:52:26 smithi136 ceph-mon[33050]: pgmap v607: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:52:27.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:52:26 smithi076 ceph-mon[32774]: pgmap v607: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:52:29.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:52:28 smithi072 ceph-mon[28184]: pgmap v608: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:52:29.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:52:28 smithi136 ceph-mon[33050]: pgmap v608: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:52:29.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:52:28 smithi076 ceph-mon[32774]: pgmap v608: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:52:31.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:52:30 smithi072 ceph-mon[28184]: pgmap v609: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:52:31.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:52:30 smithi136 ceph-mon[33050]: pgmap v609: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:52:31.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:52:30 smithi076 ceph-mon[32774]: pgmap v609: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:52:33.044 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:52:32 smithi072 ceph-mon[28184]: pgmap v610: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:52:33.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:52:32 smithi136 ceph-mon[33050]: pgmap v610: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:52:33.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:52:32 smithi076 ceph-mon[32774]: pgmap v610: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:52:35.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:52:34 smithi072 ceph-mon[28184]: pgmap v611: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:52:35.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:52:34 smithi136 ceph-mon[33050]: pgmap v611: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:52:35.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:52:34 smithi076 ceph-mon[32774]: pgmap v611: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:52:37.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:52:36 smithi072 ceph-mon[28184]: pgmap v612: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:52:37.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:52:36 smithi136 ceph-mon[33050]: pgmap v612: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:52:37.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:52:36 smithi076 ceph-mon[32774]: pgmap v612: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:52:39.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:52:38 smithi072 ceph-mon[28184]: pgmap v613: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:52:39.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:52:38 smithi136 ceph-mon[33050]: pgmap v613: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:52:39.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:52:38 smithi076 ceph-mon[32774]: pgmap v613: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:52:41.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:52:40 smithi072 ceph-mon[28184]: pgmap v614: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:52:41.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:52:40 smithi136 ceph-mon[33050]: pgmap v614: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:52:41.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:52:40 smithi076 ceph-mon[32774]: pgmap v614: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:52:43.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:52:42 smithi072 ceph-mon[28184]: pgmap v615: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:52:43.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:52:42 smithi136 ceph-mon[33050]: pgmap v615: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:52:43.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:52:42 smithi076 ceph-mon[32774]: pgmap v615: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:52:45.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:52:44 smithi072 ceph-mon[28184]: pgmap v616: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:52:45.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:52:44 smithi136 ceph-mon[33050]: pgmap v616: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:52:45.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:52:44 smithi076 ceph-mon[32774]: pgmap v616: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:52:47.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:52:46 smithi072 ceph-mon[28184]: pgmap v617: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:52:47.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:52:46 smithi136 ceph-mon[33050]: pgmap v617: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:52:47.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:52:46 smithi076 ceph-mon[32774]: pgmap v617: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:52:49.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:52:48 smithi072 ceph-mon[28184]: pgmap v618: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:52:49.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:52:48 smithi136 ceph-mon[33050]: pgmap v618: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:52:49.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:52:48 smithi076 ceph-mon[32774]: pgmap v618: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:52:51.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:52:50 smithi072 ceph-mon[28184]: pgmap v619: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:52:51.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:52:50 smithi136 ceph-mon[33050]: pgmap v619: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:52:51.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:52:50 smithi076 ceph-mon[32774]: pgmap v619: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:52:53.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:52:52 smithi072 ceph-mon[28184]: pgmap v620: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:52:53.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:52:52 smithi136 ceph-mon[33050]: pgmap v620: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:52:53.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:52:52 smithi076 ceph-mon[32774]: pgmap v620: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:52:55.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:52:54 smithi072 ceph-mon[28184]: pgmap v621: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:52:55.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:52:54 smithi136 ceph-mon[33050]: pgmap v621: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:52:55.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:52:54 smithi076 ceph-mon[32774]: pgmap v621: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:52:57.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:52:56 smithi072 ceph-mon[28184]: pgmap v622: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:52:57.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:52:56 smithi136 ceph-mon[33050]: pgmap v622: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:52:57.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:52:56 smithi076 ceph-mon[32774]: pgmap v622: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:52:59.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:52:58 smithi072 ceph-mon[28184]: pgmap v623: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:52:59.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:52:58 smithi136 ceph-mon[33050]: pgmap v623: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:52:59.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:52:58 smithi076 ceph-mon[32774]: pgmap v623: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:53:01.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:53:00 smithi072 ceph-mon[28184]: pgmap v624: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:53:01.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:53:00 smithi136 ceph-mon[33050]: pgmap v624: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:53:01.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:53:00 smithi076 ceph-mon[32774]: pgmap v624: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:53:03.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:53:02 smithi072 ceph-mon[28184]: pgmap v625: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:53:03.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:53:02 smithi136 ceph-mon[33050]: pgmap v625: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:53:03.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:53:02 smithi076 ceph-mon[32774]: pgmap v625: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:53:05.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:53:04 smithi072 ceph-mon[28184]: pgmap v626: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:53:05.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:53:04 smithi136 ceph-mon[33050]: pgmap v626: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:53:05.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:53:04 smithi076 ceph-mon[32774]: pgmap v626: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:53:06.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:53:05 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:53:06.299 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:53:05 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:53:06.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:53:05 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:53:07.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:53:06 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:53:07.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:53:06 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:53:07.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:53:06 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:53:07.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:53:06 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:53:07.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:53:06 smithi072 ceph-mon[28184]: Adjusting osd_memory_target on smithi136 to 176.2M 2024-04-26T23:53:07.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:53:06 smithi072 ceph-mon[28184]: Unable to set osd_memory_target on smithi136 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-04-26T23:53:07.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:53:06 smithi072 ceph-mon[28184]: pgmap v627: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:53:07.299 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:53:06 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:53:07.299 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:53:06 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:53:07.299 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:53:06 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:53:07.299 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:53:06 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:53:07.299 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:53:06 smithi136 ceph-mon[33050]: Adjusting osd_memory_target on smithi136 to 176.2M 2024-04-26T23:53:07.299 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:53:06 smithi136 ceph-mon[33050]: Unable to set osd_memory_target on smithi136 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-04-26T23:53:07.299 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:53:06 smithi136 ceph-mon[33050]: pgmap v627: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:53:07.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:53:06 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:53:07.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:53:06 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:53:07.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:53:06 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:53:07.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:53:06 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T23:53:07.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:53:06 smithi076 ceph-mon[32774]: Adjusting osd_memory_target on smithi136 to 176.2M 2024-04-26T23:53:07.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:53:06 smithi076 ceph-mon[32774]: Unable to set osd_memory_target on smithi136 to 184769536: error parsing value: Value '184769536' is below minimum 939524096 2024-04-26T23:53:07.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:53:06 smithi076 ceph-mon[32774]: pgmap v627: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:53:08.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:53:07 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:53:08.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:53:07 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:53:08.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:53:07 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:53:08.299 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:53:07 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:53:08.299 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:53:07 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:53:08.299 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:53:07 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:53:08.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:53:07 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:53:08.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:53:07 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:53:08.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:53:07 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:53:09.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:53:08 smithi072 ceph-mon[28184]: pgmap v628: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T23:53:09.299 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:53:08 smithi136 ceph-mon[33050]: pgmap v628: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T23:53:09.428 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:53:08 smithi076 ceph-mon[32774]: pgmap v628: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T23:53:11.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:53:10 smithi072 ceph-mon[28184]: pgmap v629: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:53:11.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:53:10 smithi136 ceph-mon[33050]: pgmap v629: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:53:11.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:53:10 smithi076 ceph-mon[32774]: pgmap v629: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:53:13.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:53:12 smithi072 ceph-mon[28184]: pgmap v630: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:53:13.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:53:12 smithi136 ceph-mon[33050]: pgmap v630: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:53:13.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:53:12 smithi076 ceph-mon[32774]: pgmap v630: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:53:15.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:53:14 smithi072 ceph-mon[28184]: pgmap v631: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:53:15.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:53:14 smithi136 ceph-mon[33050]: pgmap v631: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:53:15.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:53:14 smithi076 ceph-mon[32774]: pgmap v631: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:53:17.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:53:16 smithi072 ceph-mon[28184]: pgmap v632: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:53:17.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:53:16 smithi136 ceph-mon[33050]: pgmap v632: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:53:17.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:53:16 smithi076 ceph-mon[32774]: pgmap v632: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:53:19.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:53:18 smithi072 ceph-mon[28184]: pgmap v633: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T23:53:19.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:53:18 smithi136 ceph-mon[33050]: pgmap v633: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T23:53:19.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:53:18 smithi076 ceph-mon[32774]: pgmap v633: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-26T23:53:21.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:53:20 smithi136 ceph-mon[33050]: pgmap v634: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:53:21.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:53:20 smithi076 ceph-mon[32774]: pgmap v634: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:53:21.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:53:20 smithi072 ceph-mon[28184]: pgmap v634: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:53:23.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:53:22 smithi072 ceph-mon[28184]: pgmap v635: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:53:23.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:53:22 smithi136 ceph-mon[33050]: pgmap v635: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:53:23.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:53:22 smithi076 ceph-mon[32774]: pgmap v635: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:53:25.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:53:24 smithi072 ceph-mon[28184]: pgmap v636: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:53:25.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:53:24 smithi136 ceph-mon[33050]: pgmap v636: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:53:25.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:53:24 smithi076 ceph-mon[32774]: pgmap v636: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:53:27.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:53:26 smithi072 ceph-mon[28184]: pgmap v637: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:53:27.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:53:26 smithi136 ceph-mon[33050]: pgmap v637: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:53:27.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:53:26 smithi076 ceph-mon[32774]: pgmap v637: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:53:29.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:53:28 smithi072 ceph-mon[28184]: pgmap v638: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:53:29.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:53:28 smithi136 ceph-mon[33050]: pgmap v638: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:53:29.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:53:28 smithi076 ceph-mon[32774]: pgmap v638: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:53:31.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:53:30 smithi072 ceph-mon[28184]: pgmap v639: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:53:31.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:53:30 smithi136 ceph-mon[33050]: pgmap v639: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:53:31.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:53:30 smithi076 ceph-mon[32774]: pgmap v639: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:53:33.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:53:32 smithi072 ceph-mon[28184]: pgmap v640: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:53:33.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:53:32 smithi136 ceph-mon[33050]: pgmap v640: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:53:33.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:53:32 smithi076 ceph-mon[32774]: pgmap v640: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:53:35.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:53:34 smithi072 ceph-mon[28184]: pgmap v641: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:53:35.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:53:34 smithi136 ceph-mon[33050]: pgmap v641: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:53:35.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:53:34 smithi076 ceph-mon[32774]: pgmap v641: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:53:37.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:53:36 smithi072 ceph-mon[28184]: pgmap v642: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:53:37.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:53:36 smithi136 ceph-mon[33050]: pgmap v642: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:53:37.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:53:36 smithi076 ceph-mon[32774]: pgmap v642: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:53:39.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:53:38 smithi072 ceph-mon[28184]: pgmap v643: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:53:39.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:53:38 smithi136 ceph-mon[33050]: pgmap v643: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:53:39.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:53:38 smithi076 ceph-mon[32774]: pgmap v643: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:53:41.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:53:40 smithi136 ceph-mon[33050]: pgmap v644: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:53:41.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:53:40 smithi076 ceph-mon[32774]: pgmap v644: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:53:41.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:53:40 smithi072 ceph-mon[28184]: pgmap v644: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:53:43.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:53:42 smithi136 ceph-mon[33050]: pgmap v645: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:53:43.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:53:42 smithi076 ceph-mon[32774]: pgmap v645: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:53:43.292 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:53:42 smithi072 ceph-mon[28184]: pgmap v645: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:53:45.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:53:44 smithi136 ceph-mon[33050]: pgmap v646: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:53:45.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:53:44 smithi076 ceph-mon[32774]: pgmap v646: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:53:45.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:53:44 smithi072 ceph-mon[28184]: pgmap v646: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:53:47.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:53:46 smithi136 ceph-mon[33050]: pgmap v647: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:53:47.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:53:46 smithi076 ceph-mon[32774]: pgmap v647: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:53:47.293 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:53:46 smithi072 ceph-mon[28184]: pgmap v647: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:53:50.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:53:50 smithi076 ceph-mon[32774]: pgmap v648: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:53:50.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:53:50 smithi072 ceph-mon[28184]: pgmap v648: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:53:50.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:53:50 smithi136 ceph-mon[33050]: pgmap v648: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:53:52.678 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:53:52 smithi076 ceph-mon[32774]: pgmap v649: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:53:52.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:53:52 smithi072 ceph-mon[28184]: pgmap v649: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:53:52.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:53:52 smithi136 ceph-mon[33050]: pgmap v649: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:53:54.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:53:54 smithi076 ceph-mon[32774]: pgmap v650: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:53:54.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:53:54 smithi072 ceph-mon[28184]: pgmap v650: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:53:54.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:53:54 smithi136 ceph-mon[33050]: pgmap v650: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:53:56.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:53:56 smithi072 ceph-mon[28184]: pgmap v651: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:53:57.561 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:53:56 smithi076 ceph-mon[32774]: pgmap v651: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:53:57.561 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:53:56 smithi136 ceph-mon[33050]: pgmap v651: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:53:58.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:53:58 smithi072 ceph-mon[28184]: pgmap v652: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:53:58.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:53:58 smithi136 ceph-mon[33050]: pgmap v652: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:53:58.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:53:58 smithi076 ceph-mon[32774]: pgmap v652: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:54:00.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:54:00 smithi072 ceph-mon[28184]: pgmap v653: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:54:00.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:54:00 smithi136 ceph-mon[33050]: pgmap v653: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:54:00.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:54:00 smithi076 ceph-mon[32774]: pgmap v653: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:54:02.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:54:02 smithi072 ceph-mon[28184]: pgmap v654: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:54:02.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:54:02 smithi136 ceph-mon[33050]: pgmap v654: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:54:02.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:54:02 smithi076 ceph-mon[32774]: pgmap v654: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:54:04.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:54:04 smithi072 ceph-mon[28184]: pgmap v655: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:54:04.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:54:04 smithi136 ceph-mon[33050]: pgmap v655: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:54:04.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:54:04 smithi076 ceph-mon[32774]: pgmap v655: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:54:06.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:54:06 smithi072 ceph-mon[28184]: pgmap v656: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:54:06.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:54:06 smithi136 ceph-mon[33050]: pgmap v656: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:54:06.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:54:06 smithi076 ceph-mon[32774]: pgmap v656: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:54:07.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:54:07 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:54:07.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:54:07 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:54:07.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:54:07 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:54:08.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:54:08 smithi072 ceph-mon[28184]: pgmap v657: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:54:08.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:54:08 smithi136 ceph-mon[33050]: pgmap v657: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:54:08.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:54:08 smithi076 ceph-mon[32774]: pgmap v657: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:54:09.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:54:09 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:54:09.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:54:09 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:54:09.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:54:09 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:54:09.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:54:09 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:54:09.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:54:09 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:54:09.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:54:09 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:54:09.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:54:09 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:54:09.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:54:09 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:54:09.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:54:09 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:54:10.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:54:10 smithi072 ceph-mon[28184]: pgmap v658: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:54:10.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:54:10 smithi136 ceph-mon[33050]: pgmap v658: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:54:10.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:54:10 smithi076 ceph-mon[32774]: pgmap v658: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:54:12.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:54:12 smithi072 ceph-mon[28184]: pgmap v659: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:54:12.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:54:12 smithi136 ceph-mon[33050]: pgmap v659: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:54:12.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:54:12 smithi076 ceph-mon[32774]: pgmap v659: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:54:14.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:54:14 smithi072 ceph-mon[28184]: pgmap v660: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:54:14.800 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:54:14 smithi136 ceph-mon[33050]: pgmap v660: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:54:14.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:54:14 smithi076 ceph-mon[32774]: pgmap v660: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:54:16.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:54:16 smithi072 ceph-mon[28184]: pgmap v661: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:54:16.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:54:16 smithi136 ceph-mon[33050]: pgmap v661: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:54:16.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:54:16 smithi076 ceph-mon[32774]: pgmap v661: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:54:18.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:54:18 smithi072 ceph-mon[28184]: pgmap v662: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:54:18.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:54:18 smithi136 ceph-mon[33050]: pgmap v662: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:54:18.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:54:18 smithi076 ceph-mon[32774]: pgmap v662: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:54:20.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:54:20 smithi072 ceph-mon[28184]: pgmap v663: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:54:20.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:54:20 smithi136 ceph-mon[33050]: pgmap v663: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:54:20.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:54:20 smithi076 ceph-mon[32774]: pgmap v663: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:54:22.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:54:22 smithi072 ceph-mon[28184]: pgmap v664: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:54:22.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:54:22 smithi136 ceph-mon[33050]: pgmap v664: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:54:22.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:54:22 smithi076 ceph-mon[32774]: pgmap v664: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:54:24.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:54:24 smithi072 ceph-mon[28184]: pgmap v665: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:54:24.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:54:24 smithi136 ceph-mon[33050]: pgmap v665: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:54:24.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:54:24 smithi076 ceph-mon[32774]: pgmap v665: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:54:26.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:54:26 smithi072 ceph-mon[28184]: pgmap v666: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:54:26.801 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:54:26 smithi136 ceph-mon[33050]: pgmap v666: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:54:26.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:54:26 smithi076 ceph-mon[32774]: pgmap v666: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:54:28.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:54:28 smithi072 ceph-mon[28184]: pgmap v667: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:54:28.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:54:28 smithi136 ceph-mon[33050]: pgmap v667: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:54:28.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:54:28 smithi076 ceph-mon[32774]: pgmap v667: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:54:30.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:54:30 smithi072 ceph-mon[28184]: pgmap v668: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:54:30.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:54:30 smithi136 ceph-mon[33050]: pgmap v668: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:54:30.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:54:30 smithi076 ceph-mon[32774]: pgmap v668: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:54:30.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:54:30.937+0000 7f46b4c00640 1 -- 172.21.15.72:0/1244301295 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f4694005b50 con 0x7f46b8103b70 2024-04-26T23:54:30.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:54:30.937+0000 7f46b6000640 1 -- 172.21.15.72:0/1244301295 <== mon.1 v2:172.21.15.136:3300/0 9 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f4694005b50 con 0x7f46b8103b70 2024-04-26T23:54:32.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:54:32 smithi072 ceph-mon[28184]: pgmap v669: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:54:32.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:54:32 smithi136 ceph-mon[33050]: pgmap v669: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:54:32.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:54:32 smithi076 ceph-mon[32774]: pgmap v669: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:54:34.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:54:34 smithi136 ceph-mon[33050]: pgmap v670: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:54:34.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:54:34 smithi076 ceph-mon[32774]: pgmap v670: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:54:35.042 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:54:34 smithi072 ceph-mon[28184]: pgmap v670: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:54:36.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:54:36 smithi136 ceph-mon[33050]: pgmap v671: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:54:36.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:54:36 smithi076 ceph-mon[32774]: pgmap v671: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:54:37.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:54:36 smithi072 ceph-mon[28184]: pgmap v671: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:54:38.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:54:38 smithi076 ceph-mon[32774]: pgmap v672: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:54:39.042 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:54:38 smithi072 ceph-mon[28184]: pgmap v672: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:54:39.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:54:38 smithi136 ceph-mon[33050]: pgmap v672: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:54:40.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:54:40 smithi076 ceph-mon[32774]: pgmap v673: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:54:41.042 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:54:40 smithi072 ceph-mon[28184]: pgmap v673: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:54:41.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:54:40 smithi136 ceph-mon[33050]: pgmap v673: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:54:42.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:54:42 smithi076 ceph-mon[32774]: pgmap v674: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:54:43.042 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:54:42 smithi072 ceph-mon[28184]: pgmap v674: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:54:43.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:54:42 smithi136 ceph-mon[33050]: pgmap v674: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:54:44.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:54:44 smithi076 ceph-mon[32774]: pgmap v675: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:54:45.042 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:54:44 smithi072 ceph-mon[28184]: pgmap v675: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:54:45.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:54:44 smithi136 ceph-mon[33050]: pgmap v675: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:54:46.851 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:54:46 smithi136 ceph-mon[33050]: pgmap v676: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:54:46.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:54:46 smithi076 ceph-mon[32774]: pgmap v676: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:54:47.042 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:54:46 smithi072 ceph-mon[28184]: pgmap v676: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:54:48.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:54:48 smithi076 ceph-mon[32774]: pgmap v677: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:54:49.042 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:54:48 smithi072 ceph-mon[28184]: pgmap v677: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:54:49.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:54:48 smithi136 ceph-mon[33050]: pgmap v677: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:54:50.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:54:50 smithi076 ceph-mon[32774]: pgmap v678: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:54:51.042 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:54:50 smithi072 ceph-mon[28184]: pgmap v678: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:54:51.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:54:50 smithi136 ceph-mon[33050]: pgmap v678: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:54:52.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:54:52 smithi076 ceph-mon[32774]: pgmap v679: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:54:53.042 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:54:52 smithi072 ceph-mon[28184]: pgmap v679: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:54:53.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:54:52 smithi136 ceph-mon[33050]: pgmap v679: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:54:54.845 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:54:54 smithi072 ceph-mon[28184]: pgmap v680: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:54:54.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:54:54 smithi076 ceph-mon[32774]: pgmap v680: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:54:55.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:54:54 smithi136 ceph-mon[33050]: pgmap v680: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:54:56.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:54:56 smithi076 ceph-mon[32774]: pgmap v681: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:54:57.365 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:54:56 smithi136 ceph-mon[33050]: pgmap v681: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:54:57.365 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:54:56 smithi072 ceph-mon[28184]: pgmap v681: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:54:58.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:54:58 smithi076 ceph-mon[32774]: pgmap v682: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:54:59.042 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:54:58 smithi072 ceph-mon[28184]: pgmap v682: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:54:59.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:54:58 smithi136 ceph-mon[33050]: pgmap v682: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:55:00.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:55:00 smithi076 ceph-mon[32774]: pgmap v683: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:55:01.042 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:55:00 smithi072 ceph-mon[28184]: pgmap v683: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:55:01.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:55:00 smithi136 ceph-mon[33050]: pgmap v683: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:55:02.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:55:02 smithi076 ceph-mon[32774]: pgmap v684: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:55:03.042 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:55:02 smithi072 ceph-mon[28184]: pgmap v684: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:55:03.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:55:02 smithi136 ceph-mon[33050]: pgmap v684: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:55:04.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:55:04 smithi076 ceph-mon[32774]: pgmap v685: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:55:05.042 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:55:04 smithi072 ceph-mon[28184]: pgmap v685: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:55:05.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:55:04 smithi136 ceph-mon[33050]: pgmap v685: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:55:06.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:55:06 smithi076 ceph-mon[32774]: pgmap v686: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:55:07.042 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:55:06 smithi072 ceph-mon[28184]: pgmap v686: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:55:07.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:55:06 smithi136 ceph-mon[33050]: pgmap v686: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:55:08.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:55:08 smithi076 ceph-mon[32774]: pgmap v687: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T23:55:09.042 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:55:08 smithi072 ceph-mon[28184]: pgmap v687: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T23:55:09.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:55:08 smithi136 ceph-mon[33050]: pgmap v687: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T23:55:09.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:55:09 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:55:10.042 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:55:09 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:55:10.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:55:09 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:55:10.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:55:10 smithi076 ceph-mon[32774]: pgmap v688: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-26T23:55:10.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:55:10 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:55:10.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:55:10 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:55:10.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:55:10 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:55:11.042 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:55:10 smithi072 ceph-mon[28184]: pgmap v688: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-26T23:55:11.042 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:55:10 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:55:11.042 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:55:10 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:55:11.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:55:10 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:55:11.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:55:10 smithi136 ceph-mon[33050]: pgmap v688: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-26T23:55:11.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:55:10 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:55:11.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:55:10 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:55:11.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:55:10 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:55:12.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:55:12 smithi076 ceph-mon[32774]: pgmap v689: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T23:55:13.042 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:55:12 smithi072 ceph-mon[28184]: pgmap v689: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T23:55:13.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:55:12 smithi136 ceph-mon[33050]: pgmap v689: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T23:55:14.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:55:14 smithi076 ceph-mon[32774]: pgmap v690: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T23:55:15.042 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:55:14 smithi072 ceph-mon[28184]: pgmap v690: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T23:55:15.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:55:14 smithi136 ceph-mon[33050]: pgmap v690: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T23:55:15.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:55:15.945+0000 7f46b4c00640 1 -- 172.21.15.72:0/1244301295 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f4694005eb0 con 0x7f46b8103b70 2024-04-26T23:55:15.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:55:15.946+0000 7f46b6000640 1 -- 172.21.15.72:0/1244301295 <== mon.1 v2:172.21.15.136:3300/0 10 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f4694005eb0 con 0x7f46b8103b70 2024-04-26T23:55:16.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:55:16 smithi076 ceph-mon[32774]: pgmap v691: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:55:17.042 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:55:16 smithi072 ceph-mon[28184]: pgmap v691: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:55:17.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:55:16 smithi136 ceph-mon[33050]: pgmap v691: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:55:18.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:55:18 smithi076 ceph-mon[32774]: pgmap v692: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:55:19.042 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:55:18 smithi072 ceph-mon[28184]: pgmap v692: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:55:19.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:55:18 smithi136 ceph-mon[33050]: pgmap v692: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:55:20.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:55:20 smithi076 ceph-mon[32774]: pgmap v693: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:55:21.042 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:55:20 smithi072 ceph-mon[28184]: pgmap v693: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:55:21.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:55:20 smithi136 ceph-mon[33050]: pgmap v693: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:55:22.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:55:22 smithi076 ceph-mon[32774]: pgmap v694: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:55:23.043 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:55:22 smithi072 ceph-mon[28184]: pgmap v694: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:55:23.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:55:22 smithi136 ceph-mon[33050]: pgmap v694: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:55:25.042 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:55:24 smithi072 ceph-mon[28184]: pgmap v695: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:55:25.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:55:24 smithi136 ceph-mon[33050]: pgmap v695: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:55:25.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:55:24 smithi076 ceph-mon[32774]: pgmap v695: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:55:27.042 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:55:26 smithi072 ceph-mon[28184]: pgmap v696: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:55:27.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:55:26 smithi136 ceph-mon[33050]: pgmap v696: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:55:27.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:55:26 smithi076 ceph-mon[32774]: pgmap v696: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:55:29.042 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:55:28 smithi072 ceph-mon[28184]: pgmap v697: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:55:29.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:55:28 smithi136 ceph-mon[33050]: pgmap v697: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:55:29.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:55:28 smithi076 ceph-mon[32774]: pgmap v697: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:55:31.042 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:55:30 smithi072 ceph-mon[28184]: pgmap v698: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:55:31.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:55:30 smithi136 ceph-mon[33050]: pgmap v698: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:55:31.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:55:30 smithi076 ceph-mon[32774]: pgmap v698: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:55:33.042 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:55:32 smithi072 ceph-mon[28184]: pgmap v699: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:55:33.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:55:32 smithi136 ceph-mon[33050]: pgmap v699: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:55:33.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:55:32 smithi076 ceph-mon[32774]: pgmap v699: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:55:35.042 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:55:34 smithi072 ceph-mon[28184]: pgmap v700: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:55:35.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:55:34 smithi136 ceph-mon[33050]: pgmap v700: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:55:35.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:55:34 smithi076 ceph-mon[32774]: pgmap v700: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:55:37.042 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:55:36 smithi072 ceph-mon[28184]: pgmap v701: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:55:37.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:55:36 smithi136 ceph-mon[33050]: pgmap v701: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:55:37.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:55:36 smithi076 ceph-mon[32774]: pgmap v701: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:55:39.042 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:55:38 smithi072 ceph-mon[28184]: pgmap v702: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:55:39.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:55:38 smithi136 ceph-mon[33050]: pgmap v702: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:55:39.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:55:38 smithi076 ceph-mon[32774]: pgmap v702: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:55:41.042 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:55:40 smithi072 ceph-mon[28184]: pgmap v703: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:55:41.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:55:40 smithi136 ceph-mon[33050]: pgmap v703: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:55:41.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:55:40 smithi076 ceph-mon[32774]: pgmap v703: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:55:43.042 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:55:42 smithi072 ceph-mon[28184]: pgmap v704: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:55:43.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:55:42 smithi136 ceph-mon[33050]: pgmap v704: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:55:43.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:55:42 smithi076 ceph-mon[32774]: pgmap v704: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:55:45.042 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:55:44 smithi072 ceph-mon[28184]: pgmap v705: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:55:45.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:55:44 smithi136 ceph-mon[33050]: pgmap v705: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:55:45.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:55:44 smithi076 ceph-mon[32774]: pgmap v705: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:55:47.042 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:55:46 smithi072 ceph-mon[28184]: pgmap v706: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:55:47.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:55:46 smithi136 ceph-mon[33050]: pgmap v706: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:55:47.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:55:46 smithi076 ceph-mon[32774]: pgmap v706: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:55:49.042 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:55:48 smithi072 ceph-mon[28184]: pgmap v707: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:55:49.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:55:48 smithi136 ceph-mon[33050]: pgmap v707: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:55:49.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:55:48 smithi076 ceph-mon[32774]: pgmap v707: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:55:51.042 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:55:50 smithi072 ceph-mon[28184]: pgmap v708: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:55:51.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:55:50 smithi136 ceph-mon[33050]: pgmap v708: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:55:51.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:55:50 smithi076 ceph-mon[32774]: pgmap v708: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:55:53.042 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:55:52 smithi072 ceph-mon[28184]: pgmap v709: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:55:53.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:55:52 smithi136 ceph-mon[33050]: pgmap v709: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:55:53.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:55:52 smithi076 ceph-mon[32774]: pgmap v709: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:55:55.042 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:55:54 smithi072 ceph-mon[28184]: pgmap v710: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:55:55.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:55:54 smithi136 ceph-mon[33050]: pgmap v710: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:55:55.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:55:54 smithi076 ceph-mon[32774]: pgmap v710: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:55:57.042 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:55:56 smithi072 ceph-mon[28184]: pgmap v711: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:55:57.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:55:56 smithi136 ceph-mon[33050]: pgmap v711: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:55:57.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:55:56 smithi076 ceph-mon[32774]: pgmap v711: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:55:58.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:55:57 smithi076 ceph-mon[32774]: pgmap v712: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:55:58.292 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:55:57 smithi072 ceph-mon[28184]: pgmap v712: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:55:58.298 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:55:57 smithi136 ceph-mon[33050]: pgmap v712: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:56:00.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:55:59 smithi076 ceph-mon[32774]: pgmap v713: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:56:00.292 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:55:59 smithi072 ceph-mon[28184]: pgmap v713: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:56:00.298 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:55:59 smithi136 ceph-mon[33050]: pgmap v713: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:56:02.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:56:01 smithi076 ceph-mon[32774]: pgmap v714: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:56:02.292 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:56:01 smithi072 ceph-mon[28184]: pgmap v714: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:56:02.298 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:56:01 smithi136 ceph-mon[33050]: pgmap v714: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:56:04.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:56:03 smithi076 ceph-mon[32774]: pgmap v715: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:56:04.292 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:56:03 smithi072 ceph-mon[28184]: pgmap v715: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:56:04.298 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:56:03 smithi136 ceph-mon[33050]: pgmap v715: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:56:06.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:56:05 smithi076 ceph-mon[32774]: pgmap v716: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:56:06.292 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:56:05 smithi072 ceph-mon[28184]: pgmap v716: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:56:06.298 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:56:05 smithi136 ceph-mon[33050]: pgmap v716: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:56:08.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:56:07 smithi076 ceph-mon[32774]: pgmap v717: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:56:08.292 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:56:07 smithi072 ceph-mon[28184]: pgmap v717: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:56:08.298 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:56:07 smithi136 ceph-mon[33050]: pgmap v717: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:56:10.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:56:09 smithi076 ceph-mon[32774]: pgmap v718: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:56:10.292 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:56:09 smithi072 ceph-mon[28184]: pgmap v718: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:56:10.298 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:56:09 smithi136 ceph-mon[33050]: pgmap v718: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:56:11.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:56:10 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:56:11.292 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:56:10 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:56:11.298 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:56:10 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:56:12.299 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:56:12 smithi136 ceph-mon[33050]: pgmap v719: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:56:12.299 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:56:12 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:56:12.299 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:56:12 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:56:12.299 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:56:12 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:56:12.299 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:56:12 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:56:12.299 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:56:12 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:56:12.299 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:56:12 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:56:12.427 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:56:12 smithi076 ceph-mon[32774]: pgmap v719: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:56:12.427 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:56:12 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:56:12.427 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:56:12 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:56:12.427 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:56:12 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:56:12.427 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:56:12 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:56:12.427 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:56:12 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:56:12.427 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:56:12 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:56:12.542 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:56:12 smithi072 ceph-mon[28184]: pgmap v719: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:56:12.542 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:56:12 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:56:12.542 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:56:12 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:56:12.542 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:56:12 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:56:12.542 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:56:12 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:56:12.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:56:12 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:56:12.543 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:56:12 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:56:14.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:56:13 smithi076 ceph-mon[32774]: pgmap v720: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:56:14.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:56:13 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:56:14.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:56:13 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:56:14.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:56:13 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:56:14.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:56:13 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:56:14.292 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:56:13 smithi072 ceph-mon[28184]: pgmap v720: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:56:14.292 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:56:13 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:56:14.292 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:56:13 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:56:14.292 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:56:13 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:56:14.292 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:56:13 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:56:14.298 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:56:13 smithi136 ceph-mon[33050]: pgmap v720: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:56:14.298 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:56:13 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:56:14.298 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:56:13 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:56:14.298 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:56:13 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:56:14.298 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:56:13 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:56:15.542 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:56:15 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:56:15.542 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:56:15 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:56:15.548 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:56:15 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:56:15.548 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:56:15 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:56:15.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:56:15 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:56:15.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:56:15 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:56:16.542 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:56:16 smithi072 ceph-mon[28184]: pgmap v721: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:56:16.542 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:56:16 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:56:16.542 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:56:16 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:56:16.542 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:56:16 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:56:16.548 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:56:16 smithi136 ceph-mon[33050]: pgmap v721: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:56:16.548 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:56:16 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:56:16.548 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:56:16 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:56:16.549 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:56:16 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:56:16.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:56:16 smithi076 ceph-mon[32774]: pgmap v721: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:56:16.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:56:16 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:56:16.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:56:16 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:56:16.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:56:16 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:56:18.542 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:56:18 smithi072 ceph-mon[28184]: pgmap v722: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:56:18.548 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:56:18 smithi136 ceph-mon[33050]: pgmap v722: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:56:18.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:56:18 smithi076 ceph-mon[32774]: pgmap v722: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:56:20.542 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:56:20 smithi072 ceph-mon[28184]: pgmap v723: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:56:20.548 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:56:20 smithi136 ceph-mon[33050]: pgmap v723: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:56:20.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:56:20 smithi076 ceph-mon[32774]: pgmap v723: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:56:22.542 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:56:22 smithi072 ceph-mon[28184]: pgmap v724: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:56:22.548 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:56:22 smithi136 ceph-mon[33050]: pgmap v724: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:56:22.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:56:22 smithi076 ceph-mon[32774]: pgmap v724: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:56:24.542 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:56:24 smithi072 ceph-mon[28184]: pgmap v725: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:56:24.548 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:56:24 smithi136 ceph-mon[33050]: pgmap v725: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:56:24.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:56:24 smithi076 ceph-mon[32774]: pgmap v725: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:56:26.542 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:56:26 smithi072 ceph-mon[28184]: pgmap v726: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:56:26.548 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:56:26 smithi136 ceph-mon[33050]: pgmap v726: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:56:26.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:56:26 smithi076 ceph-mon[32774]: pgmap v726: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:56:28.542 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:56:28 smithi072 ceph-mon[28184]: pgmap v727: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:56:28.548 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:56:28 smithi136 ceph-mon[33050]: pgmap v727: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:56:28.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:56:28 smithi076 ceph-mon[32774]: pgmap v727: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:56:30.548 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:56:30 smithi136 ceph-mon[33050]: pgmap v728: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:56:30.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:56:30 smithi076 ceph-mon[32774]: pgmap v728: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:56:30.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:56:30 smithi072 ceph-mon[28184]: pgmap v728: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:56:32.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:56:32 smithi076 ceph-mon[32774]: pgmap v729: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:56:32.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:56:32 smithi072 ceph-mon[28184]: pgmap v729: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:56:32.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:56:32 smithi136 ceph-mon[33050]: pgmap v729: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:56:34.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:56:34 smithi076 ceph-mon[32774]: pgmap v730: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:56:34.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:56:34 smithi072 ceph-mon[28184]: pgmap v730: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:56:34.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:56:34 smithi136 ceph-mon[33050]: pgmap v730: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:56:36.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:56:36 smithi076 ceph-mon[32774]: pgmap v731: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:56:36.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:56:36 smithi072 ceph-mon[28184]: pgmap v731: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:56:36.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:56:36 smithi136 ceph-mon[33050]: pgmap v731: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:56:38.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:56:38 smithi076 ceph-mon[32774]: pgmap v732: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:56:38.795 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:56:38 smithi072 ceph-mon[28184]: pgmap v732: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:56:38.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:56:38 smithi136 ceph-mon[33050]: pgmap v732: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:56:40.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:56:40 smithi076 ceph-mon[32774]: pgmap v733: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:56:40.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:56:40 smithi072 ceph-mon[28184]: pgmap v733: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:56:40.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:56:40 smithi136 ceph-mon[33050]: pgmap v733: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:56:42.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:56:42 smithi076 ceph-mon[32774]: pgmap v734: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:56:42.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:56:42 smithi072 ceph-mon[28184]: pgmap v734: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:56:42.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:56:42 smithi136 ceph-mon[33050]: pgmap v734: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:56:44.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:56:44 smithi076 ceph-mon[32774]: pgmap v735: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:56:44.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:56:44 smithi072 ceph-mon[28184]: pgmap v735: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:56:44.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:56:44 smithi136 ceph-mon[33050]: pgmap v735: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:56:46.600 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:56:46 smithi136 ceph-mon[33050]: pgmap v736: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:56:46.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:56:46 smithi076 ceph-mon[32774]: pgmap v736: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:56:46.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:56:46 smithi072 ceph-mon[28184]: pgmap v736: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:56:48.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:56:48 smithi076 ceph-mon[32774]: pgmap v737: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:56:48.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:56:48 smithi072 ceph-mon[28184]: pgmap v737: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:56:48.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:56:48 smithi136 ceph-mon[33050]: pgmap v737: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:56:50.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:56:50 smithi076 ceph-mon[32774]: pgmap v738: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:56:50.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:56:50 smithi072 ceph-mon[28184]: pgmap v738: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:56:50.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:56:50 smithi136 ceph-mon[33050]: pgmap v738: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:56:52.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:56:52 smithi076 ceph-mon[32774]: pgmap v739: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:56:52.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:56:52 smithi072 ceph-mon[28184]: pgmap v739: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:56:52.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:56:52 smithi136 ceph-mon[33050]: pgmap v739: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:56:54.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:56:54 smithi076 ceph-mon[32774]: pgmap v740: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:56:54.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:56:54 smithi072 ceph-mon[28184]: pgmap v740: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:56:54.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:56:54 smithi136 ceph-mon[33050]: pgmap v740: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:56:56.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:56:56 smithi076 ceph-mon[32774]: pgmap v741: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:56:56.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:56:56 smithi072 ceph-mon[28184]: pgmap v741: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:56:56.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:56:56 smithi136 ceph-mon[33050]: pgmap v741: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:56:58.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:56:58 smithi076 ceph-mon[32774]: pgmap v742: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:56:58.791 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:56:58 smithi072 ceph-mon[28184]: pgmap v742: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:56:58.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:56:58 smithi136 ceph-mon[33050]: pgmap v742: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:57:00.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:57:00 smithi076 ceph-mon[32774]: pgmap v743: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:57:00.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:57:00 smithi072 ceph-mon[28184]: pgmap v743: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:57:00.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:57:00 smithi136 ceph-mon[33050]: pgmap v743: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:57:02.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:57:02 smithi076 ceph-mon[32774]: pgmap v744: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:57:02.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:57:02 smithi072 ceph-mon[28184]: pgmap v744: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:57:02.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:57:02 smithi136 ceph-mon[33050]: pgmap v744: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:57:04.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:57:04 smithi076 ceph-mon[32774]: pgmap v745: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:57:04.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:57:04 smithi072 ceph-mon[28184]: pgmap v745: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:57:04.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:57:04 smithi136 ceph-mon[33050]: pgmap v745: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:57:06.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:57:06 smithi076 ceph-mon[32774]: pgmap v746: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:57:06.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:57:06 smithi072 ceph-mon[28184]: pgmap v746: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:57:06.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:57:06 smithi136 ceph-mon[33050]: pgmap v746: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:57:08.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:57:08 smithi076 ceph-mon[32774]: pgmap v747: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:57:08.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:57:08 smithi072 ceph-mon[28184]: pgmap v747: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:57:08.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:57:08 smithi136 ceph-mon[33050]: pgmap v747: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:57:10.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:57:10 smithi076 ceph-mon[32774]: pgmap v748: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:57:10.791 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:57:10 smithi072 ceph-mon[28184]: pgmap v748: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:57:10.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:57:10 smithi136 ceph-mon[33050]: pgmap v748: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:57:12.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:57:12 smithi076 ceph-mon[32774]: pgmap v749: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:57:12.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:57:12 smithi072 ceph-mon[28184]: pgmap v749: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:57:12.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:57:12 smithi136 ceph-mon[33050]: pgmap v749: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:57:14.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:57:14 smithi076 ceph-mon[32774]: pgmap v750: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:57:14.791 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:57:14 smithi072 ceph-mon[28184]: pgmap v750: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:57:14.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:57:14 smithi136 ceph-mon[33050]: pgmap v750: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:57:16.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:57:16 smithi076 ceph-mon[32774]: pgmap v751: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:57:16.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:57:16 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:57:16.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:57:16 smithi072 ceph-mon[28184]: pgmap v751: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:57:16.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:57:16 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:57:16.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:57:16 smithi136 ceph-mon[33050]: pgmap v751: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:57:16.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:57:16 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:57:17.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:57:17 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:57:17.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:57:17 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:57:17.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:57:17 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:57:17.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:57:17 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:57:17.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:57:17 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:57:17.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:57:17 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:57:17.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:57:17 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:57:17.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:57:17 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:57:17.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:57:17 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:57:18.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:57:18 smithi072 ceph-mon[28184]: pgmap v752: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:57:18.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:57:18 smithi136 ceph-mon[33050]: pgmap v752: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:57:18.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:57:18 smithi076 ceph-mon[32774]: pgmap v752: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:57:20.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:57:20 smithi072 ceph-mon[28184]: pgmap v753: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:57:20.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:57:20 smithi136 ceph-mon[33050]: pgmap v753: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:57:20.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:57:20 smithi076 ceph-mon[32774]: pgmap v753: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:57:22.729 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:57:22 smithi076 ceph-mon[32774]: pgmap v754: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:57:22.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:57:22 smithi072 ceph-mon[28184]: pgmap v754: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:57:22.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:57:22 smithi136 ceph-mon[33050]: pgmap v754: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:57:24.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:57:24 smithi072 ceph-mon[28184]: pgmap v755: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:57:24.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:57:24 smithi136 ceph-mon[33050]: pgmap v755: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:57:24.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:57:24 smithi076 ceph-mon[32774]: pgmap v755: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:57:26.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:57:26 smithi072 ceph-mon[28184]: pgmap v756: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:57:26.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:57:26 smithi136 ceph-mon[33050]: pgmap v756: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:57:26.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:57:26 smithi076 ceph-mon[32774]: pgmap v756: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:57:28.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:57:28 smithi072 ceph-mon[28184]: pgmap v757: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:57:28.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:57:28 smithi136 ceph-mon[33050]: pgmap v757: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:57:28.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:57:28 smithi076 ceph-mon[32774]: pgmap v757: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:57:30.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:57:30 smithi072 ceph-mon[28184]: pgmap v758: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:57:30.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:57:30 smithi136 ceph-mon[33050]: pgmap v758: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:57:30.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:57:30 smithi076 ceph-mon[32774]: pgmap v758: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:57:32.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:57:32 smithi072 ceph-mon[28184]: pgmap v759: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:57:32.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:57:32 smithi136 ceph-mon[33050]: pgmap v759: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:57:32.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:57:32 smithi076 ceph-mon[32774]: pgmap v759: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:57:34.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:57:34 smithi072 ceph-mon[28184]: pgmap v760: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:57:34.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:57:34 smithi136 ceph-mon[33050]: pgmap v760: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:57:34.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:57:34 smithi076 ceph-mon[32774]: pgmap v760: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:57:36.791 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:57:36 smithi072 ceph-mon[28184]: pgmap v761: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:57:36.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:57:36 smithi136 ceph-mon[33050]: pgmap v761: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:57:36.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:57:36 smithi076 ceph-mon[32774]: pgmap v761: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:57:38.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:57:38 smithi072 ceph-mon[28184]: pgmap v762: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:57:38.797 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:57:38 smithi136 ceph-mon[33050]: pgmap v762: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:57:38.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:57:38 smithi076 ceph-mon[32774]: pgmap v762: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:57:40.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:57:40 smithi072 ceph-mon[28184]: pgmap v763: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:57:40.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:57:40 smithi136 ceph-mon[33050]: pgmap v763: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:57:40.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:57:40 smithi076 ceph-mon[32774]: pgmap v763: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:57:42.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:57:42 smithi072 ceph-mon[28184]: pgmap v764: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:57:42.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:57:42 smithi136 ceph-mon[33050]: pgmap v764: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:57:42.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:57:42 smithi076 ceph-mon[32774]: pgmap v764: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:57:44.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:57:44 smithi072 ceph-mon[28184]: pgmap v765: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:57:44.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:57:44 smithi136 ceph-mon[33050]: pgmap v765: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:57:44.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:57:44 smithi076 ceph-mon[32774]: pgmap v765: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:57:46.791 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:57:46 smithi072 ceph-mon[28184]: pgmap v766: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:57:46.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:57:46 smithi136 ceph-mon[33050]: pgmap v766: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:57:46.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:57:46 smithi076 ceph-mon[32774]: pgmap v766: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:57:48.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:57:48 smithi072 ceph-mon[28184]: pgmap v767: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:57:48.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:57:48 smithi136 ceph-mon[33050]: pgmap v767: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:57:48.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:57:48 smithi076 ceph-mon[32774]: pgmap v767: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:57:50.791 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:57:50 smithi072 ceph-mon[28184]: pgmap v768: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:57:50.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:57:50 smithi136 ceph-mon[33050]: pgmap v768: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:57:50.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:57:50 smithi076 ceph-mon[32774]: pgmap v768: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:57:52.791 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:57:52 smithi072 ceph-mon[28184]: pgmap v769: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:57:52.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:57:52 smithi136 ceph-mon[33050]: pgmap v769: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:57:52.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:57:52 smithi076 ceph-mon[32774]: pgmap v769: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:57:54.791 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:57:54 smithi072 ceph-mon[28184]: pgmap v770: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:57:54.797 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:57:54 smithi136 ceph-mon[33050]: pgmap v770: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:57:54.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:57:54 smithi076 ceph-mon[32774]: pgmap v770: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:57:56.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:57:56 smithi072 ceph-mon[28184]: pgmap v771: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:57:56.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:57:56 smithi136 ceph-mon[33050]: pgmap v771: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:57:56.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:57:56 smithi076 ceph-mon[32774]: pgmap v771: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:57:58.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:57:58 smithi136 ceph-mon[33050]: pgmap v772: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:57:58.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:57:58 smithi076 ceph-mon[32774]: pgmap v772: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:57:59.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:57:58 smithi072 ceph-mon[28184]: pgmap v772: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:58:00.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:58:00 smithi076 ceph-mon[32774]: pgmap v773: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:58:01.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:58:00 smithi072 ceph-mon[28184]: pgmap v773: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:58:01.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:58:00 smithi136 ceph-mon[33050]: pgmap v773: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:58:02.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:58:02 smithi076 ceph-mon[32774]: pgmap v774: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:58:03.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:58:02 smithi072 ceph-mon[28184]: pgmap v774: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:58:03.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:58:02 smithi136 ceph-mon[33050]: pgmap v774: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:58:04.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:58:04 smithi076 ceph-mon[32774]: pgmap v775: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:58:05.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:58:04 smithi072 ceph-mon[28184]: pgmap v775: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:58:05.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:58:04 smithi136 ceph-mon[33050]: pgmap v775: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:58:06.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:58:06 smithi076 ceph-mon[32774]: pgmap v776: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:58:07.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:58:06 smithi072 ceph-mon[28184]: pgmap v776: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:58:07.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:58:06 smithi136 ceph-mon[33050]: pgmap v776: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:58:08.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:58:08 smithi076 ceph-mon[32774]: pgmap v777: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:58:09.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:58:08 smithi072 ceph-mon[28184]: pgmap v777: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:58:09.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:58:08 smithi136 ceph-mon[33050]: pgmap v777: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:58:10.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:58:10 smithi076 ceph-mon[32774]: pgmap v778: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:58:11.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:58:10 smithi072 ceph-mon[28184]: pgmap v778: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:58:11.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:58:10 smithi136 ceph-mon[33050]: pgmap v778: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:58:12.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:58:12 smithi076 ceph-mon[32774]: pgmap v779: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:58:13.042 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:58:12 smithi072 ceph-mon[28184]: pgmap v779: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:58:13.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:58:12 smithi136 ceph-mon[33050]: pgmap v779: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:58:14.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:58:14 smithi076 ceph-mon[32774]: pgmap v780: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:58:15.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:58:14 smithi072 ceph-mon[28184]: pgmap v780: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:58:15.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:58:14 smithi136 ceph-mon[33050]: pgmap v780: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:58:15.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:58:15.974+0000 7f46b4c00640 1 -- 172.21.15.72:0/1244301295 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f4694006080 con 0x7f46b8103b70 2024-04-26T23:58:15.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-26T23:58:15.975+0000 7f46b6000640 1 -- 172.21.15.72:0/1244301295 <== mon.1 v2:172.21.15.136:3300/0 11 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f4694006080 con 0x7f46b8103b70 2024-04-26T23:58:16.850 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:58:16 smithi136 ceph-mon[33050]: pgmap v781: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:58:16.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:58:16 smithi076 ceph-mon[32774]: pgmap v781: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:58:17.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:58:16 smithi072 ceph-mon[28184]: pgmap v781: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:58:17.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:58:17 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:58:18.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:58:17 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:58:18.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:58:17 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:58:18.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:58:18 smithi076 ceph-mon[32774]: pgmap v782: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:58:19.042 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:58:18 smithi072 ceph-mon[28184]: pgmap v782: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:58:19.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:58:18 smithi136 ceph-mon[33050]: pgmap v782: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:58:19.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:58:19 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:58:19.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:58:19 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:58:19.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:58:19 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:58:20.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:58:19 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:58:20.042 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:58:19 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:58:20.042 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:58:19 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:58:20.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:58:19 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:58:20.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:58:19 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:58:20.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:58:19 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:58:21.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:58:20 smithi072 ceph-mon[28184]: pgmap v783: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:58:21.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:58:20 smithi136 ceph-mon[33050]: pgmap v783: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:58:21.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:58:20 smithi076 ceph-mon[32774]: pgmap v783: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:58:22.979 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:58:22 smithi076 ceph-mon[32774]: pgmap v784: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:58:23.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:58:22 smithi072 ceph-mon[28184]: pgmap v784: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:58:23.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:58:22 smithi136 ceph-mon[33050]: pgmap v784: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:58:25.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:58:24 smithi072 ceph-mon[28184]: pgmap v785: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:58:25.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:58:24 smithi136 ceph-mon[33050]: pgmap v785: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:58:25.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:58:24 smithi076 ceph-mon[32774]: pgmap v785: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:58:27.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:58:26 smithi072 ceph-mon[28184]: pgmap v786: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:58:27.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:58:26 smithi136 ceph-mon[33050]: pgmap v786: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:58:27.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:58:26 smithi076 ceph-mon[32774]: pgmap v786: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:58:29.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:58:28 smithi072 ceph-mon[28184]: pgmap v787: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:58:29.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:58:28 smithi136 ceph-mon[33050]: pgmap v787: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:58:29.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:58:28 smithi076 ceph-mon[32774]: pgmap v787: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:58:31.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:58:30 smithi072 ceph-mon[28184]: pgmap v788: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-26T23:58:31.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:58:30 smithi136 ceph-mon[33050]: pgmap v788: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-26T23:58:31.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:58:30 smithi076 ceph-mon[32774]: pgmap v788: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-26T23:58:33.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:58:32 smithi072 ceph-mon[28184]: pgmap v789: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T23:58:33.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:58:32 smithi136 ceph-mon[33050]: pgmap v789: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T23:58:33.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:58:32 smithi076 ceph-mon[32774]: pgmap v789: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T23:58:35.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:58:34 smithi072 ceph-mon[28184]: pgmap v790: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T23:58:35.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:58:34 smithi136 ceph-mon[33050]: pgmap v790: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T23:58:35.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:58:34 smithi076 ceph-mon[32774]: pgmap v790: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T23:58:37.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:58:36 smithi072 ceph-mon[28184]: pgmap v791: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:58:37.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:58:36 smithi136 ceph-mon[33050]: pgmap v791: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:58:37.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:58:36 smithi076 ceph-mon[32774]: pgmap v791: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:58:39.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:58:38 smithi072 ceph-mon[28184]: pgmap v792: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T23:58:39.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:58:38 smithi136 ceph-mon[33050]: pgmap v792: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T23:58:39.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:58:38 smithi076 ceph-mon[32774]: pgmap v792: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T23:58:41.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:58:40 smithi072 ceph-mon[28184]: pgmap v793: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-26T23:58:41.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:58:40 smithi136 ceph-mon[33050]: pgmap v793: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-26T23:58:41.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:58:40 smithi076 ceph-mon[32774]: pgmap v793: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-26T23:58:43.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:58:42 smithi072 ceph-mon[28184]: pgmap v794: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:58:43.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:58:42 smithi136 ceph-mon[33050]: pgmap v794: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:58:43.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:58:42 smithi076 ceph-mon[32774]: pgmap v794: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:58:45.042 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:58:44 smithi072 ceph-mon[28184]: pgmap v795: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:58:45.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:58:44 smithi136 ceph-mon[33050]: pgmap v795: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:58:45.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:58:44 smithi076 ceph-mon[32774]: pgmap v795: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:58:47.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:58:46 smithi072 ceph-mon[28184]: pgmap v796: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:58:47.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:58:46 smithi136 ceph-mon[33050]: pgmap v796: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:58:47.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:58:46 smithi076 ceph-mon[32774]: pgmap v796: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:58:49.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:58:48 smithi072 ceph-mon[28184]: pgmap v797: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:58:49.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:58:48 smithi136 ceph-mon[33050]: pgmap v797: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:58:49.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:58:48 smithi076 ceph-mon[32774]: pgmap v797: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:58:51.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:58:50 smithi072 ceph-mon[28184]: pgmap v798: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:58:51.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:58:50 smithi136 ceph-mon[33050]: pgmap v798: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:58:51.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:58:50 smithi076 ceph-mon[32774]: pgmap v798: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:58:53.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:58:52 smithi072 ceph-mon[28184]: pgmap v799: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:58:53.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:58:52 smithi136 ceph-mon[33050]: pgmap v799: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:58:53.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:58:52 smithi076 ceph-mon[32774]: pgmap v799: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:58:55.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:58:54 smithi072 ceph-mon[28184]: pgmap v800: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:58:55.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:58:54 smithi136 ceph-mon[33050]: pgmap v800: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:58:55.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:58:54 smithi076 ceph-mon[32774]: pgmap v800: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:58:57.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:58:56 smithi136 ceph-mon[33050]: pgmap v801: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:58:57.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:58:56 smithi076 ceph-mon[32774]: pgmap v801: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:58:57.291 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:58:56 smithi072 ceph-mon[28184]: pgmap v801: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:58:58.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:58:57 smithi076 ceph-mon[32774]: pgmap v802: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T23:58:58.291 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:58:57 smithi072 ceph-mon[28184]: pgmap v802: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T23:58:58.297 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:58:57 smithi136 ceph-mon[33050]: pgmap v802: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T23:59:00.291 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:58:59 smithi072 ceph-mon[28184]: pgmap v803: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-26T23:59:00.297 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:58:59 smithi136 ceph-mon[33050]: pgmap v803: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-26T23:59:00.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:58:59 smithi076 ceph-mon[32774]: pgmap v803: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-26T23:59:02.291 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:59:01 smithi072 ceph-mon[28184]: pgmap v804: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T23:59:02.297 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:59:01 smithi136 ceph-mon[33050]: pgmap v804: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T23:59:02.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:59:01 smithi076 ceph-mon[32774]: pgmap v804: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T23:59:04.291 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:59:03 smithi072 ceph-mon[28184]: pgmap v805: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T23:59:04.298 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:59:03 smithi136 ceph-mon[33050]: pgmap v805: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T23:59:04.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:59:03 smithi076 ceph-mon[32774]: pgmap v805: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T23:59:06.291 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:59:05 smithi072 ceph-mon[28184]: pgmap v806: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:59:06.297 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:59:05 smithi136 ceph-mon[33050]: pgmap v806: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:59:06.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:59:05 smithi076 ceph-mon[32774]: pgmap v806: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:59:08.291 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:59:07 smithi072 ceph-mon[28184]: pgmap v807: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:59:08.297 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:59:07 smithi136 ceph-mon[33050]: pgmap v807: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:59:08.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:59:07 smithi076 ceph-mon[32774]: pgmap v807: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:59:10.291 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:59:09 smithi072 ceph-mon[28184]: pgmap v808: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:59:10.297 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:59:09 smithi136 ceph-mon[33050]: pgmap v808: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:59:10.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:59:09 smithi076 ceph-mon[32774]: pgmap v808: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:59:12.291 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:59:11 smithi072 ceph-mon[28184]: pgmap v809: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:59:12.297 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:59:11 smithi136 ceph-mon[33050]: pgmap v809: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:59:12.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:59:11 smithi076 ceph-mon[32774]: pgmap v809: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:59:14.291 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:59:13 smithi072 ceph-mon[28184]: pgmap v810: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:59:14.297 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:59:13 smithi136 ceph-mon[33050]: pgmap v810: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:59:14.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:59:13 smithi076 ceph-mon[32774]: pgmap v810: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:59:16.291 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:59:15 smithi072 ceph-mon[28184]: pgmap v811: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:59:16.297 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:59:15 smithi136 ceph-mon[33050]: pgmap v811: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:59:16.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:59:15 smithi076 ceph-mon[32774]: pgmap v811: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:59:18.291 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:59:18 smithi072 ceph-mon[28184]: pgmap v812: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:59:18.297 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:59:18 smithi136 ceph-mon[33050]: pgmap v812: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:59:18.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:59:18 smithi076 ceph-mon[32774]: pgmap v812: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:59:19.291 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:59:19 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:59:19.297 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:59:19 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:59:19.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:59:19 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T23:59:20.291 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:59:20 smithi072 ceph-mon[28184]: pgmap v813: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:59:20.297 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:59:20 smithi136 ceph-mon[33050]: pgmap v813: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:59:20.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:59:20 smithi076 ceph-mon[32774]: pgmap v813: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:59:21.291 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:59:21 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:59:21.291 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:59:21 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:59:21.292 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:59:21 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:59:21.297 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:59:21 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:59:21.297 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:59:21 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:59:21.298 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:59:21 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:59:21.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:59:21 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T23:59:21.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:59:21 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T23:59:21.427 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:59:21 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-26T23:59:22.541 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:59:22 smithi072 ceph-mon[28184]: pgmap v814: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:59:22.548 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:59:22 smithi136 ceph-mon[33050]: pgmap v814: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:59:22.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:59:22 smithi076 ceph-mon[32774]: pgmap v814: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:59:24.541 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:59:24 smithi072 ceph-mon[28184]: pgmap v815: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:59:24.547 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:59:24 smithi136 ceph-mon[33050]: pgmap v815: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:59:24.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:59:24 smithi076 ceph-mon[32774]: pgmap v815: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:59:26.541 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:59:26 smithi072 ceph-mon[28184]: pgmap v816: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:59:26.547 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:59:26 smithi136 ceph-mon[33050]: pgmap v816: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:59:26.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:59:26 smithi076 ceph-mon[32774]: pgmap v816: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:59:28.541 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:59:28 smithi072 ceph-mon[28184]: pgmap v817: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:59:28.547 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:59:28 smithi136 ceph-mon[33050]: pgmap v817: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:59:28.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:59:28 smithi076 ceph-mon[32774]: pgmap v817: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:59:30.541 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:59:30 smithi072 ceph-mon[28184]: pgmap v818: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:59:30.547 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:59:30 smithi136 ceph-mon[33050]: pgmap v818: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:59:30.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:59:30 smithi076 ceph-mon[32774]: pgmap v818: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:59:32.541 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:59:32 smithi072 ceph-mon[28184]: pgmap v819: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:59:32.547 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:59:32 smithi136 ceph-mon[33050]: pgmap v819: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:59:32.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:59:32 smithi076 ceph-mon[32774]: pgmap v819: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:59:34.541 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:59:34 smithi072 ceph-mon[28184]: pgmap v820: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:59:34.547 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:59:34 smithi136 ceph-mon[33050]: pgmap v820: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:59:34.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:59:34 smithi076 ceph-mon[32774]: pgmap v820: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:59:36.541 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:59:36 smithi072 ceph-mon[28184]: pgmap v821: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:59:36.547 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:59:36 smithi136 ceph-mon[33050]: pgmap v821: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:59:36.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:59:36 smithi076 ceph-mon[32774]: pgmap v821: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:59:38.541 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:59:38 smithi072 ceph-mon[28184]: pgmap v822: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:59:38.547 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:59:38 smithi136 ceph-mon[33050]: pgmap v822: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:59:38.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:59:38 smithi076 ceph-mon[32774]: pgmap v822: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:59:40.541 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:59:40 smithi072 ceph-mon[28184]: pgmap v823: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:59:40.547 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:59:40 smithi136 ceph-mon[33050]: pgmap v823: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:59:40.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:59:40 smithi076 ceph-mon[32774]: pgmap v823: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:59:42.541 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:59:42 smithi072 ceph-mon[28184]: pgmap v824: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:59:42.547 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:59:42 smithi136 ceph-mon[33050]: pgmap v824: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:59:42.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:59:42 smithi076 ceph-mon[32774]: pgmap v824: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:59:44.541 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:59:44 smithi072 ceph-mon[28184]: pgmap v825: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:59:44.547 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:59:44 smithi136 ceph-mon[33050]: pgmap v825: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:59:44.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:59:44 smithi076 ceph-mon[32774]: pgmap v825: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:59:46.541 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:59:46 smithi072 ceph-mon[28184]: pgmap v826: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:59:46.548 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:59:46 smithi136 ceph-mon[33050]: pgmap v826: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:59:46.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:59:46 smithi076 ceph-mon[32774]: pgmap v826: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:59:48.541 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:59:48 smithi072 ceph-mon[28184]: pgmap v827: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:59:48.547 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:59:48 smithi136 ceph-mon[33050]: pgmap v827: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:59:48.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:59:48 smithi076 ceph-mon[32774]: pgmap v827: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:59:50.541 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:59:50 smithi072 ceph-mon[28184]: pgmap v828: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:59:50.547 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:59:50 smithi136 ceph-mon[33050]: pgmap v828: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:59:50.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:59:50 smithi076 ceph-mon[32774]: pgmap v828: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-26T23:59:52.541 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:59:52 smithi072 ceph-mon[28184]: pgmap v829: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:59:52.547 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:59:52 smithi136 ceph-mon[33050]: pgmap v829: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:59:52.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:59:52 smithi076 ceph-mon[32774]: pgmap v829: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-26T23:59:54.547 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:59:54 smithi136 ceph-mon[33050]: pgmap v830: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:59:54.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:59:54 smithi076 ceph-mon[32774]: pgmap v830: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:59:54.791 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:59:54 smithi072 ceph-mon[28184]: pgmap v830: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:59:56.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:59:56 smithi076 ceph-mon[32774]: pgmap v831: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:59:56.791 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:59:56 smithi072 ceph-mon[28184]: pgmap v831: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:59:56.797 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:59:56 smithi136 ceph-mon[33050]: pgmap v831: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T23:59:58.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 26 23:59:58 smithi076 ceph-mon[32774]: pgmap v832: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:59:58.791 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 26 23:59:58 smithi072 ceph-mon[28184]: pgmap v832: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T23:59:58.797 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 26 23:59:58 smithi136 ceph-mon[33050]: pgmap v832: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:00.615 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:00:00 smithi076 ceph-mon[32774]: pgmap v833: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-27T00:00:00.616 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:00:00 smithi076 ceph-mon[32774]: overall HEALTH_OK 2024-04-27T00:00:00.791 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:00:00 smithi072 ceph-mon[28184]: pgmap v833: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-27T00:00:00.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:00:00 smithi072 ceph-mon[28184]: overall HEALTH_OK 2024-04-27T00:00:00.797 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:00:00 smithi136 ceph-mon[33050]: pgmap v833: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-27T00:00:00.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:00:00 smithi136 ceph-mon[33050]: overall HEALTH_OK 2024-04-27T00:00:02.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:00:02 smithi076 ceph-mon[32774]: pgmap v834: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-27T00:00:02.791 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:00:02 smithi072 ceph-mon[28184]: pgmap v834: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-27T00:00:02.797 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:00:02 smithi136 ceph-mon[33050]: pgmap v834: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-27T00:00:04.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:00:04 smithi076 ceph-mon[32774]: pgmap v835: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:04.791 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:00:04 smithi072 ceph-mon[28184]: pgmap v835: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:04.797 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:00:04 smithi136 ceph-mon[33050]: pgmap v835: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:06.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:00:06 smithi076 ceph-mon[32774]: pgmap v836: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:00:06.791 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:00:06 smithi072 ceph-mon[28184]: pgmap v836: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:00:06.797 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:00:06 smithi136 ceph-mon[33050]: pgmap v836: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:00:08.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:00:08 smithi076 ceph-mon[32774]: pgmap v837: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:08.791 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:00:08 smithi072 ceph-mon[28184]: pgmap v837: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:08.797 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:00:08 smithi136 ceph-mon[33050]: pgmap v837: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:10.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:00:10 smithi076 ceph-mon[32774]: pgmap v838: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-27T00:00:10.791 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:00:10 smithi072 ceph-mon[28184]: pgmap v838: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-27T00:00:10.797 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:00:10 smithi136 ceph-mon[33050]: pgmap v838: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-27T00:00:12.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:00:12 smithi076 ceph-mon[32774]: pgmap v839: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-27T00:00:12.791 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:00:12 smithi072 ceph-mon[28184]: pgmap v839: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-27T00:00:12.797 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:00:12 smithi136 ceph-mon[33050]: pgmap v839: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-27T00:00:14.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:00:14 smithi076 ceph-mon[32774]: pgmap v840: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:14.791 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:00:14 smithi072 ceph-mon[28184]: pgmap v840: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:14.797 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:00:14 smithi136 ceph-mon[33050]: pgmap v840: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:16.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:00:16 smithi076 ceph-mon[32774]: pgmap v841: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:00:16.791 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:00:16 smithi072 ceph-mon[28184]: pgmap v841: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:00:16.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:00:16 smithi136 ceph-mon[33050]: pgmap v841: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:00:18.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:00:18 smithi076 ceph-mon[32774]: pgmap v842: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:18.791 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:00:18 smithi072 ceph-mon[28184]: pgmap v842: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:18.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:00:18 smithi136 ceph-mon[33050]: pgmap v842: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:20.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:00:20 smithi076 ceph-mon[32774]: pgmap v843: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-27T00:00:20.791 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:00:20 smithi072 ceph-mon[28184]: pgmap v843: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-27T00:00:20.797 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:00:20 smithi136 ceph-mon[33050]: pgmap v843: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-27T00:00:21.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:00:21 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:00:21.790 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:00:21 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:00:21.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:00:21 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:00:22.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:00:22 smithi076 ceph-mon[32774]: pgmap v844: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-27T00:00:22.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:00:22 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T00:00:22.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:00:22 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T00:00:22.677 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:00:22 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-27T00:00:22.743 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:00:22 smithi072 ceph-mon[28184]: pgmap v844: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-27T00:00:22.744 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:00:22 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T00:00:22.744 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:00:22 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T00:00:22.744 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:00:22 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-27T00:00:22.797 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:00:22 smithi136 ceph-mon[33050]: pgmap v844: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-27T00:00:22.797 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:00:22 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T00:00:22.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:00:22 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T00:00:22.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:00:22 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-27T00:00:24.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:00:24 smithi076 ceph-mon[32774]: pgmap v845: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:24.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:00:24 smithi072 ceph-mon[28184]: pgmap v845: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:24.797 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:00:24 smithi136 ceph-mon[33050]: pgmap v845: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:26.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:00:26 smithi076 ceph-mon[32774]: pgmap v846: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:00:26.791 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:00:26 smithi072 ceph-mon[28184]: pgmap v846: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:00:26.797 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:00:26 smithi136 ceph-mon[33050]: pgmap v846: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:00:28.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:00:28 smithi076 ceph-mon[32774]: pgmap v847: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:28.791 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:00:28 smithi072 ceph-mon[28184]: pgmap v847: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:28.797 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:00:28 smithi136 ceph-mon[33050]: pgmap v847: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:30.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:00:30 smithi076 ceph-mon[32774]: pgmap v848: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-27T00:00:30.791 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:00:30 smithi072 ceph-mon[28184]: pgmap v848: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-27T00:00:30.797 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:00:30 smithi136 ceph-mon[33050]: pgmap v848: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-27T00:00:32.676 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:00:32 smithi076 ceph-mon[32774]: pgmap v849: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-27T00:00:32.791 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:00:32 smithi072 ceph-mon[28184]: pgmap v849: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-27T00:00:32.797 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:00:32 smithi136 ceph-mon[33050]: pgmap v849: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-27T00:00:34.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:00:34 smithi076 ceph-mon[32774]: pgmap v850: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:35.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:00:34 smithi072 ceph-mon[28184]: pgmap v850: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:35.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:00:34 smithi136 ceph-mon[33050]: pgmap v850: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:36.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:00:36 smithi076 ceph-mon[32774]: pgmap v851: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:00:37.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:00:36 smithi072 ceph-mon[28184]: pgmap v851: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:00:37.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:00:36 smithi136 ceph-mon[33050]: pgmap v851: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:00:38.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:00:38 smithi076 ceph-mon[32774]: pgmap v852: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:39.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:00:38 smithi072 ceph-mon[28184]: pgmap v852: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:39.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:00:38 smithi136 ceph-mon[33050]: pgmap v852: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:40.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:00:40 smithi076 ceph-mon[32774]: pgmap v853: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-27T00:00:41.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:00:40 smithi072 ceph-mon[28184]: pgmap v853: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-27T00:00:41.182 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:00:40 smithi136 ceph-mon[33050]: pgmap v853: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-27T00:00:42.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:00:42 smithi076 ceph-mon[32774]: pgmap v854: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-27T00:00:43.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:00:42 smithi072 ceph-mon[28184]: pgmap v854: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-27T00:00:43.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:00:42 smithi136 ceph-mon[33050]: pgmap v854: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-27T00:00:44.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:00:44 smithi076 ceph-mon[32774]: pgmap v855: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:45.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:00:44 smithi072 ceph-mon[28184]: pgmap v855: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:45.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:00:44 smithi136 ceph-mon[33050]: pgmap v855: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:46.849 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:00:46 smithi136 ceph-mon[33050]: pgmap v856: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:00:46.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:00:46 smithi076 ceph-mon[32774]: pgmap v856: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:00:47.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:00:46 smithi072 ceph-mon[28184]: pgmap v856: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:00:48.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:00:48 smithi076 ceph-mon[32774]: pgmap v857: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:49.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:00:48 smithi072 ceph-mon[28184]: pgmap v857: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:49.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:00:48 smithi136 ceph-mon[33050]: pgmap v857: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:50.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:00:50 smithi076 ceph-mon[32774]: pgmap v858: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:00:51.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:00:50 smithi072 ceph-mon[28184]: pgmap v858: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:00:51.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:00:50 smithi136 ceph-mon[33050]: pgmap v858: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:00:52.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:00:52 smithi076 ceph-mon[32774]: pgmap v859: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-27T00:00:53.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:00:52 smithi072 ceph-mon[28184]: pgmap v859: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-27T00:00:53.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:00:52 smithi136 ceph-mon[33050]: pgmap v859: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-27T00:00:54.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:00:54 smithi076 ceph-mon[32774]: pgmap v860: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:55.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:00:54 smithi072 ceph-mon[28184]: pgmap v860: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:55.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:00:54 smithi136 ceph-mon[33050]: pgmap v860: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:56.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:00:56 smithi076 ceph-mon[32774]: pgmap v861: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:00:57.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:00:56 smithi072 ceph-mon[28184]: pgmap v861: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:00:57.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:00:56 smithi136 ceph-mon[33050]: pgmap v861: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:00:58.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:00:58 smithi076 ceph-mon[32774]: pgmap v862: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:59.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:00:58 smithi072 ceph-mon[28184]: pgmap v862: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:00:59.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:00:58 smithi136 ceph-mon[33050]: pgmap v862: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:01:00.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:00 smithi076 ceph-mon[32774]: pgmap v863: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:01:01.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:00 smithi072 ceph-mon[28184]: pgmap v863: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:01:01.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:00 smithi136 ceph-mon[33050]: pgmap v863: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:01:02.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:02 smithi076 ceph-mon[32774]: pgmap v864: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:01:03.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:02 smithi072 ceph-mon[28184]: pgmap v864: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:01:03.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:02 smithi136 ceph-mon[33050]: pgmap v864: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:01:04.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:04 smithi076 ceph-mon[32774]: pgmap v865: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:01:05.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:04 smithi072 ceph-mon[28184]: pgmap v865: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:01:05.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:04 smithi136 ceph-mon[33050]: pgmap v865: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:01:06.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:06 smithi076 ceph-mon[32774]: pgmap v866: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:01:07.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:06 smithi072 ceph-mon[28184]: pgmap v866: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:01:07.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:06 smithi136 ceph-mon[33050]: pgmap v866: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:01:08.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:08 smithi076 ceph-mon[32774]: pgmap v867: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:01:09.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:08 smithi072 ceph-mon[28184]: pgmap v867: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:01:09.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:08 smithi136 ceph-mon[33050]: pgmap v867: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:01:10.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:10 smithi076 ceph-mon[32774]: pgmap v868: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:01:11.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:10 smithi072 ceph-mon[28184]: pgmap v868: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:01:11.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:10 smithi136 ceph-mon[33050]: pgmap v868: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:01:12.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:12 smithi076 ceph-mon[32774]: pgmap v869: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:01:13.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:12 smithi072 ceph-mon[28184]: pgmap v869: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:01:13.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:12 smithi136 ceph-mon[33050]: pgmap v869: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:01:14.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:14 smithi076 ceph-mon[32774]: pgmap v870: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:01:15.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:14 smithi072 ceph-mon[28184]: pgmap v870: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:01:15.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:14 smithi136 ceph-mon[33050]: pgmap v870: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:01:15.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:15.888+0000 7f46bd400640 1 -- 172.21.15.72:0/1244301295 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f46a00a3cf0 msgr2=0x7f46a00a61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900025399) for more than 900000000 us, fault. 2024-04-27T00:01:15.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:15.888+0000 7f46bd400640 1 --2- 172.21.15.72:0/1244301295 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f46a00a3cf0 0x7f46a00a61b0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f46ac009bd0 tx=0x7f46ac005f90 comp rx=0 tx=0).stop 2024-04-27T00:01:15.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:15.888+0000 7f46b6000640 1 -- 172.21.15.72:0/1244301295 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f46a00a3cf0 msgr2=0x7f46a00a61b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-27T00:01:15.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:15.888+0000 7f46b6000640 1 --2- 172.21.15.72:0/1244301295 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f46a00a3cf0 0x7f46a00a61b0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:15.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:15.888+0000 7f46b6000640 1 --2- 172.21.15.72:0/1244301295 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f46a00af520 0x7f46a00b1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:15.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:15.888+0000 7f46b6000640 1 -- 172.21.15.72:0/1244301295 --> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f46a4046e50 con 0x7f46a00af520 2024-04-27T00:01:15.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:15.889+0000 7f46bd400640 1 --2- 172.21.15.72:0/1244301295 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f46a00af520 0x7f46a00b1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:15.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:15.891+0000 7f46bd400640 1 --2- 172.21.15.72:0/1244301295 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f46a00af520 0x7f46a00b1910 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f46ac005fd0 tx=0x7f46ac017000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:16.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.003+0000 7f46b4c00640 1 -- 172.21.15.72:0/1244301295 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f46940062f0 con 0x7f46b8103b70 2024-04-27T00:01:16.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.004+0000 7f46b6000640 1 -- 172.21.15.72:0/1244301295 <== mon.1 v2:172.21.15.136:3300/0 12 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f46940062f0 con 0x7f46b8103b70 2024-04-27T00:01:16.102 INFO:teuthology.orchestra.run.smithi072.stderr:Error ETIMEDOUT: Connection timed out. Please check the client's permission and connection. 2024-04-27T00:01:16.119 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-27T00:01:16.120 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/exec.py", line 66, in task remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi072 with status 110: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10'" 2024-04-27T00:01:16.354 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=ea5ac8f5d9454cb0b647089e80b72433 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/exec.py", line 66, in task remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi072 with status 110: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10'" 2024-04-27T00:01:16.356 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-27T00:01:16.380 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-27T00:01:16.380 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-27T00:01:16.380 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-27T00:01:16.445 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-27T00:01:16.445 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-27T00:01:16.501 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-27T00:01:16.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.702+0000 7fa430c00640 1 -- 172.21.15.72:0/1617364764 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa42c105b80 msgr2=0x7fa42c106000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:16.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.702+0000 7fa430c00640 1 --2- 172.21.15.72:0/1617364764 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa42c105b80 0x7fa42c106000 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa414009ca0 tx=0x7fa41402a8b0 comp rx=0 tx=0).stop 2024-04-27T00:01:16.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.702+0000 7fa430c00640 1 -- 172.21.15.72:0/1617364764 shutdown_connections 2024-04-27T00:01:16.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.702+0000 7fa430c00640 1 --2- 172.21.15.72:0/1617364764 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa42c106540 0x7fa42c10cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:16.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.702+0000 7fa430c00640 1 --2- 172.21.15.72:0/1617364764 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa42c105b80 0x7fa42c106000 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:16.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.702+0000 7fa430c00640 1 --2- 172.21.15.72:0/1617364764 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa42c104990 0x7fa42c104db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:16.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.702+0000 7fa430c00640 1 -- 172.21.15.72:0/1617364764 >> 172.21.15.72:0/1617364764 conn(0x7fa42c0fff00 msgr2=0x7fa42c102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:16.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.702+0000 7fa430c00640 1 -- 172.21.15.72:0/1617364764 shutdown_connections 2024-04-27T00:01:16.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.702+0000 7fa430c00640 1 -- 172.21.15.72:0/1617364764 wait complete. 2024-04-27T00:01:16.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.703+0000 7fa430c00640 1 Processor -- start 2024-04-27T00:01:16.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.703+0000 7fa430c00640 1 -- start start 2024-04-27T00:01:16.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.704+0000 7fa430c00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa42c104990 0x7fa42c19ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:16.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.704+0000 7fa430c00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa42c105b80 0x7fa42c19b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:16.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.704+0000 7fa430c00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa42c106540 0x7fa42c1a2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:16.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.704+0000 7fa430c00640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fa42c1104c0 con 0x7fa42c106540 2024-04-27T00:01:16.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.704+0000 7fa430c00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fa42c110340 con 0x7fa42c105b80 2024-04-27T00:01:16.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.704+0000 7fa430c00640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fa42c110640 con 0x7fa42c104990 2024-04-27T00:01:16.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.704+0000 7fa42be00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa42c106540 0x7fa42c1a2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:16.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.704+0000 7fa42b400640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa42c104990 0x7fa42c19ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:16.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.704+0000 7fa42aa00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa42c105b80 0x7fa42c19b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:16.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.704+0000 7fa42be00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa42c106540 0x7fa42c1a2140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54054/0 (socket says 172.21.15.72:54054) 2024-04-27T00:01:16.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.704+0000 7fa42b400640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa42c104990 0x7fa42c19ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.72:51378/0 (socket says 172.21.15.72:51378) 2024-04-27T00:01:16.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.704+0000 7fa42be00640 1 -- 172.21.15.72:0/735079168 learned_addr learned my addr 172.21.15.72:0/735079168 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-27T00:01:16.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.705+0000 7fa42b400640 1 -- 172.21.15.72:0/735079168 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa42c105b80 msgr2=0x7fa42c19b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:16.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.705+0000 7fa42b400640 1 --2- 172.21.15.72:0/735079168 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa42c105b80 0x7fa42c19b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:16.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.705+0000 7fa42b400640 1 -- 172.21.15.72:0/735079168 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa42c106540 msgr2=0x7fa42c1a2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:16.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.705+0000 7fa42b400640 1 --2- 172.21.15.72:0/735079168 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa42c106540 0x7fa42c1a2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:16.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.705+0000 7fa42b400640 1 -- 172.21.15.72:0/735079168 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa42c1a27b0 con 0x7fa42c104990 2024-04-27T00:01:16.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.705+0000 7fa42b400640 1 --2- 172.21.15.72:0/735079168 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa42c104990 0x7fa42c19ab90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa41c00e9c0 tx=0x7fa41c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:16.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.705+0000 7fa423e00640 1 -- 172.21.15.72:0/735079168 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa41c00ccc0 con 0x7fa42c104990 2024-04-27T00:01:16.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.706+0000 7fa430c00640 1 -- 172.21.15.72:0/735079168 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa42c1a2aa0 con 0x7fa42c104990 2024-04-27T00:01:16.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.706+0000 7fa423e00640 1 -- 172.21.15.72:0/735079168 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa41c0035f0 con 0x7fa42c104990 2024-04-27T00:01:16.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.706+0000 7fa423e00640 1 -- 172.21.15.72:0/735079168 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa41c00ce60 con 0x7fa42c104990 2024-04-27T00:01:16.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.706+0000 7fa430c00640 1 -- 172.21.15.72:0/735079168 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa42c1a2ff0 con 0x7fa42c104990 2024-04-27T00:01:16.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.707+0000 7fa430c00640 1 -- 172.21.15.72:0/735079168 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa42c10bdd0 con 0x7fa42c104990 2024-04-27T00:01:16.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.709+0000 7fa423e00640 1 -- 172.21.15.72:0/735079168 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fa41c004000 con 0x7fa42c104990 2024-04-27T00:01:16.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.710+0000 7fa423e00640 1 --2- 172.21.15.72:0/735079168 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fa40c0a3c40 0x7fa40c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:16.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.710+0000 7fa423e00640 1 -- 172.21.15.72:0/735079168 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7fa41c0d0d50 con 0x7fa42c104990 2024-04-27T00:01:16.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.710+0000 7fa42aa00640 1 --2- 172.21.15.72:0/735079168 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fa40c0a3c40 0x7fa40c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:16.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.711+0000 7fa42aa00640 1 --2- 172.21.15.72:0/735079168 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fa40c0a3c40 0x7fa40c0a6100 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fa42c19c0a0 tx=0x7fa414009360 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:16.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.715+0000 7fa423e00640 1 -- 172.21.15.72:0/735079168 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fa41c09d180 con 0x7fa42c104990 2024-04-27T00:01:16.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.919+0000 7fa430c00640 1 -- 172.21.15.72:0/735079168 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa42c02dce0 con 0x7fa42c104990 2024-04-27T00:01:16.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.922+0000 7fa423e00640 1 -- 172.21.15.72:0/735079168 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24841 (secure 0 0 0) 0x7fa41c09ca80 con 0x7fa42c104990 2024-04-27T00:01:16.924 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-27T00:01:16.924 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":93,"fsid":"3c8453ae-0425-11ef-bc93-c7b262605968","created":"2024-04-26T23:33:48.088300+0000","modified":"2024-04-26T23:45:00.532006+0000","last_up_change":"2024-04-26T23:42:04.482891+0000","last_in_change":"2024-04-26T23:41:44.784415+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:37:44.428727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:43:00.431829+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:43:01.872977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:43:05.512050+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e20c9ebf-56f6-47c1-a11b-e1e615af042b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6803","nonce":1639298875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6805","nonce":1639298875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6809","nonce":1639298875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6807","nonce":1639298875}]},"public_addr":"172.21.15.72:6803/1639298875","cluster_addr":"172.21.15.72:6805/1639298875","heartbeat_back_addr":"172.21.15.72:6809/1639298875","heartbeat_front_addr":"172.21.15.72:6807/1639298875","state":["exists","up"]},{"osd":1,"uuid":"e6cac469-780c-4d66-b560-b686072371fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6801","nonce":4185342193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6803","nonce":4185342193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6807","nonce":4185342193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6805","nonce":4185342193}]},"public_addr":"172.21.15.76:6801/4185342193","cluster_addr":"172.21.15.76:6803/4185342193","heartbeat_back_addr":"172.21.15.76:6807/4185342193","heartbeat_front_addr":"172.21.15.76:6805/4185342193","state":["exists","up"]},{"osd":2,"uuid":"838a49fc-18de-446c-8336-67b3e05da231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6801","nonce":3033860799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6803","nonce":3033860799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6807","nonce":3033860799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6805","nonce":3033860799}]},"public_addr":"172.21.15.136:6801/3033860799","cluster_addr":"172.21.15.136:6803/3033860799","heartbeat_back_addr":"172.21.15.136:6807/3033860799","heartbeat_front_addr":"172.21.15.136:6805/3033860799","state":["exists","up"]},{"osd":3,"uuid":"e6da89be-5ba7-48a4-9306-f75498660e0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6811","nonce":2071107815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6813","nonce":2071107815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6817","nonce":2071107815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6815","nonce":2071107815}]},"public_addr":"172.21.15.72:6811/2071107815","cluster_addr":"172.21.15.72:6813/2071107815","heartbeat_back_addr":"172.21.15.72:6817/2071107815","heartbeat_front_addr":"172.21.15.72:6815/2071107815","state":["exists","up"]},{"osd":4,"uuid":"0b17c491-d798-488f-892d-521ef2fbc51e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6809","nonce":147449238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6811","nonce":147449238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6815","nonce":147449238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6813","nonce":147449238}]},"public_addr":"172.21.15.76:6809/147449238","cluster_addr":"172.21.15.76:6811/147449238","heartbeat_back_addr":"172.21.15.76:6815/147449238","heartbeat_front_addr":"172.21.15.76:6813/147449238","state":["exists","up"]},{"osd":5,"uuid":"c7e887b3-ab38-4608-9bfd-1ed49956e8af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6809","nonce":2320149212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6811","nonce":2320149212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6815","nonce":2320149212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6813","nonce":2320149212}]},"public_addr":"172.21.15.136:6809/2320149212","cluster_addr":"172.21.15.136:6811/2320149212","heartbeat_back_addr":"172.21.15.136:6815/2320149212","heartbeat_front_addr":"172.21.15.136:6813/2320149212","state":["exists","up"]},{"osd":6,"uuid":"f0bd604b-0f57-4a38-9295-eb200264f1ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6819","nonce":389499207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6821","nonce":389499207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6825","nonce":389499207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6823","nonce":389499207}]},"public_addr":"172.21.15.72:6819/389499207","cluster_addr":"172.21.15.72:6821/389499207","heartbeat_back_addr":"172.21.15.72:6825/389499207","heartbeat_front_addr":"172.21.15.72:6823/389499207","state":["exists","up"]},{"osd":7,"uuid":"1879d4e4-038e-4cd6-a3ec-3a940bcc3797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6817","nonce":375888575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6819","nonce":375888575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6823","nonce":375888575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6821","nonce":375888575}]},"public_addr":"172.21.15.76:6817/375888575","cluster_addr":"172.21.15.76:6819/375888575","heartbeat_back_addr":"172.21.15.76:6823/375888575","heartbeat_front_addr":"172.21.15.76:6821/375888575","state":["exists","up"]},{"osd":8,"uuid":"6f404062-97e7-4ccb-a582-76dc1ffae548","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6817","nonce":3446774052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6819","nonce":3446774052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6823","nonce":3446774052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6821","nonce":3446774052}]},"public_addr":"172.21.15.136:6817/3446774052","cluster_addr":"172.21.15.136:6819/3446774052","heartbeat_back_addr":"172.21.15.136:6823/3446774052","heartbeat_front_addr":"172.21.15.136:6821/3446774052","state":["exists","up"]},{"osd":9,"uuid":"2989ca68-3164-4667-9bcf-230dcbb7b8f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6827","nonce":3423789152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6829","nonce":3423789152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6832","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6833","nonce":3423789152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6831","nonce":3423789152}]},"public_addr":"172.21.15.72:6827/3423789152","cluster_addr":"172.21.15.72:6829/3423789152","heartbeat_back_addr":"172.21.15.72:6833/3423789152","heartbeat_front_addr":"172.21.15.72:6831/3423789152","state":["exists","up"]},{"osd":10,"uuid":"9ec6d46a-0913-4208-97b5-167188ee8cb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6825","nonce":1120991574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6827","nonce":1120991574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6831","nonce":1120991574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6829","nonce":1120991574}]},"public_addr":"172.21.15.76:6825/1120991574","cluster_addr":"172.21.15.76:6827/1120991574","heartbeat_back_addr":"172.21.15.76:6831/1120991574","heartbeat_front_addr":"172.21.15.76:6829/1120991574","state":["exists","up"]},{"osd":11,"uuid":"bdd0774f-6e58-4918-b4cc-afd4f1118d3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6825","nonce":1135995193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6827","nonce":1135995193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6831","nonce":1135995193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6829","nonce":1135995193}]},"public_addr":"172.21.15.136:6825/1135995193","cluster_addr":"172.21.15.136:6827/1135995193","heartbeat_back_addr":"172.21.15.136:6831/1135995193","heartbeat_front_addr":"172.21.15.136:6829/1135995193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:36:52.128788+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:37:17.523654+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:37:41.634677+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:38:15.060797+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:38:41.234111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:39:06.783560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:39:40.884386+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:40:08.340465+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:40:34.344858+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:41:08.525309+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:41:36.782624+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:42:02.756888+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":6}]},{"pgid":"3.37","mappings":[{"from":5,"to":7}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:0/3625071330":"2024-04-27T23:34:58.266218+0000","172.21.15.72:6801/2903979134":"2024-04-27T23:34:58.266218+0000","172.21.15.72:6800/2903979134":"2024-04-27T23:34:58.266218+0000","172.21.15.72:0/1519142576":"2024-04-27T23:34:58.266218+0000","172.21.15.72:0/3669626460":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/945623297":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/3479150073":"2024-04-27T23:34:24.233319+0000","172.21.15.72:6800/4175896290":"2024-04-27T23:34:24.233319+0000","172.21.15.72:6801/4175896290":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/3783175012":"2024-04-27T23:34:58.266218+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-27T00:01:16.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.924+0000 7fa430c00640 1 -- 172.21.15.72:0/735079168 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fa40c0a3c40 msgr2=0x7fa40c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:16.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.924+0000 7fa430c00640 1 --2- 172.21.15.72:0/735079168 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fa40c0a3c40 0x7fa40c0a6100 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fa42c19c0a0 tx=0x7fa414009360 comp rx=0 tx=0).stop 2024-04-27T00:01:16.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.925+0000 7fa430c00640 1 -- 172.21.15.72:0/735079168 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa42c104990 msgr2=0x7fa42c19ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:16.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.925+0000 7fa430c00640 1 --2- 172.21.15.72:0/735079168 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa42c104990 0x7fa42c19ab90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa41c00e9c0 tx=0x7fa41c00ee80 comp rx=0 tx=0).stop 2024-04-27T00:01:16.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.925+0000 7fa430c00640 1 -- 172.21.15.72:0/735079168 shutdown_connections 2024-04-27T00:01:16.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.925+0000 7fa430c00640 1 --2- 172.21.15.72:0/735079168 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fa40c0a3c40 0x7fa40c0a6100 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:16.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.925+0000 7fa430c00640 1 --2- 172.21.15.72:0/735079168 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa42c105b80 0x7fa42c19b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:16.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.925+0000 7fa430c00640 1 --2- 172.21.15.72:0/735079168 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa42c106540 0x7fa42c1a2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:16.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.925+0000 7fa430c00640 1 --2- 172.21.15.72:0/735079168 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa42c104990 0x7fa42c19ab90 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:16.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.925+0000 7fa430c00640 1 -- 172.21.15.72:0/735079168 >> 172.21.15.72:0/735079168 conn(0x7fa42c0fff00 msgr2=0x7fa42c108fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:16.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.925+0000 7fa430c00640 1 -- 172.21.15.72:0/735079168 shutdown_connections 2024-04-27T00:01:16.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:16.925+0000 7fa430c00640 1 -- 172.21.15.72:0/735079168 wait complete. 2024-04-27T00:01:16.942 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:16 smithi072 ceph-mon[28184]: pgmap v871: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:01:16.945 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T23:37:44.428727+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T23:43:00.431829+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T23:43:01.872977+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T23:43:05.512050+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-27T00:01:16.946 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-27T00:01:17.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:16 smithi136 ceph-mon[33050]: pgmap v871: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:01:17.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.153+0000 7f000a600640 1 -- 172.21.15.72:0/3190392562 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f000410a180 msgr2=0x7f000410a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:17.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.153+0000 7f000a600640 1 --2- 172.21.15.72:0/3190392562 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f000410a180 0x7f000410a600 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7effec009900 tx=0x7effec02a3f0 comp rx=0 tx=0).stop 2024-04-27T00:01:17.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.153+0000 7f000a600640 1 -- 172.21.15.72:0/3190392562 shutdown_connections 2024-04-27T00:01:17.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.153+0000 7f000a600640 1 --2- 172.21.15.72:0/3190392562 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f000410a180 0x7f000410a600 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:17.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.153+0000 7f000a600640 1 --2- 172.21.15.72:0/3190392562 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f000410ab40 0x7f00041113d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:17.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.153+0000 7f000a600640 1 --2- 172.21.15.72:0/3190392562 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0004108f90 0x7f00041093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:17.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.153+0000 7f000a600640 1 -- 172.21.15.72:0/3190392562 >> 172.21.15.72:0/3190392562 conn(0x7f0004077f30 msgr2=0x7f000407a370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:17.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.153+0000 7f000a600640 1 -- 172.21.15.72:0/3190392562 shutdown_connections 2024-04-27T00:01:17.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.153+0000 7f000a600640 1 -- 172.21.15.72:0/3190392562 wait complete. 2024-04-27T00:01:17.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.154+0000 7f000a600640 1 Processor -- start 2024-04-27T00:01:17.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.154+0000 7f000a600640 1 -- start start 2024-04-27T00:01:17.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.154+0000 7f000a600640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0004108f90 0x7f000419f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:17.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.154+0000 7f000a600640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f000410a180 0x7f000419f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:17.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.154+0000 7f000a600640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f000410ab40 0x7f00041a6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:17.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.154+0000 7f000a600640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f0004114ad0 con 0x7f000410ab40 2024-04-27T00:01:17.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.154+0000 7f000a600640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f0004114950 con 0x7f0004108f90 2024-04-27T00:01:17.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.154+0000 7f000a600640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f0004114c50 con 0x7f000410a180 2024-04-27T00:01:17.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.155+0000 7f0003e00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f000410ab40 0x7f00041a6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:17.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.155+0000 7f0003e00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f000410ab40 0x7f00041a6750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43214/0 (socket says 172.21.15.72:43214) 2024-04-27T00:01:17.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.155+0000 7f0003400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0004108f90 0x7f000419f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:17.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.155+0000 7f0003e00640 1 -- 172.21.15.72:0/1489934784 learned_addr learned my addr 172.21.15.72:0/1489934784 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-27T00:01:17.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.155+0000 7efffaa00640 1 --2- 172.21.15.72:0/1489934784 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f000410a180 0x7f000419f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:17.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.155+0000 7f0003e00640 1 -- 172.21.15.72:0/1489934784 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f000410a180 msgr2=0x7f000419f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:17.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.155+0000 7f0003e00640 1 --2- 172.21.15.72:0/1489934784 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f000410a180 0x7f000419f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:17.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.155+0000 7f0003e00640 1 -- 172.21.15.72:0/1489934784 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0004108f90 msgr2=0x7f000419f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:17.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.155+0000 7f0003e00640 1 --2- 172.21.15.72:0/1489934784 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0004108f90 0x7f000419f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:17.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.155+0000 7f0003e00640 1 -- 172.21.15.72:0/1489934784 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00041a6dc0 con 0x7f000410ab40 2024-04-27T00:01:17.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.155+0000 7efffaa00640 1 --2- 172.21.15.72:0/1489934784 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f000410a180 0x7f000419f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:01:17.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.155+0000 7f0003400640 1 --2- 172.21.15.72:0/1489934784 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0004108f90 0x7f000419f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:01:17.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.155+0000 7f0003e00640 1 --2- 172.21.15.72:0/1489934784 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f000410ab40 0x7f00041a6750 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7efffc00eb10 tx=0x7efffc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:17.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.156+0000 7f0000c00640 1 -- 172.21.15.72:0/1489934784 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efffc004b20 con 0x7f000410ab40 2024-04-27T00:01:17.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.156+0000 7f0000c00640 1 -- 172.21.15.72:0/1489934784 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efffc004cc0 con 0x7f000410ab40 2024-04-27T00:01:17.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.156+0000 7f000a600640 1 -- 172.21.15.72:0/1489934784 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00041a70b0 con 0x7f000410ab40 2024-04-27T00:01:17.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.156+0000 7f0000c00640 1 -- 172.21.15.72:0/1489934784 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efffc010670 con 0x7f000410ab40 2024-04-27T00:01:17.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.156+0000 7f000a600640 1 -- 172.21.15.72:0/1489934784 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00041a75b0 con 0x7f000410ab40 2024-04-27T00:01:17.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.157+0000 7f000a600640 1 -- 172.21.15.72:0/1489934784 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00041102d0 con 0x7f000410ab40 2024-04-27T00:01:17.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.159+0000 7f0000c00640 1 -- 172.21.15.72:0/1489934784 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7efffc005020 con 0x7f000410ab40 2024-04-27T00:01:17.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.160+0000 7f0000c00640 1 --2- 172.21.15.72:0/1489934784 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7effe80a3c40 0x7effe80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:17.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.160+0000 7f0000c00640 1 -- 172.21.15.72:0/1489934784 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7efffc0d0fe0 con 0x7f000410ab40 2024-04-27T00:01:17.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.161+0000 7f0003400640 1 --2- 172.21.15.72:0/1489934784 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7effe80a3c40 0x7effe80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:17.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.161+0000 7f0003400640 1 --2- 172.21.15.72:0/1489934784 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7effe80a3c40 0x7effe80a6100 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f0004109fe0 tx=0x7efff4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:17.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.165+0000 7f0000c00640 1 -- 172.21.15.72:0/1489934784 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7efffc09d470 con 0x7f000410ab40 2024-04-27T00:01:17.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:16 smithi076 ceph-mon[32774]: pgmap v871: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:01:17.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.351+0000 7f000a600640 1 -- 172.21.15.72:0/1489934784 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0004063530 con 0x7f000410ab40 2024-04-27T00:01:17.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.351+0000 7f0000c00640 1 -- 172.21.15.72:0/1489934784 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7efffc09cd70 con 0x7f000410ab40 2024-04-27T00:01:17.353 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-04-27T00:01:17.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.354+0000 7f000a600640 1 -- 172.21.15.72:0/1489934784 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7effe80a3c40 msgr2=0x7effe80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:17.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.354+0000 7f000a600640 1 --2- 172.21.15.72:0/1489934784 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7effe80a3c40 0x7effe80a6100 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f0004109fe0 tx=0x7efff4005f90 comp rx=0 tx=0).stop 2024-04-27T00:01:17.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.354+0000 7f000a600640 1 -- 172.21.15.72:0/1489934784 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f000410ab40 msgr2=0x7f00041a6750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:17.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.354+0000 7f000a600640 1 --2- 172.21.15.72:0/1489934784 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f000410ab40 0x7f00041a6750 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7efffc00eb10 tx=0x7efffc00c2d0 comp rx=0 tx=0).stop 2024-04-27T00:01:17.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.354+0000 7f000a600640 1 -- 172.21.15.72:0/1489934784 shutdown_connections 2024-04-27T00:01:17.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.354+0000 7f000a600640 1 --2- 172.21.15.72:0/1489934784 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7effe80a3c40 0x7effe80a6100 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:17.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.354+0000 7f000a600640 1 --2- 172.21.15.72:0/1489934784 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f000410ab40 0x7f00041a6750 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:17.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.354+0000 7f000a600640 1 --2- 172.21.15.72:0/1489934784 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f000410a180 0x7f000419f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:17.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.354+0000 7f000a600640 1 --2- 172.21.15.72:0/1489934784 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0004108f90 0x7f000419f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:17.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.354+0000 7f000a600640 1 -- 172.21.15.72:0/1489934784 >> 172.21.15.72:0/1489934784 conn(0x7f0004077f30 msgr2=0x7f000410d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:17.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.354+0000 7f000a600640 1 -- 172.21.15.72:0/1489934784 shutdown_connections 2024-04-27T00:01:17.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.354+0000 7f000a600640 1 -- 172.21.15.72:0/1489934784 wait complete. 2024-04-27T00:01:17.373 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-27T00:01:17.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.566+0000 7f08b6800640 1 -- 172.21.15.72:0/424577568 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08b0102930 msgr2=0x7f08b0102db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:17.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.566+0000 7f08b6800640 1 --2- 172.21.15.72:0/424577568 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08b0102930 0x7f08b0102db0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f089c009900 tx=0x7f089c02a400 comp rx=0 tx=0).stop 2024-04-27T00:01:17.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.567+0000 7f08b6800640 1 -- 172.21.15.72:0/424577568 shutdown_connections 2024-04-27T00:01:17.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.567+0000 7f08b6800640 1 --2- 172.21.15.72:0/424577568 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f08b01032f0 0x7f08b0109b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:17.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.567+0000 7f08b6800640 1 --2- 172.21.15.72:0/424577568 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08b0102930 0x7f08b0102db0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:17.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.567+0000 7f08b6800640 1 --2- 172.21.15.72:0/424577568 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f08b0101740 0x7f08b0101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:17.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.567+0000 7f08b6800640 1 -- 172.21.15.72:0/424577568 >> 172.21.15.72:0/424577568 conn(0x7f08b00fccd0 msgr2=0x7f08b00ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:17.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.567+0000 7f08b6800640 1 -- 172.21.15.72:0/424577568 shutdown_connections 2024-04-27T00:01:17.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.567+0000 7f08b6800640 1 -- 172.21.15.72:0/424577568 wait complete. 2024-04-27T00:01:17.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.568+0000 7f08b6800640 1 Processor -- start 2024-04-27T00:01:17.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.568+0000 7f08b6800640 1 -- start start 2024-04-27T00:01:17.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.568+0000 7f08b6800640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f08b0101740 0x7f08b019ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:17.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.568+0000 7f08b6800640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f08b0102930 0x7f08b019b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:17.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.568+0000 7f08b6800640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08b01032f0 0x7f08b01a21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:17.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.568+0000 7f08b6800640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f08b010d2c0 con 0x7f08b0102930 2024-04-27T00:01:17.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.568+0000 7f08b6800640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f08b010d140 con 0x7f08b0101740 2024-04-27T00:01:17.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.568+0000 7f08b6800640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f08b010d440 con 0x7f08b01032f0 2024-04-27T00:01:17.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.569+0000 7f08b4a00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f08b0102930 0x7f08b019b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:17.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.569+0000 7f08b5400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f08b0101740 0x7f08b019ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:17.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.569+0000 7f08b4a00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f08b0102930 0x7f08b019b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43238/0 (socket says 172.21.15.72:43238) 2024-04-27T00:01:17.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.569+0000 7f08b4a00640 1 -- 172.21.15.72:0/3241348274 learned_addr learned my addr 172.21.15.72:0/3241348274 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-27T00:01:17.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.569+0000 7f08b5e00640 1 --2- 172.21.15.72:0/3241348274 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08b01032f0 0x7f08b01a21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:17.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.569+0000 7f08b4a00640 1 -- 172.21.15.72:0/3241348274 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08b01032f0 msgr2=0x7f08b01a21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:17.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.569+0000 7f08b4a00640 1 --2- 172.21.15.72:0/3241348274 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08b01032f0 0x7f08b01a21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:17.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.569+0000 7f08b4a00640 1 -- 172.21.15.72:0/3241348274 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f08b0101740 msgr2=0x7f08b019ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:17.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.569+0000 7f08b4a00640 1 --2- 172.21.15.72:0/3241348274 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f08b0101740 0x7f08b019ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:17.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.569+0000 7f08b4a00640 1 -- 172.21.15.72:0/3241348274 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08b01a2880 con 0x7f08b0102930 2024-04-27T00:01:17.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.569+0000 7f08b4a00640 1 --2- 172.21.15.72:0/3241348274 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f08b0102930 0x7f08b019b220 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f089c005ab0 tx=0x7f089c003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:17.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.570+0000 7f08ae000640 1 -- 172.21.15.72:0/3241348274 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f089c03f070 con 0x7f08b0102930 2024-04-27T00:01:17.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.570+0000 7f08ae000640 1 -- 172.21.15.72:0/3241348274 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f089c031c50 con 0x7f08b0102930 2024-04-27T00:01:17.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.570+0000 7f08ae000640 1 -- 172.21.15.72:0/3241348274 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f089c035030 con 0x7f08b0102930 2024-04-27T00:01:17.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.570+0000 7f08b6800640 1 -- 172.21.15.72:0/3241348274 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08b01a2b10 con 0x7f08b0102930 2024-04-27T00:01:17.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.570+0000 7f08b6800640 1 -- 172.21.15.72:0/3241348274 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08b01a2fb0 con 0x7f08b0102930 2024-04-27T00:01:17.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.571+0000 7f08b6800640 1 -- 172.21.15.72:0/3241348274 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08b01a3180 con 0x7f08b0102930 2024-04-27T00:01:17.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.573+0000 7f08ae000640 1 -- 172.21.15.72:0/3241348274 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f089c042070 con 0x7f08b0102930 2024-04-27T00:01:17.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.575+0000 7f08ae000640 1 --2- 172.21.15.72:0/3241348274 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f08980a3c40 0x7f08980a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:17.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.576+0000 7f08ae000640 1 -- 172.21.15.72:0/3241348274 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f089c0f11b0 con 0x7f08b0102930 2024-04-27T00:01:17.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.576+0000 7f08b5400640 1 --2- 172.21.15.72:0/3241348274 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f08980a3c40 0x7f08980a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:17.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.576+0000 7f08b5400640 1 --2- 172.21.15.72:0/3241348274 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f08980a3c40 0x7f08980a6100 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f08b019c140 tx=0x7f08a4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:17.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.580+0000 7f08ae000640 1 -- 172.21.15.72:0/3241348274 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f089c0bcf90 con 0x7f08b0102930 2024-04-27T00:01:17.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.772+0000 7f08b6800640 1 -- 172.21.15.72:0/3241348274 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f08b01a3510 con 0x7f08b0102930 2024-04-27T00:01:17.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.773+0000 7f08ae000640 1 -- 172.21.15.72:0/3241348274 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f089c0c1290 con 0x7f08b0102930 2024-04-27T00:01:17.775 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-27T00:01:17.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.776+0000 7f08b6800640 1 -- 172.21.15.72:0/3241348274 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f08980a3c40 msgr2=0x7f08980a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:17.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.776+0000 7f08b6800640 1 --2- 172.21.15.72:0/3241348274 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f08980a3c40 0x7f08980a6100 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f08b019c140 tx=0x7f08a4005f90 comp rx=0 tx=0).stop 2024-04-27T00:01:17.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.776+0000 7f08b6800640 1 -- 172.21.15.72:0/3241348274 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f08b0102930 msgr2=0x7f08b019b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:17.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.776+0000 7f08b6800640 1 --2- 172.21.15.72:0/3241348274 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f08b0102930 0x7f08b019b220 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f089c005ab0 tx=0x7f089c003b90 comp rx=0 tx=0).stop 2024-04-27T00:01:17.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.776+0000 7f08b6800640 1 -- 172.21.15.72:0/3241348274 shutdown_connections 2024-04-27T00:01:17.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.776+0000 7f08b6800640 1 --2- 172.21.15.72:0/3241348274 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f08980a3c40 0x7f08980a6100 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:17.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.776+0000 7f08b6800640 1 --2- 172.21.15.72:0/3241348274 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08b01032f0 0x7f08b01a21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:17.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.776+0000 7f08b6800640 1 --2- 172.21.15.72:0/3241348274 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f08b0102930 0x7f08b019b220 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:17.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.776+0000 7f08b6800640 1 --2- 172.21.15.72:0/3241348274 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f08b0101740 0x7f08b019ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:17.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.776+0000 7f08b6800640 1 -- 172.21.15.72:0/3241348274 >> 172.21.15.72:0/3241348274 conn(0x7f08b00fccd0 msgr2=0x7f08b00fe780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:17.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.776+0000 7f08b6800640 1 -- 172.21.15.72:0/3241348274 shutdown_connections 2024-04-27T00:01:17.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.776+0000 7f08b6800640 1 -- 172.21.15.72:0/3241348274 wait complete. 2024-04-27T00:01:17.796 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-27T00:01:17.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.989+0000 7f3f9be00640 1 -- 172.21.15.72:0/487205003 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f9c105b00 msgr2=0x7f3f9c107ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:17.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.989+0000 7f3f9be00640 1 --2- 172.21.15.72:0/487205003 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f9c105b00 0x7f3f9c107ef0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f3f84009900 tx=0x7f3f8402a400 comp rx=0 tx=0).stop 2024-04-27T00:01:17.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.989+0000 7f3f9be00640 1 -- 172.21.15.72:0/487205003 shutdown_connections 2024-04-27T00:01:17.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.989+0000 7f3f9be00640 1 --2- 172.21.15.72:0/487205003 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f9c108430 0x7f3f9c10a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:17.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.989+0000 7f3f9be00640 1 --2- 172.21.15.72:0/487205003 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f9c105b00 0x7f3f9c107ef0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:17.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.989+0000 7f3f9be00640 1 --2- 172.21.15.72:0/487205003 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3f9c1031d0 0x7f3f9c1055c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:17.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.989+0000 7f3f9be00640 1 -- 172.21.15.72:0/487205003 >> 172.21.15.72:0/487205003 conn(0x7f3f9c0fcbe0 msgr2=0x7f3f9c0ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:17.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.989+0000 7f3f9be00640 1 -- 172.21.15.72:0/487205003 shutdown_connections 2024-04-27T00:01:17.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.989+0000 7f3f9be00640 1 -- 172.21.15.72:0/487205003 wait complete. 2024-04-27T00:01:17.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.990+0000 7f3f9be00640 1 Processor -- start 2024-04-27T00:01:17.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.990+0000 7f3f9be00640 1 -- start start 2024-04-27T00:01:17.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.990+0000 7f3f9be00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f9c1031d0 0x7f3f9c19abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:17.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.990+0000 7f3f9be00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f9c105b00 0x7f3f9c19b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:17.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.990+0000 7f3f9be00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3f9c108430 0x7f3f9c1a2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:17.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.990+0000 7f3f9be00640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f3f9c10e2c0 con 0x7f3f9c1031d0 2024-04-27T00:01:17.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.990+0000 7f3f9be00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f3f9c10e140 con 0x7f3f9c105b00 2024-04-27T00:01:17.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.990+0000 7f3f9be00640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f3f9c10e440 con 0x7f3f9c108430 2024-04-27T00:01:17.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.990+0000 7f3f9aa00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f9c1031d0 0x7f3f9c19abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:17.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.991+0000 7f3f9b400640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3f9c108430 0x7f3f9c1a2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:17.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.991+0000 7f3f9a000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f9c105b00 0x7f3f9c19b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:17.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.991+0000 7f3f9b400640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3f9c108430 0x7f3f9c1a2160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.72:47090/0 (socket says 172.21.15.72:47090) 2024-04-27T00:01:17.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.991+0000 7f3f9aa00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f9c1031d0 0x7f3f9c19abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43258/0 (socket says 172.21.15.72:43258) 2024-04-27T00:01:17.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.991+0000 7f3f9a000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f9c105b00 0x7f3f9c19b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:57322/0 (socket says 172.21.15.72:57322) 2024-04-27T00:01:17.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.991+0000 7f3f9b400640 1 -- 172.21.15.72:0/3705027638 learned_addr learned my addr 172.21.15.72:0/3705027638 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-27T00:01:17.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.991+0000 7f3f9aa00640 1 -- 172.21.15.72:0/3705027638 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3f9c108430 msgr2=0x7f3f9c1a2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:17.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.991+0000 7f3f9aa00640 1 --2- 172.21.15.72:0/3705027638 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3f9c108430 0x7f3f9c1a2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:17.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.991+0000 7f3f9aa00640 1 -- 172.21.15.72:0/3705027638 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f9c105b00 msgr2=0x7f3f9c19b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:17.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.991+0000 7f3f9aa00640 1 --2- 172.21.15.72:0/3705027638 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f9c105b00 0x7f3f9c19b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:17.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.991+0000 7f3f9aa00640 1 -- 172.21.15.72:0/3705027638 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f9c1a27d0 con 0x7f3f9c1031d0 2024-04-27T00:01:17.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.991+0000 7f3f9a000640 1 --2- 172.21.15.72:0/3705027638 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f9c105b00 0x7f3f9c19b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:01:17.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.991+0000 7f3f9aa00640 1 --2- 172.21.15.72:0/3705027638 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f9c1031d0 0x7f3f9c19abb0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c00d6d0 tx=0x7f3f8c00db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:17.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.991+0000 7f3fa0a00640 1 -- 172.21.15.72:0/3705027638 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f8c017070 con 0x7f3f9c1031d0 2024-04-27T00:01:17.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.992+0000 7f3fa0a00640 1 -- 172.21.15.72:0/3705027638 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f8c009820 con 0x7f3f9c1031d0 2024-04-27T00:01:17.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.992+0000 7f3f9be00640 1 -- 172.21.15.72:0/3705027638 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f9c1a2ac0 con 0x7f3f9c1031d0 2024-04-27T00:01:17.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.992+0000 7f3fa0a00640 1 -- 172.21.15.72:0/3705027638 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f8c004c40 con 0x7f3f9c1031d0 2024-04-27T00:01:17.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.992+0000 7f3f9be00640 1 -- 172.21.15.72:0/3705027638 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f9c1a2fe0 con 0x7f3f9c1031d0 2024-04-27T00:01:17.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.993+0000 7f3f9be00640 1 -- 172.21.15.72:0/3705027638 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f9c109f80 con 0x7f3f9c1031d0 2024-04-27T00:01:17.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.995+0000 7f3fa0a00640 1 -- 172.21.15.72:0/3705027638 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3f8c005220 con 0x7f3f9c1031d0 2024-04-27T00:01:17.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.996+0000 7f3fa0a00640 1 --2- 172.21.15.72:0/3705027638 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f3f800a3c40 0x7f3f800a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:17.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.996+0000 7f3fa0a00640 1 -- 172.21.15.72:0/3705027638 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f3f8c0d0420 con 0x7f3f9c1031d0 2024-04-27T00:01:17.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.996+0000 7f3f9a000640 1 --2- 172.21.15.72:0/3705027638 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f3f800a3c40 0x7f3f800a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:17.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:17.997+0000 7f3f9a000640 1 --2- 172.21.15.72:0/3705027638 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f3f800a3c40 0x7f3f800a6100 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f3f84005ab0 tx=0x7f3f84005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:18.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.001+0000 7f3fa0a00640 1 -- 172.21.15.72:0/3705027638 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f3f8c09c200 con 0x7f3f9c1031d0 2024-04-27T00:01:18.040 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:17 smithi072 ceph-mon[28184]: from='client.25384 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T00:01:18.040 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:17 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/735079168' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T00:01:18.040 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:17 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/1489934784' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-27T00:01:18.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:17 smithi136 ceph-mon[33050]: from='client.25384 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T00:01:18.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:17 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/735079168' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T00:01:18.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:17 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/1489934784' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-27T00:01:18.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:17 smithi076 ceph-mon[32774]: from='client.25384 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-27T00:01:18.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:17 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/735079168' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T00:01:18.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:17 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/1489934784' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-27T00:01:18.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.188+0000 7f3f9be00640 1 -- 172.21.15.72:0/3705027638 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f3f9c063560 con 0x7f3f9c1031d0 2024-04-27T00:01:18.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.189+0000 7f3fa0a00640 1 -- 172.21.15.72:0/3705027638 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f3f8c0a0500 con 0x7f3f9c1031d0 2024-04-27T00:01:18.190 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-04-27T00:01:18.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.191+0000 7f3f9be00640 1 -- 172.21.15.72:0/3705027638 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f3f800a3c40 msgr2=0x7f3f800a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:18.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.192+0000 7f3f9be00640 1 --2- 172.21.15.72:0/3705027638 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f3f800a3c40 0x7f3f800a6100 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f3f84005ab0 tx=0x7f3f84005a20 comp rx=0 tx=0).stop 2024-04-27T00:01:18.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.192+0000 7f3f9be00640 1 -- 172.21.15.72:0/3705027638 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f9c1031d0 msgr2=0x7f3f9c19abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:18.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.192+0000 7f3f9be00640 1 --2- 172.21.15.72:0/3705027638 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f9c1031d0 0x7f3f9c19abb0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c00d6d0 tx=0x7f3f8c00db90 comp rx=0 tx=0).stop 2024-04-27T00:01:18.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.192+0000 7f3f9be00640 1 -- 172.21.15.72:0/3705027638 shutdown_connections 2024-04-27T00:01:18.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.192+0000 7f3f9be00640 1 --2- 172.21.15.72:0/3705027638 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f3f800a3c40 0x7f3f800a6100 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:18.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.192+0000 7f3f9be00640 1 --2- 172.21.15.72:0/3705027638 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3f9c105b00 0x7f3f9c19b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:18.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.192+0000 7f3f9be00640 1 --2- 172.21.15.72:0/3705027638 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3f9c108430 0x7f3f9c1a2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:18.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.192+0000 7f3f9be00640 1 --2- 172.21.15.72:0/3705027638 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f9c1031d0 0x7f3f9c19abb0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:18.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.192+0000 7f3f9be00640 1 -- 172.21.15.72:0/3705027638 >> 172.21.15.72:0/3705027638 conn(0x7f3f9c0fcbe0 msgr2=0x7f3f9c102940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:18.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.192+0000 7f3f9be00640 1 -- 172.21.15.72:0/3705027638 shutdown_connections 2024-04-27T00:01:18.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.192+0000 7f3f9be00640 1 -- 172.21.15.72:0/3705027638 wait complete. 2024-04-27T00:01:18.210 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-27T00:01:18.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.399+0000 7fbea8a00640 1 -- 172.21.15.72:0/2873847087 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbea4103320 msgr2=0x7fbea4105710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:18.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.399+0000 7fbea8a00640 1 --2- 172.21.15.72:0/2873847087 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbea4103320 0x7fbea4105710 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fbe94009900 tx=0x7fbe9402a3f0 comp rx=0 tx=0).stop 2024-04-27T00:01:18.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.399+0000 7fbea8a00640 1 -- 172.21.15.72:0/2873847087 shutdown_connections 2024-04-27T00:01:18.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.399+0000 7fbea8a00640 1 --2- 172.21.15.72:0/2873847087 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbea4105c50 0x7fbea4108040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:18.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.399+0000 7fbea8a00640 1 --2- 172.21.15.72:0/2873847087 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbea4108580 0x7fbea410a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:18.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.399+0000 7fbea8a00640 1 --2- 172.21.15.72:0/2873847087 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbea4103320 0x7fbea4105710 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:18.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.399+0000 7fbea8a00640 1 -- 172.21.15.72:0/2873847087 >> 172.21.15.72:0/2873847087 conn(0x7fbea40fccd0 msgr2=0x7fbea40ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:18.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.399+0000 7fbea8a00640 1 -- 172.21.15.72:0/2873847087 shutdown_connections 2024-04-27T00:01:18.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.399+0000 7fbea8a00640 1 -- 172.21.15.72:0/2873847087 wait complete. 2024-04-27T00:01:18.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.401+0000 7fbea8a00640 1 Processor -- start 2024-04-27T00:01:18.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.401+0000 7fbea8a00640 1 -- start start 2024-04-27T00:01:18.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.401+0000 7fbea8a00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbea4103320 0x7fbea4198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:18.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.401+0000 7fbea8a00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbea4105c50 0x7fbea4198fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:18.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.401+0000 7fbea8a00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbea4108580 0x7fbea41a0040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:18.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.401+0000 7fbea8a00640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fbea410e3b0 con 0x7fbea4103320 2024-04-27T00:01:18.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.401+0000 7fbea8a00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fbea410e230 con 0x7fbea4108580 2024-04-27T00:01:18.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.401+0000 7fbea8a00640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fbea410e530 con 0x7fbea4105c50 2024-04-27T00:01:18.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.401+0000 7fbea3400640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbea4103320 0x7fbea4198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:18.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.401+0000 7fbea3400640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbea4103320 0x7fbea4198a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43272/0 (socket says 172.21.15.72:43272) 2024-04-27T00:01:18.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.401+0000 7fbea3400640 1 -- 172.21.15.72:0/2195173958 learned_addr learned my addr 172.21.15.72:0/2195173958 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-27T00:01:18.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.401+0000 7fbe9aa00640 1 --2- 172.21.15.72:0/2195173958 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbea4105c50 0x7fbea4198fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:18.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.402+0000 7fbea3e00640 1 --2- 172.21.15.72:0/2195173958 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbea4108580 0x7fbea41a0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:18.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.402+0000 7fbea3400640 1 -- 172.21.15.72:0/2195173958 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbea4105c50 msgr2=0x7fbea4198fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:18.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.402+0000 7fbea3400640 1 --2- 172.21.15.72:0/2195173958 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbea4105c50 0x7fbea4198fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:18.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.402+0000 7fbea3400640 1 -- 172.21.15.72:0/2195173958 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbea4108580 msgr2=0x7fbea41a0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:18.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.402+0000 7fbea3400640 1 --2- 172.21.15.72:0/2195173958 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbea4108580 0x7fbea41a0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:18.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.402+0000 7fbea3400640 1 -- 172.21.15.72:0/2195173958 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbea41a06b0 con 0x7fbea4103320 2024-04-27T00:01:18.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.402+0000 7fbea3400640 1 --2- 172.21.15.72:0/2195173958 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbea4103320 0x7fbea4198a90 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fbe94005ab0 tx=0x7fbe94003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:18.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.402+0000 7fbea0c00640 1 -- 172.21.15.72:0/2195173958 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe9403f070 con 0x7fbea4103320 2024-04-27T00:01:18.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.402+0000 7fbea0c00640 1 -- 172.21.15.72:0/2195173958 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe94031c50 con 0x7fbea4103320 2024-04-27T00:01:18.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.403+0000 7fbea8a00640 1 -- 172.21.15.72:0/2195173958 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbea41a0940 con 0x7fbea4103320 2024-04-27T00:01:18.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.403+0000 7fbea0c00640 1 -- 172.21.15.72:0/2195173958 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe94035030 con 0x7fbea4103320 2024-04-27T00:01:18.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.403+0000 7fbea8a00640 1 -- 172.21.15.72:0/2195173958 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbea41a0e80 con 0x7fbea4103320 2024-04-27T00:01:18.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.404+0000 7fbea8a00640 1 -- 172.21.15.72:0/2195173958 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbea40fe7c0 con 0x7fbea4103320 2024-04-27T00:01:18.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.406+0000 7fbea0c00640 1 -- 172.21.15.72:0/2195173958 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fbe94042070 con 0x7fbea4103320 2024-04-27T00:01:18.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.407+0000 7fbea0c00640 1 --2- 172.21.15.72:0/2195173958 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fbe880a3cc0 0x7fbe880a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:18.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.407+0000 7fbea0c00640 1 -- 172.21.15.72:0/2195173958 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7fbe940f0fc0 con 0x7fbea4103320 2024-04-27T00:01:18.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.408+0000 7fbe9aa00640 1 --2- 172.21.15.72:0/2195173958 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fbe880a3cc0 0x7fbe880a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:18.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.408+0000 7fbe9aa00640 1 --2- 172.21.15.72:0/2195173958 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fbe880a3cc0 0x7fbe880a6180 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fbea4199fa0 tx=0x7fbe8c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:18.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.412+0000 7fbea0c00640 1 -- 172.21.15.72:0/2195173958 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fbe940bd450 con 0x7fbea4103320 2024-04-27T00:01:18.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.614+0000 7fbea8a00640 1 -- 172.21.15.72:0/2195173958 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fbea402dd20 con 0x7fbea4103320 2024-04-27T00:01:18.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.615+0000 7fbea0c00640 1 -- 172.21.15.72:0/2195173958 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7fbe940bcd50 con 0x7fbea4103320 2024-04-27T00:01:18.616 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-27T00:01:18.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.617+0000 7fbea8a00640 1 -- 172.21.15.72:0/2195173958 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fbe880a3cc0 msgr2=0x7fbe880a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:18.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.617+0000 7fbea8a00640 1 --2- 172.21.15.72:0/2195173958 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fbe880a3cc0 0x7fbe880a6180 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fbea4199fa0 tx=0x7fbe8c005f90 comp rx=0 tx=0).stop 2024-04-27T00:01:18.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.617+0000 7fbea8a00640 1 -- 172.21.15.72:0/2195173958 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbea4103320 msgr2=0x7fbea4198a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:18.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.617+0000 7fbea8a00640 1 --2- 172.21.15.72:0/2195173958 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbea4103320 0x7fbea4198a90 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fbe94005ab0 tx=0x7fbe94003b90 comp rx=0 tx=0).stop 2024-04-27T00:01:18.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.617+0000 7fbea8a00640 1 -- 172.21.15.72:0/2195173958 shutdown_connections 2024-04-27T00:01:18.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.617+0000 7fbea8a00640 1 --2- 172.21.15.72:0/2195173958 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fbe880a3cc0 0x7fbe880a6180 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:18.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.617+0000 7fbea8a00640 1 --2- 172.21.15.72:0/2195173958 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbea4108580 0x7fbea41a0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:18.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.617+0000 7fbea8a00640 1 --2- 172.21.15.72:0/2195173958 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbea4105c50 0x7fbea4198fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:18.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.617+0000 7fbea8a00640 1 --2- 172.21.15.72:0/2195173958 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbea4103320 0x7fbea4198a90 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:18.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.617+0000 7fbea8a00640 1 -- 172.21.15.72:0/2195173958 >> 172.21.15.72:0/2195173958 conn(0x7fbea40fccd0 msgr2=0x7fbea4103c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:18.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.618+0000 7fbea8a00640 1 -- 172.21.15.72:0/2195173958 shutdown_connections 2024-04-27T00:01:18.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.618+0000 7fbea8a00640 1 -- 172.21.15.72:0/2195173958 wait complete. 2024-04-27T00:01:18.636 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-27T00:01:18.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.822+0000 7fe8c2c00640 1 -- 172.21.15.72:0/808747733 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe8bc1043e0 msgr2=0x7fe8bc10ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:18.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.822+0000 7fe8c2c00640 1 --2- 172.21.15.72:0/808747733 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe8bc1043e0 0x7fe8bc10ac70 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe8b4009cd0 tx=0x7fe8b402a8f0 comp rx=0 tx=0).stop 2024-04-27T00:01:18.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.822+0000 7fe8c2c00640 1 -- 172.21.15.72:0/808747733 shutdown_connections 2024-04-27T00:01:18.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.822+0000 7fe8c2c00640 1 --2- 172.21.15.72:0/808747733 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe8bc1043e0 0x7fe8bc10ac70 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:18.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.822+0000 7fe8c2c00640 1 --2- 172.21.15.72:0/808747733 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8bc103a20 0x7fe8bc103ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:18.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.822+0000 7fe8c2c00640 1 --2- 172.21.15.72:0/808747733 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe8bc100a50 0x7fe8bc100e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:18.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.822+0000 7fe8c2c00640 1 -- 172.21.15.72:0/808747733 >> 172.21.15.72:0/808747733 conn(0x7fe8bc0fc380 msgr2=0x7fe8bc0fe7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:18.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.822+0000 7fe8c2c00640 1 -- 172.21.15.72:0/808747733 shutdown_connections 2024-04-27T00:01:18.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.822+0000 7fe8c2c00640 1 -- 172.21.15.72:0/808747733 wait complete. 2024-04-27T00:01:18.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.823+0000 7fe8c2c00640 1 Processor -- start 2024-04-27T00:01:18.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.823+0000 7fe8c2c00640 1 -- start start 2024-04-27T00:01:18.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.824+0000 7fe8c2c00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8bc100a50 0x7fe8bc198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:18.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.824+0000 7fe8c2c00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe8bc103a20 0x7fe8bc198e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:18.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.824+0000 7fe8c2c00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe8bc1043e0 0x7fe8bc19fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:18.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.824+0000 7fe8c2c00640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fe8bc10e300 con 0x7fe8bc100a50 2024-04-27T00:01:18.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.824+0000 7fe8c2c00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7fe8bc10e180 con 0x7fe8bc1043e0 2024-04-27T00:01:18.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.824+0000 7fe8c2c00640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7fe8bc10e480 con 0x7fe8bc103a20 2024-04-27T00:01:18.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.824+0000 7fe8c1800640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8bc100a50 0x7fe8bc198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:18.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.824+0000 7fe8c1800640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8bc100a50 0x7fe8bc198910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43280/0 (socket says 172.21.15.72:43280) 2024-04-27T00:01:18.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.824+0000 7fe8c1800640 1 -- 172.21.15.72:0/1170590162 learned_addr learned my addr 172.21.15.72:0/1170590162 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-27T00:01:18.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.824+0000 7fe8c0e00640 1 --2- 172.21.15.72:0/1170590162 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe8bc103a20 0x7fe8bc198e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:18.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.824+0000 7fe8c2200640 1 --2- 172.21.15.72:0/1170590162 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe8bc1043e0 0x7fe8bc19fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:18.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.824+0000 7fe8c1800640 1 -- 172.21.15.72:0/1170590162 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe8bc103a20 msgr2=0x7fe8bc198e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:18.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.824+0000 7fe8c1800640 1 --2- 172.21.15.72:0/1170590162 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe8bc103a20 0x7fe8bc198e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:18.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.824+0000 7fe8c1800640 1 -- 172.21.15.72:0/1170590162 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe8bc1043e0 msgr2=0x7fe8bc19fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:18.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.824+0000 7fe8c1800640 1 --2- 172.21.15.72:0/1170590162 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe8bc1043e0 0x7fe8bc19fec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:18.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.824+0000 7fe8c1800640 1 -- 172.21.15.72:0/1170590162 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8bc1a0530 con 0x7fe8bc100a50 2024-04-27T00:01:18.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.824+0000 7fe8c2200640 1 --2- 172.21.15.72:0/1170590162 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe8bc1043e0 0x7fe8bc19fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:01:18.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.825+0000 7fe8c1800640 1 --2- 172.21.15.72:0/1170590162 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8bc100a50 0x7fe8bc198910 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fe8b000d950 tx=0x7fe8b000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:18.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.825+0000 7fe8ba000640 1 -- 172.21.15.72:0/1170590162 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b0017020 con 0x7fe8bc100a50 2024-04-27T00:01:18.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.825+0000 7fe8ba000640 1 -- 172.21.15.72:0/1170590162 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8b0009c30 con 0x7fe8bc100a50 2024-04-27T00:01:18.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.825+0000 7fe8c2c00640 1 -- 172.21.15.72:0/1170590162 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8bc1a0820 con 0x7fe8bc100a50 2024-04-27T00:01:18.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.825+0000 7fe8ba000640 1 -- 172.21.15.72:0/1170590162 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b000f600 con 0x7fe8bc100a50 2024-04-27T00:01:18.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.825+0000 7fe8c2c00640 1 -- 172.21.15.72:0/1170590162 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8bc1a0d70 con 0x7fe8bc100a50 2024-04-27T00:01:18.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.826+0000 7fe8c2c00640 1 -- 172.21.15.72:0/1170590162 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8bc109b50 con 0x7fe8bc100a50 2024-04-27T00:01:18.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.828+0000 7fe8ba000640 1 -- 172.21.15.72:0/1170590162 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fe8b001a020 con 0x7fe8bc100a50 2024-04-27T00:01:18.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.829+0000 7fe8ba000640 1 --2- 172.21.15.72:0/1170590162 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fe8a40a3c40 0x7fe8a40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:18.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.830+0000 7fe8ba000640 1 -- 172.21.15.72:0/1170590162 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7fe8b00d0900 con 0x7fe8bc100a50 2024-04-27T00:01:18.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.830+0000 7fe8c0e00640 1 --2- 172.21.15.72:0/1170590162 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fe8a40a3c40 0x7fe8a40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:18.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.830+0000 7fe8c0e00640 1 --2- 172.21.15.72:0/1170590162 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fe8a40a3c40 0x7fe8a40a6100 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fe8bc199e20 tx=0x7fe8a8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:18.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:18.834+0000 7fe8ba000640 1 -- 172.21.15.72:0/1170590162 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe8b0021020 con 0x7fe8bc100a50 2024-04-27T00:01:19.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.034+0000 7fe8c2c00640 1 -- 172.21.15.72:0/1170590162 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe8bc02dad0 con 0x7fe8bc100a50 2024-04-27T00:01:19.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.035+0000 7fe8ba000640 1 -- 172.21.15.72:0/1170590162 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v93) ==== 81+0+591 (secure 0 0 0) 0x7fe8b009cd90 con 0x7fe8bc100a50 2024-04-27T00:01:19.037 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:0/3625071330 2024-04-27T23:34:58.266218+0000 2024-04-27T00:01:19.037 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:6801/2903979134 2024-04-27T23:34:58.266218+0000 2024-04-27T00:01:19.037 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:6800/2903979134 2024-04-27T23:34:58.266218+0000 2024-04-27T00:01:19.037 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:0/1519142576 2024-04-27T23:34:58.266218+0000 2024-04-27T00:01:19.037 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:0/3669626460 2024-04-27T23:34:24.233319+0000 2024-04-27T00:01:19.037 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:0/945623297 2024-04-27T23:34:24.233319+0000 2024-04-27T00:01:19.038 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:0/3479150073 2024-04-27T23:34:24.233319+0000 2024-04-27T00:01:19.038 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:6800/4175896290 2024-04-27T23:34:24.233319+0000 2024-04-27T00:01:19.038 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:6801/4175896290 2024-04-27T23:34:24.233319+0000 2024-04-27T00:01:19.038 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:0/3783175012 2024-04-27T23:34:58.266218+0000 2024-04-27T00:01:19.038 INFO:teuthology.orchestra.run.smithi072.stderr:listed 10 entries 2024-04-27T00:01:19.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.037+0000 7fe8c2c00640 1 -- 172.21.15.72:0/1170590162 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fe8a40a3c40 msgr2=0x7fe8a40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:19.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.037+0000 7fe8c2c00640 1 --2- 172.21.15.72:0/1170590162 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fe8a40a3c40 0x7fe8a40a6100 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fe8bc199e20 tx=0x7fe8a8005f90 comp rx=0 tx=0).stop 2024-04-27T00:01:19.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.037+0000 7fe8c2c00640 1 -- 172.21.15.72:0/1170590162 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8bc100a50 msgr2=0x7fe8bc198910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:19.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.037+0000 7fe8c2c00640 1 --2- 172.21.15.72:0/1170590162 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8bc100a50 0x7fe8bc198910 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fe8b000d950 tx=0x7fe8b000de10 comp rx=0 tx=0).stop 2024-04-27T00:01:19.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.038+0000 7fe8c2c00640 1 -- 172.21.15.72:0/1170590162 shutdown_connections 2024-04-27T00:01:19.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.038+0000 7fe8c2c00640 1 --2- 172.21.15.72:0/1170590162 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7fe8a40a3c40 0x7fe8a40a6100 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:19.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.038+0000 7fe8c2c00640 1 --2- 172.21.15.72:0/1170590162 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe8bc1043e0 0x7fe8bc19fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:19.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.038+0000 7fe8c2c00640 1 --2- 172.21.15.72:0/1170590162 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe8bc103a20 0x7fe8bc198e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:19.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.038+0000 7fe8c2c00640 1 --2- 172.21.15.72:0/1170590162 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8bc100a50 0x7fe8bc198910 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:19.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.038+0000 7fe8c2c00640 1 -- 172.21.15.72:0/1170590162 >> 172.21.15.72:0/1170590162 conn(0x7fe8bc0fc380 msgr2=0x7fe8bc106d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:19.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.038+0000 7fe8c2c00640 1 -- 172.21.15.72:0/1170590162 shutdown_connections 2024-04-27T00:01:19.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.038+0000 7fe8c2c00640 1 -- 172.21.15.72:0/1170590162 wait complete. 2024-04-27T00:01:19.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:18 smithi072 ceph-mon[28184]: pgmap v872: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:01:19.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:18 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/3241348274' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-27T00:01:19.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:18 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/3705027638' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-27T00:01:19.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:18 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/2195173958' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-27T00:01:19.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:18 smithi136 ceph-mon[33050]: pgmap v872: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:01:19.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:18 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/3241348274' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-27T00:01:19.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:18 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/3705027638' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-27T00:01:19.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:18 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/2195173958' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-27T00:01:19.057 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-27T00:01:19.057 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T00:01:19.058 DEBUG:teuthology.orchestra.run.smithi072:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-27T00:01:19.141 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi072.front.sepia.ceph.com 2024-04-27T00:01:19.142 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T00:01:19.142 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-27T00:01:19.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:18 smithi076 ceph-mon[32774]: pgmap v872: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-27T00:01:19.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:18 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/3241348274' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-27T00:01:19.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:18 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/3705027638' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-27T00:01:19.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:18 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/2195173958' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-27T00:01:19.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T00:01:19.183 DEBUG:teuthology.orchestra.run.smithi072:> ip netns list 2024-04-27T00:01:19.271 INFO:teuthology.orchestra.run.smithi072.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-27T00:01:19.271 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T00:01:19.271 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-27T00:01:19.399 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-27T00:01:19.399 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link delete ceph-brx 2024-04-27T00:01:19.509 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-27T00:01:19.640 INFO:tasks.cephadm:Teardown begin 2024-04-27T00:01:19.640 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-27T00:01:19.640 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-27T00:01:19.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.840+0000 7f539a200640 1 -- 172.21.15.72:0/3336346274 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5394078530 msgr2=0x7f5394078990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:19.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.840+0000 7f539a200640 1 --2- 172.21.15.72:0/3336346274 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5394078530 0x7f5394078990 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f537c009ca0 tx=0x7f537c02a8c0 comp rx=0 tx=0).stop 2024-04-27T00:01:19.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.840+0000 7f539a200640 1 -- 172.21.15.72:0/3336346274 shutdown_connections 2024-04-27T00:01:19.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.840+0000 7f539a200640 1 --2- 172.21.15.72:0/3336346274 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5394078530 0x7f5394078990 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:19.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.840+0000 7f539a200640 1 --2- 172.21.15.72:0/3336346274 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5394078ed0 0x7f5394079770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:19.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.840+0000 7f539a200640 1 --2- 172.21.15.72:0/3336346274 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53940772f0 0x7f5394077710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:19.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.840+0000 7f539a200640 1 -- 172.21.15.72:0/3336346274 >> 172.21.15.72:0/3336346274 conn(0x7f5394100120 msgr2=0x7f5394102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:19.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.840+0000 7f539a200640 1 -- 172.21.15.72:0/3336346274 shutdown_connections 2024-04-27T00:01:19.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.840+0000 7f539a200640 1 -- 172.21.15.72:0/3336346274 wait complete. 2024-04-27T00:01:19.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.841+0000 7f539a200640 1 Processor -- start 2024-04-27T00:01:19.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.842+0000 7f539a200640 1 -- start start 2024-04-27T00:01:19.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.842+0000 7f539a200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f53940772f0 0x7f539419f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:19.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.842+0000 7f539a200640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5394078530 0x7f539419f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:19.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.842+0000 7f539a200640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5394078ed0 0x7f53941a6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:19.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.842+0000 7f539a200640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f5394114ab0 con 0x7f5394078530 2024-04-27T00:01:19.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.842+0000 7f539a200640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f5394114930 con 0x7f53940772f0 2024-04-27T00:01:19.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.842+0000 7f539a200640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f5394114c30 con 0x7f5394078ed0 2024-04-27T00:01:19.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.842+0000 7f5392a00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5394078530 0x7f539419f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:19.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.842+0000 7f5393400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f53940772f0 0x7f539419f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:19.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.842+0000 7f5392a00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5394078530 0x7f539419f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43290/0 (socket says 172.21.15.72:43290) 2024-04-27T00:01:19.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.842+0000 7f5392a00640 1 -- 172.21.15.72:0/3257053374 learned_addr learned my addr 172.21.15.72:0/3257053374 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-27T00:01:19.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.842+0000 7f5393e00640 1 --2- 172.21.15.72:0/3257053374 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5394078ed0 0x7f53941a6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:19.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.843+0000 7f5393400640 1 -- 172.21.15.72:0/3257053374 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5394078ed0 msgr2=0x7f53941a6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:19.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.843+0000 7f5393400640 1 --2- 172.21.15.72:0/3257053374 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5394078ed0 0x7f53941a6750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:19.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.843+0000 7f5393400640 1 -- 172.21.15.72:0/3257053374 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5394078530 msgr2=0x7f539419f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:19.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.843+0000 7f5393400640 1 --2- 172.21.15.72:0/3257053374 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5394078530 0x7f539419f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:19.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.843+0000 7f5393400640 1 -- 172.21.15.72:0/3257053374 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53941a6dc0 con 0x7f53940772f0 2024-04-27T00:01:19.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.843+0000 7f5392a00640 1 --2- 172.21.15.72:0/3257053374 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5394078530 0x7f539419f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:01:19.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.843+0000 7f5393e00640 1 --2- 172.21.15.72:0/3257053374 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5394078ed0 0x7f53941a6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:01:19.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.843+0000 7f5393400640 1 --2- 172.21.15.72:0/3257053374 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f53940772f0 0x7f539419f1a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f538c00e9f0 tx=0x7f538c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:19.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.843+0000 7f538be00640 1 -- 172.21.15.72:0/3257053374 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f538c00ccc0 con 0x7f53940772f0 2024-04-27T00:01:19.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.843+0000 7f538be00640 1 -- 172.21.15.72:0/3257053374 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f538c005020 con 0x7f53940772f0 2024-04-27T00:01:19.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.843+0000 7f539a200640 1 -- 172.21.15.72:0/3257053374 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53941a70b0 con 0x7f53940772f0 2024-04-27T00:01:19.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.844+0000 7f538be00640 1 -- 172.21.15.72:0/3257053374 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f538c004a00 con 0x7f53940772f0 2024-04-27T00:01:19.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.844+0000 7f539a200640 1 -- 172.21.15.72:0/3257053374 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53941a7600 con 0x7f53940772f0 2024-04-27T00:01:19.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.845+0000 7f539a200640 1 -- 172.21.15.72:0/3257053374 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53941114e0 con 0x7f53940772f0 2024-04-27T00:01:19.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.847+0000 7f538be00640 1 -- 172.21.15.72:0/3257053374 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f538c004ba0 con 0x7f53940772f0 2024-04-27T00:01:19.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.848+0000 7f538be00640 1 --2- 172.21.15.72:0/3257053374 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f53780a3cc0 0x7f53780a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:19.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.849+0000 7f538be00640 1 -- 172.21.15.72:0/3257053374 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f538c0d0b00 con 0x7f53940772f0 2024-04-27T00:01:19.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.849+0000 7f5392a00640 1 --2- 172.21.15.72:0/3257053374 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f53780a3cc0 0x7f53780a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:19.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.849+0000 7f5392a00640 1 --2- 172.21.15.72:0/3257053374 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f53780a3cc0 0x7f53780a6180 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f53941a06b0 tx=0x7f537c005ef0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:19.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:19.851+0000 7f538be00640 1 -- 172.21.15.72:0/3257053374 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f538c09cf30 con 0x7f53940772f0 2024-04-27T00:01:20.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:19 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/1170590162' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-27T00:01:20.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:19 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/1170590162' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-27T00:01:20.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.082+0000 7f539a200640 1 -- 172.21.15.72:0/3257053374 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f539402dad0 con 0x7f53940772f0 2024-04-27T00:01:20.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.083+0000 7f538be00640 1 -- 172.21.15.72:0/3257053374 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11874 (secure 0 0 0) 0x7f538c09c830 con 0x7f53940772f0 2024-04-27T00:01:20.085 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-27T00:01:20.085 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/2386456166","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2386456166},{"type":"v1","addr":"172.21.15.72:6835","nonce":2386456166}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1528342053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1528342053},{"type":"v1","addr":"172.21.15.136:6833","nonce":1528342053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6833/2427238559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":2427238559},{"type":"v1","addr":"172.21.15.76:6833","nonce":2427238559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24491,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/4050689894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":4050689894},{"type":"v1","addr":"172.21.15.72:6837","nonce":4050689894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3681925367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3681925367},{"type":"v1","addr":"172.21.15.136:6835","nonce":3681925367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4251109085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4251109085},{"type":"v1","addr":"172.21.15.76:6835","nonce":4251109085}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T23:43:02.928649+0000","modified":"2024-04-26T23:46:14.854801+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24590,"mds_1":24571,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24514":{"gid":24514,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6837/3639308555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3639308555},{"type":"v1","addr":"172.21.15.136:6837","nonce":3639308555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.76:6837/2506930945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2506930945},{"type":"v1","addr":"172.21.15.76:6837","nonce":2506930945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6839/3938677630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3938677630},{"type":"v1","addr":"172.21.15.136:6839","nonce":3938677630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.72:6839/1403162102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6838","nonce":1403162102},{"type":"v1","addr":"172.21.15.72:6839","nonce":1403162102}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24571":{"gid":24571,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.72:6841/1356345077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6840","nonce":1356345077},{"type":"v1","addr":"172.21.15.72:6841","nonce":1356345077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.76:6839/3405829058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":3405829058},{"type":"v1","addr":"172.21.15.76:6839","nonce":3405829058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24590,24553,24571]},"id":1}]} 2024-04-27T00:01:20.085 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 24 2024-04-27T00:01:20.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.086+0000 7f539a200640 1 -- 172.21.15.72:0/3257053374 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f53780a3cc0 msgr2=0x7f53780a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:20.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.086+0000 7f539a200640 1 --2- 172.21.15.72:0/3257053374 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f53780a3cc0 0x7f53780a6180 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f53941a06b0 tx=0x7f537c005ef0 comp rx=0 tx=0).stop 2024-04-27T00:01:20.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.086+0000 7f539a200640 1 -- 172.21.15.72:0/3257053374 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f53940772f0 msgr2=0x7f539419f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:20.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.086+0000 7f539a200640 1 --2- 172.21.15.72:0/3257053374 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f53940772f0 0x7f539419f1a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f538c00e9f0 tx=0x7f538c00eeb0 comp rx=0 tx=0).stop 2024-04-27T00:01:20.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.086+0000 7f539a200640 1 -- 172.21.15.72:0/3257053374 shutdown_connections 2024-04-27T00:01:20.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.086+0000 7f539a200640 1 --2- 172.21.15.72:0/3257053374 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f53780a3cc0 0x7f53780a6180 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:20.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.086+0000 7f539a200640 1 --2- 172.21.15.72:0/3257053374 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5394078ed0 0x7f53941a6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:20.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.086+0000 7f539a200640 1 --2- 172.21.15.72:0/3257053374 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5394078530 0x7f539419f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:20.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.086+0000 7f539a200640 1 --2- 172.21.15.72:0/3257053374 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f53940772f0 0x7f539419f1a0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:20.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.086+0000 7f539a200640 1 -- 172.21.15.72:0/3257053374 >> 172.21.15.72:0/3257053374 conn(0x7f5394100120 msgr2=0x7f5394101c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:20.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.086+0000 7f539a200640 1 -- 172.21.15.72:0/3257053374 shutdown_connections 2024-04-27T00:01:20.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.086+0000 7f539a200640 1 -- 172.21.15.72:0/3257053374 wait complete. 2024-04-27T00:01:20.106 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-27T00:01:20.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:19 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/1170590162' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-27T00:01:20.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.299+0000 7f4a6fe00640 1 -- 172.21.15.72:0/796158503 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a700772f0 msgr2=0x7f4a70077710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:20.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.299+0000 7f4a6fe00640 1 --2- 172.21.15.72:0/796158503 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a700772f0 0x7f4a70077710 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f4a60009900 tx=0x7f4a6002a3f0 comp rx=0 tx=0).stop 2024-04-27T00:01:20.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.299+0000 7f4a6fe00640 1 -- 172.21.15.72:0/796158503 shutdown_connections 2024-04-27T00:01:20.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.299+0000 7f4a6fe00640 1 --2- 172.21.15.72:0/796158503 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4a70078ed0 0x7f4a70079770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:20.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.299+0000 7f4a6fe00640 1 --2- 172.21.15.72:0/796158503 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a70078530 0x7f4a70078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:20.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.299+0000 7f4a6fe00640 1 --2- 172.21.15.72:0/796158503 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a700772f0 0x7f4a70077710 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:20.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.299+0000 7f4a6fe00640 1 -- 172.21.15.72:0/796158503 >> 172.21.15.72:0/796158503 conn(0x7f4a700ffeb0 msgr2=0x7f4a70102310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:20.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.299+0000 7f4a6fe00640 1 -- 172.21.15.72:0/796158503 shutdown_connections 2024-04-27T00:01:20.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.299+0000 7f4a6fe00640 1 -- 172.21.15.72:0/796158503 wait complete. 2024-04-27T00:01:20.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.300+0000 7f4a6fe00640 1 Processor -- start 2024-04-27T00:01:20.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.300+0000 7f4a6fe00640 1 -- start start 2024-04-27T00:01:20.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.301+0000 7f4a6fe00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a700772f0 0x7f4a7019efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:20.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.301+0000 7f4a6fe00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a70078530 0x7f4a7019f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:20.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.301+0000 7f4a6fe00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4a70078ed0 0x7f4a701a6520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:20.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.301+0000 7f4a6fe00640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f4a701148e0 con 0x7f4a700772f0 2024-04-27T00:01:20.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.301+0000 7f4a6fe00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f4a70114760 con 0x7f4a70078ed0 2024-04-27T00:01:20.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.301+0000 7f4a6fe00640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f4a70114a60 con 0x7f4a70078530 2024-04-27T00:01:20.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.301+0000 7f4a6ea00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a700772f0 0x7f4a7019efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:20.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.301+0000 7f4a6ea00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a700772f0 0x7f4a7019efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43308/0 (socket says 172.21.15.72:43308) 2024-04-27T00:01:20.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.301+0000 7f4a6ea00640 1 -- 172.21.15.72:0/1955958949 learned_addr learned my addr 172.21.15.72:0/1955958949 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-27T00:01:20.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.301+0000 7f4a6f400640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4a70078ed0 0x7f4a701a6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:20.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.301+0000 7f4a6e000640 1 --2- 172.21.15.72:0/1955958949 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a70078530 0x7f4a7019f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:20.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.301+0000 7f4a6ea00640 1 -- 172.21.15.72:0/1955958949 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a70078530 msgr2=0x7f4a7019f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:20.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.301+0000 7f4a6ea00640 1 --2- 172.21.15.72:0/1955958949 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a70078530 0x7f4a7019f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:20.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.301+0000 7f4a6ea00640 1 -- 172.21.15.72:0/1955958949 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4a70078ed0 msgr2=0x7f4a701a6520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:20.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.301+0000 7f4a6ea00640 1 --2- 172.21.15.72:0/1955958949 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4a70078ed0 0x7f4a701a6520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:20.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.301+0000 7f4a6ea00640 1 -- 172.21.15.72:0/1955958949 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a701a6b90 con 0x7f4a700772f0 2024-04-27T00:01:20.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.301+0000 7f4a6e000640 1 --2- 172.21.15.72:0/1955958949 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a70078530 0x7f4a7019f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:01:20.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.301+0000 7f4a6f400640 1 --2- 172.21.15.72:0/1955958949 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4a70078ed0 0x7f4a701a6520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:01:20.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.302+0000 7f4a6ea00640 1 --2- 172.21.15.72:0/1955958949 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a700772f0 0x7f4a7019efc0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f4a60005ab0 tx=0x7f4a60003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:20.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.302+0000 7f4a67400640 1 -- 172.21.15.72:0/1955958949 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a6003f070 con 0x7f4a700772f0 2024-04-27T00:01:20.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.302+0000 7f4a6fe00640 1 -- 172.21.15.72:0/1955958949 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a701a6e20 con 0x7f4a700772f0 2024-04-27T00:01:20.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.302+0000 7f4a67400640 1 -- 172.21.15.72:0/1955958949 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a60031c50 con 0x7f4a700772f0 2024-04-27T00:01:20.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.302+0000 7f4a67400640 1 -- 172.21.15.72:0/1955958949 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a60035030 con 0x7f4a700772f0 2024-04-27T00:01:20.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.302+0000 7f4a6fe00640 1 -- 172.21.15.72:0/1955958949 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a701a7230 con 0x7f4a700772f0 2024-04-27T00:01:20.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.303+0000 7f4a6fe00640 1 -- 172.21.15.72:0/1955958949 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a701112d0 con 0x7f4a700772f0 2024-04-27T00:01:20.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.305+0000 7f4a67400640 1 -- 172.21.15.72:0/1955958949 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4a60042070 con 0x7f4a700772f0 2024-04-27T00:01:20.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.307+0000 7f4a67400640 1 --2- 172.21.15.72:0/1955958949 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f4a540a3c40 0x7f4a540a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:20.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.307+0000 7f4a67400640 1 -- 172.21.15.72:0/1955958949 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f4a600f1170 con 0x7f4a700772f0 2024-04-27T00:01:20.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.307+0000 7f4a6e000640 1 --2- 172.21.15.72:0/1955958949 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f4a540a3c40 0x7f4a540a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:20.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.308+0000 7f4a6e000640 1 --2- 172.21.15.72:0/1955958949 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f4a540a3c40 0x7f4a540a6100 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f4a701a0480 tx=0x7f4a5c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:20.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.311+0000 7f4a67400640 1 -- 172.21.15.72:0/1955958949 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4a600bd5a0 con 0x7f4a700772f0 2024-04-27T00:01:20.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.557+0000 7f4a6fe00640 1 -- 172.21.15.72:0/1955958949 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4a7002dcf0 con 0x7f4a700772f0 2024-04-27T00:01:20.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.559+0000 7f4a67400640 1 -- 172.21.15.72:0/1955958949 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11874 (secure 0 0 0) 0x7f4a600bcea0 con 0x7f4a700772f0 2024-04-27T00:01:20.560 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-27T00:01:20.561 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14397,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/2386456166","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2386456166},{"type":"v1","addr":"172.21.15.72:6835","nonce":2386456166}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24445,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6833/1528342053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":1528342053},{"type":"v1","addr":"172.21.15.136:6833","nonce":1528342053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6833/2427238559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":2427238559},{"type":"v1","addr":"172.21.15.76:6833","nonce":2427238559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24491,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/4050689894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":4050689894},{"type":"v1","addr":"172.21.15.72:6837","nonce":4050689894}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24502,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3681925367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3681925367},{"type":"v1","addr":"172.21.15.136:6835","nonce":3681925367}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24503,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/4251109085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4251109085},{"type":"v1","addr":"172.21.15.76:6835","nonce":4251109085}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T23:43:02.928649+0000","modified":"2024-04-26T23:46:14.854801+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24590,"mds_1":24571,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24514":{"gid":24514,"name":"c","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6837/3639308555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":3639308555},{"type":"v1","addr":"172.21.15.136:6837","nonce":3639308555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.76:6837/2506930945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6836","nonce":2506930945},{"type":"v1","addr":"172.21.15.76:6837","nonce":2506930945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.136:6839/3938677630","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3938677630},{"type":"v1","addr":"172.21.15.136:6839","nonce":3938677630}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"g","rank":2,"incarnation":21,"state":"up:active","state_seq":11,"addr":"172.21.15.72:6839/1403162102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6838","nonce":1403162102},{"type":"v1","addr":"172.21.15.72:6839","nonce":1403162102}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24571":{"gid":24571,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.72:6841/1356345077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6840","nonce":1356345077},{"type":"v1","addr":"172.21.15.72:6841","nonce":1356345077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.76:6839/3405829058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6838","nonce":3405829058},{"type":"v1","addr":"172.21.15.76:6839","nonce":3405829058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24590,24553,24571]},"id":1}]} 2024-04-27T00:01:20.561 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 24 2024-04-27T00:01:20.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.562+0000 7f4a6fe00640 1 -- 172.21.15.72:0/1955958949 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f4a540a3c40 msgr2=0x7f4a540a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:20.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.562+0000 7f4a6fe00640 1 --2- 172.21.15.72:0/1955958949 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f4a540a3c40 0x7f4a540a6100 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f4a701a0480 tx=0x7f4a5c005f90 comp rx=0 tx=0).stop 2024-04-27T00:01:20.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.562+0000 7f4a6fe00640 1 -- 172.21.15.72:0/1955958949 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a700772f0 msgr2=0x7f4a7019efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:20.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.562+0000 7f4a6fe00640 1 --2- 172.21.15.72:0/1955958949 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a700772f0 0x7f4a7019efc0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f4a60005ab0 tx=0x7f4a60003b90 comp rx=0 tx=0).stop 2024-04-27T00:01:20.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.562+0000 7f4a6fe00640 1 -- 172.21.15.72:0/1955958949 shutdown_connections 2024-04-27T00:01:20.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.562+0000 7f4a6fe00640 1 --2- 172.21.15.72:0/1955958949 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f4a540a3c40 0x7f4a540a6100 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:20.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.562+0000 7f4a6fe00640 1 --2- 172.21.15.72:0/1955958949 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4a70078ed0 0x7f4a701a6520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:20.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.562+0000 7f4a6fe00640 1 --2- 172.21.15.72:0/1955958949 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a70078530 0x7f4a7019f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:20.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.562+0000 7f4a6fe00640 1 --2- 172.21.15.72:0/1955958949 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a700772f0 0x7f4a7019efc0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:20.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.562+0000 7f4a6fe00640 1 -- 172.21.15.72:0/1955958949 >> 172.21.15.72:0/1955958949 conn(0x7f4a700ffeb0 msgr2=0x7f4a701019b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:20.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.563+0000 7f4a6fe00640 1 -- 172.21.15.72:0/1955958949 shutdown_connections 2024-04-27T00:01:20.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.563+0000 7f4a6fe00640 1 -- 172.21.15.72:0/1955958949 wait complete. 2024-04-27T00:01:20.583 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-27T00:01:20.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.784+0000 7f0168c00640 1 -- 172.21.15.72:0/2547957242 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0164102970 msgr2=0x7f0164102d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:20.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.784+0000 7f0168c00640 1 --2- 172.21.15.72:0/2547957242 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0164102970 0x7f0164102d90 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0154009900 tx=0x7f015402a3f0 comp rx=0 tx=0).stop 2024-04-27T00:01:20.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.784+0000 7f0168c00640 1 -- 172.21.15.72:0/2547957242 shutdown_connections 2024-04-27T00:01:20.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.784+0000 7f0168c00640 1 --2- 172.21.15.72:0/2547957242 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0164104520 0x7f016410adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:20.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.784+0000 7f0168c00640 1 --2- 172.21.15.72:0/2547957242 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0164103b60 0x7f0164103fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:20.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.784+0000 7f0168c00640 1 --2- 172.21.15.72:0/2547957242 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0164102970 0x7f0164102d90 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:20.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.784+0000 7f0168c00640 1 -- 172.21.15.72:0/2547957242 >> 172.21.15.72:0/2547957242 conn(0x7f01640fdee0 msgr2=0x7f0164100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:20.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.784+0000 7f0168c00640 1 -- 172.21.15.72:0/2547957242 shutdown_connections 2024-04-27T00:01:20.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.784+0000 7f0168c00640 1 -- 172.21.15.72:0/2547957242 wait complete. 2024-04-27T00:01:20.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.785+0000 7f0168c00640 1 Processor -- start 2024-04-27T00:01:20.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.785+0000 7f0168c00640 1 -- start start 2024-04-27T00:01:20.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.785+0000 7f0168c00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0164102970 0x7f0164198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:20.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.785+0000 7f0168c00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0164103b60 0x7f0164199110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:20.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.785+0000 7f0168c00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0164104520 0x7f01641a0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:20.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.785+0000 7f0168c00640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f016410e490 con 0x7f0164104520 2024-04-27T00:01:20.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.785+0000 7f0168c00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f016410e310 con 0x7f0164102970 2024-04-27T00:01:20.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.785+0000 7f0168c00640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f016410e610 con 0x7f0164103b60 2024-04-27T00:01:20.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.786+0000 7f0161e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0164102970 0x7f0164198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:20.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.786+0000 7f0161e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0164102970 0x7f0164198bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:57428/0 (socket says 172.21.15.72:57428) 2024-04-27T00:01:20.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.786+0000 7f0161e00640 1 -- 172.21.15.72:0/572021347 learned_addr learned my addr 172.21.15.72:0/572021347 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-27T00:01:20.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.786+0000 7f0159400640 1 --2- 172.21.15.72:0/572021347 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0164103b60 0x7f0164199110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:20.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.786+0000 7f0162800640 1 --2- 172.21.15.72:0/572021347 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0164104520 0x7f01641a0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:20.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.786+0000 7f0161e00640 1 -- 172.21.15.72:0/572021347 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0164103b60 msgr2=0x7f0164199110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:20.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.786+0000 7f0161e00640 1 --2- 172.21.15.72:0/572021347 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0164103b60 0x7f0164199110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:20.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.786+0000 7f0161e00640 1 -- 172.21.15.72:0/572021347 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0164104520 msgr2=0x7f01641a0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:20.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.786+0000 7f0161e00640 1 --2- 172.21.15.72:0/572021347 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0164104520 0x7f01641a0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:20.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.786+0000 7f0161e00640 1 -- 172.21.15.72:0/572021347 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01641a07f0 con 0x7f0164102970 2024-04-27T00:01:20.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.787+0000 7f0161e00640 1 --2- 172.21.15.72:0/572021347 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0164102970 0x7f0164198bd0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0154006010 tx=0x7f01540042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:20.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.787+0000 7f015b400640 1 -- 172.21.15.72:0/572021347 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f015403f030 con 0x7f0164102970 2024-04-27T00:01:20.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.787+0000 7f015b400640 1 -- 172.21.15.72:0/572021347 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0154031910 con 0x7f0164102970 2024-04-27T00:01:20.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.787+0000 7f0168c00640 1 -- 172.21.15.72:0/572021347 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01641a0a80 con 0x7f0164102970 2024-04-27T00:01:20.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.787+0000 7f015b400640 1 -- 172.21.15.72:0/572021347 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0154035070 con 0x7f0164102970 2024-04-27T00:01:20.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.787+0000 7f0168c00640 1 -- 172.21.15.72:0/572021347 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01641a0fc0 con 0x7f0164102970 2024-04-27T00:01:20.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.788+0000 7f0168c00640 1 -- 172.21.15.72:0/572021347 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0164109db0 con 0x7f0164102970 2024-04-27T00:01:20.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.790+0000 7f015b400640 1 -- 172.21.15.72:0/572021347 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f0154042030 con 0x7f0164102970 2024-04-27T00:01:20.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.791+0000 7f015b400640 1 --2- 172.21.15.72:0/572021347 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f01480a3cc0 0x7f01480a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:20.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.792+0000 7f015b400640 1 -- 172.21.15.72:0/572021347 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f01540f0d70 con 0x7f0164102970 2024-04-27T00:01:20.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.792+0000 7f0159400640 1 --2- 172.21.15.72:0/572021347 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f01480a3cc0 0x7f01480a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:20.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.792+0000 7f0159400640 1 --2- 172.21.15.72:0/572021347 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f01480a3cc0 0x7f01480a6180 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f016419a0e0 tx=0x7f014c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:20.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:20.796+0000 7f015b400640 1 -- 172.21.15.72:0/572021347 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f01540bd1a0 con 0x7f0164102970 2024-04-27T00:01:21.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.002+0000 7f0168c00640 1 -- 172.21.15.72:0/572021347 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f016402dd20 con 0x7f0164102970 2024-04-27T00:01:21.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.004+0000 7f015b400640 1 -- 172.21.15.72:0/572021347 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24841 (secure 0 0 0) 0x7f01540dd0e0 con 0x7f0164102970 2024-04-27T00:01:21.006 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-27T00:01:21.006 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":93,"fsid":"3c8453ae-0425-11ef-bc93-c7b262605968","created":"2024-04-26T23:33:48.088300+0000","modified":"2024-04-26T23:45:00.532006+0000","last_up_change":"2024-04-26T23:42:04.482891+0000","last_in_change":"2024-04-26T23:41:44.784415+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T23:37:44.428727+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T23:43:00.431829+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T23:43:01.872977+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T23:43:05.512050+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e20c9ebf-56f6-47c1-a11b-e1e615af042b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6803","nonce":1639298875}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6805","nonce":1639298875}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6809","nonce":1639298875}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1639298875},{"type":"v1","addr":"172.21.15.72:6807","nonce":1639298875}]},"public_addr":"172.21.15.72:6803/1639298875","cluster_addr":"172.21.15.72:6805/1639298875","heartbeat_back_addr":"172.21.15.72:6809/1639298875","heartbeat_front_addr":"172.21.15.72:6807/1639298875","state":["exists","up"]},{"osd":1,"uuid":"e6cac469-780c-4d66-b560-b686072371fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6801","nonce":4185342193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6803","nonce":4185342193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6807","nonce":4185342193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":4185342193},{"type":"v1","addr":"172.21.15.76:6805","nonce":4185342193}]},"public_addr":"172.21.15.76:6801/4185342193","cluster_addr":"172.21.15.76:6803/4185342193","heartbeat_back_addr":"172.21.15.76:6807/4185342193","heartbeat_front_addr":"172.21.15.76:6805/4185342193","state":["exists","up"]},{"osd":2,"uuid":"838a49fc-18de-446c-8336-67b3e05da231","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6801","nonce":3033860799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6803","nonce":3033860799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6807","nonce":3033860799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":3033860799},{"type":"v1","addr":"172.21.15.136:6805","nonce":3033860799}]},"public_addr":"172.21.15.136:6801/3033860799","cluster_addr":"172.21.15.136:6803/3033860799","heartbeat_back_addr":"172.21.15.136:6807/3033860799","heartbeat_front_addr":"172.21.15.136:6805/3033860799","state":["exists","up"]},{"osd":3,"uuid":"e6da89be-5ba7-48a4-9306-f75498660e0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6811","nonce":2071107815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6813","nonce":2071107815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6817","nonce":2071107815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":2071107815},{"type":"v1","addr":"172.21.15.72:6815","nonce":2071107815}]},"public_addr":"172.21.15.72:6811/2071107815","cluster_addr":"172.21.15.72:6813/2071107815","heartbeat_back_addr":"172.21.15.72:6817/2071107815","heartbeat_front_addr":"172.21.15.72:6815/2071107815","state":["exists","up"]},{"osd":4,"uuid":"0b17c491-d798-488f-892d-521ef2fbc51e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6809","nonce":147449238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6811","nonce":147449238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6815","nonce":147449238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":147449238},{"type":"v1","addr":"172.21.15.76:6813","nonce":147449238}]},"public_addr":"172.21.15.76:6809/147449238","cluster_addr":"172.21.15.76:6811/147449238","heartbeat_back_addr":"172.21.15.76:6815/147449238","heartbeat_front_addr":"172.21.15.76:6813/147449238","state":["exists","up"]},{"osd":5,"uuid":"c7e887b3-ab38-4608-9bfd-1ed49956e8af","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6809","nonce":2320149212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6811","nonce":2320149212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6815","nonce":2320149212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2320149212},{"type":"v1","addr":"172.21.15.136:6813","nonce":2320149212}]},"public_addr":"172.21.15.136:6809/2320149212","cluster_addr":"172.21.15.136:6811/2320149212","heartbeat_back_addr":"172.21.15.136:6815/2320149212","heartbeat_front_addr":"172.21.15.136:6813/2320149212","state":["exists","up"]},{"osd":6,"uuid":"f0bd604b-0f57-4a38-9295-eb200264f1ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6819","nonce":389499207}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6821","nonce":389499207}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6825","nonce":389499207}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":389499207},{"type":"v1","addr":"172.21.15.72:6823","nonce":389499207}]},"public_addr":"172.21.15.72:6819/389499207","cluster_addr":"172.21.15.72:6821/389499207","heartbeat_back_addr":"172.21.15.72:6825/389499207","heartbeat_front_addr":"172.21.15.72:6823/389499207","state":["exists","up"]},{"osd":7,"uuid":"1879d4e4-038e-4cd6-a3ec-3a940bcc3797","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6817","nonce":375888575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6819","nonce":375888575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6823","nonce":375888575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":375888575},{"type":"v1","addr":"172.21.15.76:6821","nonce":375888575}]},"public_addr":"172.21.15.76:6817/375888575","cluster_addr":"172.21.15.76:6819/375888575","heartbeat_back_addr":"172.21.15.76:6823/375888575","heartbeat_front_addr":"172.21.15.76:6821/375888575","state":["exists","up"]},{"osd":8,"uuid":"6f404062-97e7-4ccb-a582-76dc1ffae548","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6817","nonce":3446774052}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6819","nonce":3446774052}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6823","nonce":3446774052}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":3446774052},{"type":"v1","addr":"172.21.15.136:6821","nonce":3446774052}]},"public_addr":"172.21.15.136:6817/3446774052","cluster_addr":"172.21.15.136:6819/3446774052","heartbeat_back_addr":"172.21.15.136:6823/3446774052","heartbeat_front_addr":"172.21.15.136:6821/3446774052","state":["exists","up"]},{"osd":9,"uuid":"2989ca68-3164-4667-9bcf-230dcbb7b8f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6827","nonce":3423789152}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6829","nonce":3423789152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6832","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6833","nonce":3423789152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3423789152},{"type":"v1","addr":"172.21.15.72:6831","nonce":3423789152}]},"public_addr":"172.21.15.72:6827/3423789152","cluster_addr":"172.21.15.72:6829/3423789152","heartbeat_back_addr":"172.21.15.72:6833/3423789152","heartbeat_front_addr":"172.21.15.72:6831/3423789152","state":["exists","up"]},{"osd":10,"uuid":"9ec6d46a-0913-4208-97b5-167188ee8cb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6825","nonce":1120991574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6827","nonce":1120991574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6831","nonce":1120991574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":1120991574},{"type":"v1","addr":"172.21.15.76:6829","nonce":1120991574}]},"public_addr":"172.21.15.76:6825/1120991574","cluster_addr":"172.21.15.76:6827/1120991574","heartbeat_back_addr":"172.21.15.76:6831/1120991574","heartbeat_front_addr":"172.21.15.76:6829/1120991574","state":["exists","up"]},{"osd":11,"uuid":"bdd0774f-6e58-4918-b4cc-afd4f1118d3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6825","nonce":1135995193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6827","nonce":1135995193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6831","nonce":1135995193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":1135995193},{"type":"v1","addr":"172.21.15.136:6829","nonce":1135995193}]},"public_addr":"172.21.15.136:6825/1135995193","cluster_addr":"172.21.15.136:6827/1135995193","heartbeat_back_addr":"172.21.15.136:6831/1135995193","heartbeat_front_addr":"172.21.15.136:6829/1135995193","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:36:52.128788+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:37:17.523654+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:37:41.634677+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:38:15.060797+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:38:41.234111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:39:06.783560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:39:40.884386+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:40:08.340465+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:40:34.344858+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:41:08.525309+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:41:36.782624+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T23:42:02.756888+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":6}]},{"pgid":"3.37","mappings":[{"from":5,"to":7}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:0/3625071330":"2024-04-27T23:34:58.266218+0000","172.21.15.72:6801/2903979134":"2024-04-27T23:34:58.266218+0000","172.21.15.72:6800/2903979134":"2024-04-27T23:34:58.266218+0000","172.21.15.72:0/1519142576":"2024-04-27T23:34:58.266218+0000","172.21.15.72:0/3669626460":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/945623297":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/3479150073":"2024-04-27T23:34:24.233319+0000","172.21.15.72:6800/4175896290":"2024-04-27T23:34:24.233319+0000","172.21.15.72:6801/4175896290":"2024-04-27T23:34:24.233319+0000","172.21.15.72:0/3783175012":"2024-04-27T23:34:58.266218+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-27T00:01:21.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.007+0000 7f0168c00640 1 -- 172.21.15.72:0/572021347 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f01480a3cc0 msgr2=0x7f01480a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:21.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.007+0000 7f0168c00640 1 --2- 172.21.15.72:0/572021347 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f01480a3cc0 0x7f01480a6180 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f016419a0e0 tx=0x7f014c005f90 comp rx=0 tx=0).stop 2024-04-27T00:01:21.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.007+0000 7f0168c00640 1 -- 172.21.15.72:0/572021347 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0164102970 msgr2=0x7f0164198bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:21.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.007+0000 7f0168c00640 1 --2- 172.21.15.72:0/572021347 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0164102970 0x7f0164198bd0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0154006010 tx=0x7f01540042f0 comp rx=0 tx=0).stop 2024-04-27T00:01:21.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.007+0000 7f0168c00640 1 -- 172.21.15.72:0/572021347 shutdown_connections 2024-04-27T00:01:21.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.007+0000 7f0168c00640 1 --2- 172.21.15.72:0/572021347 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f01480a3cc0 0x7f01480a6180 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:21.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.007+0000 7f0168c00640 1 --2- 172.21.15.72:0/572021347 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0164104520 0x7f01641a0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:21.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.007+0000 7f0168c00640 1 --2- 172.21.15.72:0/572021347 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0164103b60 0x7f0164199110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:21.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.007+0000 7f0168c00640 1 --2- 172.21.15.72:0/572021347 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0164102970 0x7f0164198bd0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:21.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.007+0000 7f0168c00640 1 -- 172.21.15.72:0/572021347 >> 172.21.15.72:0/572021347 conn(0x7f01640fdee0 msgr2=0x7f0164106f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:21.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.007+0000 7f0168c00640 1 -- 172.21.15.72:0/572021347 shutdown_connections 2024-04-27T00:01:21.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.007+0000 7f0168c00640 1 -- 172.21.15.72:0/572021347 wait complete. 2024-04-27T00:01:21.025 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:20 smithi072 ceph-mon[28184]: pgmap v873: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:01:21.025 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:20 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/3257053374' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-27T00:01:21.025 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:20 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/1955958949' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-27T00:01:21.030 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-27T00:01:21.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:20 smithi136 ceph-mon[33050]: pgmap v873: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:01:21.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:20 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/3257053374' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-27T00:01:21.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:20 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/1955958949' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-27T00:01:21.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:20 smithi076 ceph-mon[32774]: pgmap v873: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-27T00:01:21.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:20 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/3257053374' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-27T00:01:21.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:20 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/1955958949' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-27T00:01:21.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.222+0000 7f5703000640 1 -- 172.21.15.72:0/2509823007 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f56fc103450 msgr2=0x7f56fc105840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:21.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.222+0000 7f5703000640 1 --2- 172.21.15.72:0/2509823007 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f56fc103450 0x7f56fc105840 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f56ec009900 tx=0x7f56ec02a410 comp rx=0 tx=0).stop 2024-04-27T00:01:21.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.222+0000 7f5703000640 1 -- 172.21.15.72:0/2509823007 shutdown_connections 2024-04-27T00:01:21.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.222+0000 7f5703000640 1 --2- 172.21.15.72:0/2509823007 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56fc1086b0 0x7f56fc10aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:21.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.222+0000 7f5703000640 1 --2- 172.21.15.72:0/2509823007 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f56fc105d80 0x7f56fc108170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:21.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.222+0000 7f5703000640 1 --2- 172.21.15.72:0/2509823007 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f56fc103450 0x7f56fc105840 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:21.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.222+0000 7f5703000640 1 -- 172.21.15.72:0/2509823007 >> 172.21.15.72:0/2509823007 conn(0x7f56fc0fcde0 msgr2=0x7f56fc0ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:21.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.222+0000 7f5703000640 1 -- 172.21.15.72:0/2509823007 shutdown_connections 2024-04-27T00:01:21.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.222+0000 7f5703000640 1 -- 172.21.15.72:0/2509823007 wait complete. 2024-04-27T00:01:21.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.223+0000 7f5703000640 1 Processor -- start 2024-04-27T00:01:21.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.223+0000 7f5703000640 1 -- start start 2024-04-27T00:01:21.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.224+0000 7f5703000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f56fc103450 0x7f56fc19ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:21.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.224+0000 7f5703000640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f56fc105d80 0x7f56fc19b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:21.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.224+0000 7f5703000640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56fc1086b0 0x7f56fc1a23b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:21.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.224+0000 7f5703000640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f56fc10e510 con 0x7f56fc105d80 2024-04-27T00:01:21.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.224+0000 7f5703000640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f56fc10e390 con 0x7f56fc103450 2024-04-27T00:01:21.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.224+0000 7f5703000640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f56fc10e690 con 0x7f56fc1086b0 2024-04-27T00:01:21.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.224+0000 7f56f3e00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f56fc105d80 0x7f56fc19b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:21.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.224+0000 7f56f3e00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f56fc105d80 0x7f56fc19b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43338/0 (socket says 172.21.15.72:43338) 2024-04-27T00:01:21.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.224+0000 7f56f3e00640 1 -- 172.21.15.72:0/1122408760 learned_addr learned my addr 172.21.15.72:0/1122408760 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-27T00:01:21.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.224+0000 7f56fbe00640 1 --2- 172.21.15.72:0/1122408760 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f56fc103450 0x7f56fc19ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:21.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.224+0000 7f5701000640 1 --2- 172.21.15.72:0/1122408760 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56fc1086b0 0x7f56fc1a23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:21.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.224+0000 7f56f3e00640 1 -- 172.21.15.72:0/1122408760 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56fc1086b0 msgr2=0x7f56fc1a23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:21.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.224+0000 7f56f3e00640 1 --2- 172.21.15.72:0/1122408760 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56fc1086b0 0x7f56fc1a23b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:21.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.224+0000 7f56f3e00640 1 -- 172.21.15.72:0/1122408760 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f56fc103450 msgr2=0x7f56fc19ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:21.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.224+0000 7f56f3e00640 1 --2- 172.21.15.72:0/1122408760 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f56fc103450 0x7f56fc19ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:21.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.224+0000 7f56f3e00640 1 -- 172.21.15.72:0/1122408760 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56fc1a2a20 con 0x7f56fc105d80 2024-04-27T00:01:21.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.225+0000 7f56f3e00640 1 --2- 172.21.15.72:0/1122408760 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f56fc105d80 0x7f56fc19b340 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f56e400e9c0 tx=0x7f56e400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:21.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.225+0000 7f56f9600640 1 -- 172.21.15.72:0/1122408760 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56e400cd80 con 0x7f56fc105d80 2024-04-27T00:01:21.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.225+0000 7f56f9600640 1 -- 172.21.15.72:0/1122408760 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f56e40054d0 con 0x7f56fc105d80 2024-04-27T00:01:21.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.225+0000 7f5703000640 1 -- 172.21.15.72:0/1122408760 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56fc1a2d10 con 0x7f56fc105d80 2024-04-27T00:01:21.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.225+0000 7f56f9600640 1 -- 172.21.15.72:0/1122408760 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56e4010430 con 0x7f56fc105d80 2024-04-27T00:01:21.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.225+0000 7f5703000640 1 -- 172.21.15.72:0/1122408760 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56fc1a3200 con 0x7f56fc105d80 2024-04-27T00:01:21.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.226+0000 7f5703000640 1 -- 172.21.15.72:0/1122408760 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56fc10a1a0 con 0x7f56fc105d80 2024-04-27T00:01:21.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.229+0000 7f56f9600640 1 -- 172.21.15.72:0/1122408760 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f56e4004fd0 con 0x7f56fc105d80 2024-04-27T00:01:21.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.230+0000 7f56f9600640 1 --2- 172.21.15.72:0/1122408760 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f56e00a3c40 0x7f56e00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:21.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.230+0000 7f56f9600640 1 -- 172.21.15.72:0/1122408760 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9237+0+0 (secure 0 0 0) 0x7f56e40d0d70 con 0x7f56fc105d80 2024-04-27T00:01:21.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.230+0000 7f56fbe00640 1 --2- 172.21.15.72:0/1122408760 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f56e00a3c40 0x7f56e00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:21.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.231+0000 7f56fbe00640 1 --2- 172.21.15.72:0/1122408760 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f56e00a3c40 0x7f56e00a6100 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f56ec031500 tx=0x7f56ec005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:21.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.234+0000 7f56f9600640 1 -- 172.21.15.72:0/1122408760 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f56e409d200 con 0x7f56fc105d80 2024-04-27T00:01:21.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.479+0000 7f5703000640 1 -- 172.21.15.72:0/1122408760 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f56fc02dd20 con 0x7f56fc105d80 2024-04-27T00:01:21.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.483+0000 7f56f9600640 1 -- 172.21.15.72:0/1122408760 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v25) ==== 161+0+0 (secure 0 0 0) 0x7f56e409cb00 con 0x7f56fc105d80 2024-04-27T00:01:21.485 INFO:teuthology.orchestra.run.smithi072.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-27T00:01:21.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.487+0000 7f5703000640 1 -- 172.21.15.72:0/1122408760 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f56e00a3c40 msgr2=0x7f56e00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:21.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.487+0000 7f5703000640 1 --2- 172.21.15.72:0/1122408760 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f56e00a3c40 0x7f56e00a6100 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f56ec031500 tx=0x7f56ec005a20 comp rx=0 tx=0).stop 2024-04-27T00:01:21.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.487+0000 7f5703000640 1 -- 172.21.15.72:0/1122408760 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f56fc105d80 msgr2=0x7f56fc19b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:21.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.487+0000 7f5703000640 1 --2- 172.21.15.72:0/1122408760 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f56fc105d80 0x7f56fc19b340 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f56e400e9c0 tx=0x7f56e400ee80 comp rx=0 tx=0).stop 2024-04-27T00:01:21.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.489+0000 7f5703000640 1 -- 172.21.15.72:0/1122408760 shutdown_connections 2024-04-27T00:01:21.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.489+0000 7f5703000640 1 --2- 172.21.15.72:0/1122408760 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f56e00a3c40 0x7f56e00a6100 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:21.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.489+0000 7f5703000640 1 --2- 172.21.15.72:0/1122408760 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56fc1086b0 0x7f56fc1a23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:21.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.489+0000 7f5703000640 1 --2- 172.21.15.72:0/1122408760 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f56fc105d80 0x7f56fc19b340 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:21.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.489+0000 7f5703000640 1 --2- 172.21.15.72:0/1122408760 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f56fc103450 0x7f56fc19ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:21.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.489+0000 7f5703000640 1 -- 172.21.15.72:0/1122408760 >> 172.21.15.72:0/1122408760 conn(0x7f56fc0fcde0 msgr2=0x7f56fc102be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:21.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.490+0000 7f5703000640 1 -- 172.21.15.72:0/1122408760 shutdown_connections 2024-04-27T00:01:21.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.490+0000 7f5703000640 1 -- 172.21.15.72:0/1122408760 wait complete. 2024-04-27T00:01:21.518 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-27T00:01:21.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.698+0000 7f4ce7200640 1 -- 172.21.15.72:0/3082274280 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ce8101750 msgr2=0x7f4ce8101b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:21.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.698+0000 7f4ce7200640 1 --2- 172.21.15.72:0/3082274280 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ce8101750 0x7f4ce8101b70 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8009900 tx=0x7f4cd802a400 comp rx=0 tx=0).stop 2024-04-27T00:01:21.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.699+0000 7f4ce7200640 1 -- 172.21.15.72:0/3082274280 shutdown_connections 2024-04-27T00:01:21.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.699+0000 7f4ce7200640 1 --2- 172.21.15.72:0/3082274280 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4ce8103300 0x7f4ce8109b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:21.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.699+0000 7f4ce7200640 1 --2- 172.21.15.72:0/3082274280 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ce8102940 0x7f4ce8102dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:21.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.699+0000 7f4ce7200640 1 --2- 172.21.15.72:0/3082274280 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ce8101750 0x7f4ce8101b70 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:21.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.699+0000 7f4ce7200640 1 -- 172.21.15.72:0/3082274280 >> 172.21.15.72:0/3082274280 conn(0x7f4ce80fcce0 msgr2=0x7f4ce80ff100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:21.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.699+0000 7f4ce7200640 1 -- 172.21.15.72:0/3082274280 shutdown_connections 2024-04-27T00:01:21.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.699+0000 7f4ce7200640 1 -- 172.21.15.72:0/3082274280 wait complete. 2024-04-27T00:01:21.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.700+0000 7f4ce7200640 1 Processor -- start 2024-04-27T00:01:21.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.700+0000 7f4ce7200640 1 -- start start 2024-04-27T00:01:21.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.700+0000 7f4ce7200640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4ce8101750 0x7f4ce819acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:21.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.700+0000 7f4ce7200640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ce8102940 0x7f4ce819b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:21.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.700+0000 7f4ce7200640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ce8103300 0x7f4ce81a2260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:21.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.700+0000 7f4ce7200640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f4ce810d270 con 0x7f4ce8103300 2024-04-27T00:01:21.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.700+0000 7f4ce7200640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f4ce810d0f0 con 0x7f4ce8101750 2024-04-27T00:01:21.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.700+0000 7f4ce7200640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f4ce810d3f0 con 0x7f4ce8102940 2024-04-27T00:01:21.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.700+0000 7f4ce5e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4ce8101750 0x7f4ce819acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:21.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.700+0000 7f4ce5e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4ce8101750 0x7f4ce819acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.72:57504/0 (socket says 172.21.15.72:57504) 2024-04-27T00:01:21.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.700+0000 7f4ce5e00640 1 -- 172.21.15.72:0/785120440 learned_addr learned my addr 172.21.15.72:0/785120440 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-27T00:01:21.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.700+0000 7f4ce5400640 1 --2- 172.21.15.72:0/785120440 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ce8102940 0x7f4ce819b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:21.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.701+0000 7f4ce6800640 1 --2- 172.21.15.72:0/785120440 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ce8103300 0x7f4ce81a2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:21.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.701+0000 7f4ce5e00640 1 -- 172.21.15.72:0/785120440 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ce8102940 msgr2=0x7f4ce819b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:21.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.701+0000 7f4ce5e00640 1 --2- 172.21.15.72:0/785120440 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ce8102940 0x7f4ce819b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:21.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.701+0000 7f4ce5e00640 1 -- 172.21.15.72:0/785120440 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ce8103300 msgr2=0x7f4ce81a2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:21.706 INFO:journalctl@ceph.mds.a.smithi072.stdout:Apr 27 00:01:21 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-a[68103]: ignoring --setuser ceph since I am not root 2024-04-27T00:01:21.707 INFO:journalctl@ceph.mds.a.smithi072.stdout:Apr 27 00:01:21 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-a[68103]: ignoring --setgroup ceph since I am not root 2024-04-27T00:01:21.707 INFO:journalctl@ceph.mds.a.smithi072.stdout:Apr 27 00:01:21 smithi072 ceph-mds[68107]: -- 172.21.15.72:0/2515698038 <== mon.0 v2:172.21.15.72:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56263b34fa40 con 0x56263b323000 2024-04-27T00:01:21.707 INFO:journalctl@ceph.mds.a.smithi072.stdout:Apr 27 00:01:21 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-a[68103]: starting mds.a at 2024-04-27T00:01:21.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.701+0000 7f4ce5e00640 1 --2- 172.21.15.72:0/785120440 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ce8103300 0x7f4ce81a2260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:21.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.701+0000 7f4ce5e00640 1 -- 172.21.15.72:0/785120440 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ce81a2900 con 0x7f4ce8101750 2024-04-27T00:01:21.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.701+0000 7f4ce5e00640 1 --2- 172.21.15.72:0/785120440 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4ce8101750 0x7f4ce819acf0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8006010 tx=0x7f4cd80042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:21.709 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:21 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mon-a[28180]: 2024-04-27T00:01:21.480+0000 7f5363200700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-27T00:01:21.709 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:21 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/572021347' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T00:01:21.709 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:21 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/1122408760' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-27T00:01:21.709 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:21 smithi072 ceph-mon[28184]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-27T00:01:21.709 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:21 smithi072 ceph-mon[28184]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-27T00:01:21.709 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:21 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/1122408760' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-27T00:01:21.709 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:21 smithi072 ceph-mon[28184]: osdmap e94: 12 total, 12 up, 12 in 2024-04-27T00:01:21.710 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:21 smithi072 ceph-mon[28184]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-27T00:01:21.710 INFO:journalctl@ceph.mds.g.smithi072.stdout:Apr 27 00:01:21 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-g[67714]: ignoring --setuser ceph since I am not root 2024-04-27T00:01:21.710 INFO:journalctl@ceph.mds.g.smithi072.stdout:Apr 27 00:01:21 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-g[67714]: ignoring --setgroup ceph since I am not root 2024-04-27T00:01:21.710 INFO:journalctl@ceph.mds.g.smithi072.stdout:Apr 27 00:01:21 smithi072 ceph-mds[67723]: -- 172.21.15.72:0/2125862623 <== mon.0 v2:172.21.15.72:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f5f77b5a40 con 0x55f5f7788800 2024-04-27T00:01:21.710 INFO:journalctl@ceph.mds.g.smithi072.stdout:Apr 27 00:01:21 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-g[67714]: starting mds.g at 2024-04-27T00:01:21.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.704+0000 7f4cdea00640 1 -- 172.21.15.72:0/785120440 <== mon.2 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd803f030 con 0x7f4ce8101750 2024-04-27T00:01:21.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.704+0000 7f4cdea00640 1 -- 172.21.15.72:0/785120440 <== mon.2 v2:172.21.15.76:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4cd8031910 con 0x7f4ce8101750 2024-04-27T00:01:21.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.704+0000 7f4cdea00640 1 -- 172.21.15.72:0/785120440 <== mon.2 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd8035070 con 0x7f4ce8101750 2024-04-27T00:01:21.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.704+0000 7f4ce7200640 1 -- 172.21.15.72:0/785120440 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ce81a2b90 con 0x7f4ce8101750 2024-04-27T00:01:21.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.704+0000 7f4ce7200640 1 -- 172.21.15.72:0/785120440 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ce81a3030 con 0x7f4ce8101750 2024-04-27T00:01:21.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.705+0000 7f4ce7200640 1 -- 172.21.15.72:0/785120440 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ce81a3200 con 0x7f4ce8101750 2024-04-27T00:01:21.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.707+0000 7f4cdea00640 1 -- 172.21.15.72:0/785120440 <== mon.2 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4cd8042030 con 0x7f4ce8101750 2024-04-27T00:01:21.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.708+0000 7f4cdea00640 1 --2- 172.21.15.72:0/785120440 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f4ccc0a3cc0 0x7f4ccc0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:21.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.708+0000 7f4cdea00640 1 -- 172.21.15.72:0/785120440 <== mon.2 v2:172.21.15.76:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f4cd80f10c0 con 0x7f4ce8101750 2024-04-27T00:01:21.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.709+0000 7f4ce5400640 1 --2- 172.21.15.72:0/785120440 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f4ccc0a3cc0 0x7f4ccc0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:21.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.709+0000 7f4ce5400640 1 --2- 172.21.15.72:0/785120440 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f4ccc0a3cc0 0x7f4ccc0a6180 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f4cd4009bd0 tx=0x7f4cd4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:21.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.713+0000 7f4cdea00640 1 -- 172.21.15.72:0/785120440 <== mon.2 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4cd80bd3f0 con 0x7f4ce8101750 2024-04-27T00:01:21.797 INFO:journalctl@ceph.mds.c.smithi136.stdout:Apr 27 00:01:21 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-c[62838]: ignoring --setuser ceph since I am not root 2024-04-27T00:01:21.798 INFO:journalctl@ceph.mds.c.smithi136.stdout:Apr 27 00:01:21 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-c[62838]: ignoring --setgroup ceph since I am not root 2024-04-27T00:01:21.798 INFO:journalctl@ceph.mds.c.smithi136.stdout:Apr 27 00:01:21 smithi136 ceph-mds[62842]: -- 172.21.15.136:0/1490973492 <== mon.1 v2:172.21.15.136:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5592757bda40 con 0x559275791000 2024-04-27T00:01:21.798 INFO:journalctl@ceph.mds.c.smithi136.stdout:Apr 27 00:01:21 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-c[62838]: starting mds.c at 2024-04-27T00:01:21.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:21 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/572021347' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T00:01:21.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:21 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/1122408760' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-27T00:01:21.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:21 smithi136 ceph-mon[33050]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-27T00:01:21.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:21 smithi136 ceph-mon[33050]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-27T00:01:21.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:21 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/1122408760' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-27T00:01:21.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:21 smithi136 ceph-mon[33050]: osdmap e94: 12 total, 12 up, 12 in 2024-04-27T00:01:21.799 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:21 smithi136 ceph-mon[33050]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-27T00:01:21.799 INFO:journalctl@ceph.mds.i.smithi136.stdout:Apr 27 00:01:21 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-i[63154]: ignoring --setuser ceph since I am not root 2024-04-27T00:01:21.799 INFO:journalctl@ceph.mds.i.smithi136.stdout:Apr 27 00:01:21 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-i[63154]: ignoring --setgroup ceph since I am not root 2024-04-27T00:01:21.800 INFO:journalctl@ceph.mds.i.smithi136.stdout:Apr 27 00:01:21 smithi136 ceph-mds[63158]: -- 172.21.15.136:0/445788234 <== mon.1 v2:172.21.15.136:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564cd71b3a40 con 0x564cd7186800 2024-04-27T00:01:21.800 INFO:journalctl@ceph.mds.i.smithi136.stdout:Apr 27 00:01:21 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-i[63154]: starting mds.i at 2024-04-27T00:01:21.926 INFO:journalctl@ceph.mds.h.smithi076.stdout:Apr 27 00:01:21 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-h[61930]: ignoring --setuser ceph since I am not root 2024-04-27T00:01:21.927 INFO:journalctl@ceph.mds.h.smithi076.stdout:Apr 27 00:01:21 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-h[61930]: ignoring --setgroup ceph since I am not root 2024-04-27T00:01:21.927 INFO:journalctl@ceph.mds.h.smithi076.stdout:Apr 27 00:01:21 smithi076 ceph-mds[61934]: -- 172.21.15.76:0/554589359 <== mon.2 v2:172.21.15.76:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c3e2a17a40 con 0x55c3e29eb000 2024-04-27T00:01:21.927 INFO:journalctl@ceph.mds.h.smithi076.stdout:Apr 27 00:01:21 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-h[61930]: starting mds.h at 2024-04-27T00:01:21.927 INFO:journalctl@ceph.mds.h.smithi076.stdout:Apr 27 00:01:21 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-h[61930]: 2024-04-27T00:01:21.528+0000 7fdd4e4c0700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6836/0: (98) Address already in use 2024-04-27T00:01:21.928 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:21 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/572021347' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-27T00:01:21.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:21 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/1122408760' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-27T00:01:21.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:21 smithi076 ceph-mon[32774]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-27T00:01:21.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:21 smithi076 ceph-mon[32774]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-27T00:01:21.929 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:21 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/1122408760' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-27T00:01:21.930 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:21 smithi076 ceph-mon[32774]: osdmap e94: 12 total, 12 up, 12 in 2024-04-27T00:01:21.930 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:21 smithi076 ceph-mon[32774]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-27T00:01:21.930 INFO:journalctl@ceph.mds.b.smithi076.stdout:Apr 27 00:01:21 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-b[62253]: ignoring --setuser ceph since I am not root 2024-04-27T00:01:21.930 INFO:journalctl@ceph.mds.b.smithi076.stdout:Apr 27 00:01:21 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-b[62253]: ignoring --setgroup ceph since I am not root 2024-04-27T00:01:21.931 INFO:journalctl@ceph.mds.b.smithi076.stdout:Apr 27 00:01:21 smithi076 ceph-mds[62257]: -- 172.21.15.76:0/1758043297 <== mon.2 v2:172.21.15.76:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564293aa1a40 con 0x564293a74c00 2024-04-27T00:01:21.931 INFO:journalctl@ceph.mds.b.smithi076.stdout:Apr 27 00:01:21 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-b[62253]: starting mds.b at 2024-04-27T00:01:21.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:21.954+0000 7f4ce7200640 1 -- 172.21.15.72:0/785120440 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f4ce81a33d0 con 0x7f4ce8101750 2024-04-27T00:01:22.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.489+0000 7f4cdea00640 1 -- 172.21.15.72:0/785120440 <== mon.2 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v26) ==== 104+0+0 (secure 0 0 0) 0x7f4cd80bccf0 con 0x7f4ce8101750 2024-04-27T00:01:22.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.493+0000 7f4ce7200640 1 -- 172.21.15.72:0/785120440 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f4ccc0a3cc0 msgr2=0x7f4ccc0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:22.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.493+0000 7f4ce7200640 1 --2- 172.21.15.72:0/785120440 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f4ccc0a3cc0 0x7f4ccc0a6180 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f4cd4009bd0 tx=0x7f4cd4005f90 comp rx=0 tx=0).stop 2024-04-27T00:01:22.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.493+0000 7f4ce7200640 1 -- 172.21.15.72:0/785120440 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4ce8101750 msgr2=0x7f4ce819acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:22.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.493+0000 7f4ce7200640 1 --2- 172.21.15.72:0/785120440 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4ce8101750 0x7f4ce819acf0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8006010 tx=0x7f4cd80042f0 comp rx=0 tx=0).stop 2024-04-27T00:01:22.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.494+0000 7f4ce7200640 1 -- 172.21.15.72:0/785120440 shutdown_connections 2024-04-27T00:01:22.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.494+0000 7f4ce7200640 1 --2- 172.21.15.72:0/785120440 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f4ccc0a3cc0 0x7f4ccc0a6180 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:22.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.494+0000 7f4ce7200640 1 --2- 172.21.15.72:0/785120440 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ce8103300 0x7f4ce81a2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:22.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.494+0000 7f4ce7200640 1 --2- 172.21.15.72:0/785120440 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ce8102940 0x7f4ce819b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:22.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.494+0000 7f4ce7200640 1 --2- 172.21.15.72:0/785120440 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4ce8101750 0x7f4ce819acf0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:22.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.494+0000 7f4ce7200640 1 -- 172.21.15.72:0/785120440 >> 172.21.15.72:0/785120440 conn(0x7f4ce80fcce0 msgr2=0x7f4ce806a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:22.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.494+0000 7f4ce7200640 1 -- 172.21.15.72:0/785120440 shutdown_connections 2024-04-27T00:01:22.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.494+0000 7f4ce7200640 1 -- 172.21.15.72:0/785120440 wait complete. 2024-04-27T00:01:22.515 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-27T00:01:22.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.702+0000 7f6d5ce00640 1 -- 172.21.15.72:0/354053007 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6d581085b0 msgr2=0x7f6d5810a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:22.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.702+0000 7f6d5ce00640 1 --2- 172.21.15.72:0/354053007 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6d581085b0 0x7f6d5810a9a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6d50009ca0 tx=0x7f6d5002a8c0 comp rx=0 tx=0).stop 2024-04-27T00:01:22.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.702+0000 7f6d5ce00640 1 -- 172.21.15.72:0/354053007 shutdown_connections 2024-04-27T00:01:22.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.702+0000 7f6d5ce00640 1 --2- 172.21.15.72:0/354053007 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6d581085b0 0x7f6d5810a9a0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:22.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.702+0000 7f6d5ce00640 1 --2- 172.21.15.72:0/354053007 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d58105c80 0x7f6d58108070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:22.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.702+0000 7f6d5ce00640 1 --2- 172.21.15.72:0/354053007 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6d58103350 0x7f6d58105740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:22.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.702+0000 7f6d5ce00640 1 -- 172.21.15.72:0/354053007 >> 172.21.15.72:0/354053007 conn(0x7f6d580fcca0 msgr2=0x7f6d580ff0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:22.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.703+0000 7f6d5ce00640 1 -- 172.21.15.72:0/354053007 shutdown_connections 2024-04-27T00:01:22.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.703+0000 7f6d5ce00640 1 -- 172.21.15.72:0/354053007 wait complete. 2024-04-27T00:01:22.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.704+0000 7f6d5ce00640 1 Processor -- start 2024-04-27T00:01:22.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.704+0000 7f6d5ce00640 1 -- start start 2024-04-27T00:01:22.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.704+0000 7f6d5ce00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d58103350 0x7f6d5819ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:22.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.704+0000 7f6d5ce00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6d58105c80 0x7f6d5819b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:22.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.704+0000 7f6d5ce00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6d581085b0 0x7f6d581a21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:22.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.704+0000 7f6d5ce00640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f6d5810e3c0 con 0x7f6d58103350 2024-04-27T00:01:22.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.704+0000 7f6d5ce00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f6d5810e240 con 0x7f6d58105c80 2024-04-27T00:01:22.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.704+0000 7f6d5ce00640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f6d5810e540 con 0x7f6d581085b0 2024-04-27T00:01:22.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.704+0000 7f6d57400640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d58103350 0x7f6d5819ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:22.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.705+0000 7f6d57400640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d58103350 0x7f6d5819ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43404/0 (socket says 172.21.15.72:43404) 2024-04-27T00:01:22.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.705+0000 7f6d57400640 1 -- 172.21.15.72:0/3972588868 learned_addr learned my addr 172.21.15.72:0/3972588868 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-27T00:01:22.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.705+0000 7f6d56a00640 1 --2- 172.21.15.72:0/3972588868 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6d58105c80 0x7f6d5819b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:22.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.705+0000 7f6d57e00640 1 --2- 172.21.15.72:0/3972588868 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6d581085b0 0x7f6d581a21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:22.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.705+0000 7f6d57400640 1 -- 172.21.15.72:0/3972588868 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6d581085b0 msgr2=0x7f6d581a21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:22.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.705+0000 7f6d57400640 1 --2- 172.21.15.72:0/3972588868 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6d581085b0 0x7f6d581a21a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:22.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.705+0000 7f6d57400640 1 -- 172.21.15.72:0/3972588868 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6d58105c80 msgr2=0x7f6d5819b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:22.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.705+0000 7f6d57400640 1 --2- 172.21.15.72:0/3972588868 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6d58105c80 0x7f6d5819b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:22.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.705+0000 7f6d57400640 1 -- 172.21.15.72:0/3972588868 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d581a2840 con 0x7f6d58103350 2024-04-27T00:01:22.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.705+0000 7f6d57400640 1 --2- 172.21.15.72:0/3972588868 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d58103350 0x7f6d5819ac30 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6d4800e9f0 tx=0x7f6d4800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:22.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.706+0000 7f6d4fe00640 1 -- 172.21.15.72:0/3972588868 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d4800cd80 con 0x7f6d58103350 2024-04-27T00:01:22.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.706+0000 7f6d5ce00640 1 -- 172.21.15.72:0/3972588868 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d581a2b30 con 0x7f6d58103350 2024-04-27T00:01:22.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.706+0000 7f6d4fe00640 1 -- 172.21.15.72:0/3972588868 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6d480054d0 con 0x7f6d58103350 2024-04-27T00:01:22.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.706+0000 7f6d4fe00640 1 -- 172.21.15.72:0/3972588868 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d48010690 con 0x7f6d58103350 2024-04-27T00:01:22.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.706+0000 7f6d5ce00640 1 -- 172.21.15.72:0/3972588868 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d581a3080 con 0x7f6d58103350 2024-04-27T00:01:22.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.707+0000 7f6d5ce00640 1 -- 172.21.15.72:0/3972588868 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d5810a0a0 con 0x7f6d58103350 2024-04-27T00:01:22.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.710+0000 7f6d4fe00640 1 -- 172.21.15.72:0/3972588868 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f6d48004e90 con 0x7f6d58103350 2024-04-27T00:01:22.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.712+0000 7f6d4fe00640 1 --2- 172.21.15.72:0/3972588868 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6d3c0a3cc0 0x7f6d3c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:22.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.712+0000 7f6d4fe00640 1 -- 172.21.15.72:0/3972588868 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9495+0+0 (secure 0 0 0) 0x7f6d480d0ea0 con 0x7f6d58103350 2024-04-27T00:01:22.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.712+0000 7f6d56a00640 1 --2- 172.21.15.72:0/3972588868 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6d3c0a3cc0 0x7f6d3c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:22.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.713+0000 7f6d56a00640 1 --2- 172.21.15.72:0/3972588868 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6d3c0a3cc0 0x7f6d3c0a6180 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f6d5806a560 tx=0x7f6d44005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:22.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.718+0000 7f6d4fe00640 1 -- 172.21.15.72:0/3972588868 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6d4809d230 con 0x7f6d58103350 2024-04-27T00:01:22.791 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:22 smithi072 ceph-mon[28184]: pgmap v874: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-27T00:01:22.791 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:22 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/785120440' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-27T00:01:22.791 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:22 smithi072 ceph-mon[28184]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-27T00:01:22.791 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:22 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:01:22.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:22 smithi072 ceph-mon[28184]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-27T00:01:22.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:22 smithi072 ceph-mon[28184]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-27T00:01:22.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:22 smithi072 ceph-mon[28184]: Cluster is now healthy 2024-04-27T00:01:22.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:22 smithi072 ceph-mon[28184]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-27T00:01:22.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:22 smithi072 ceph-mon[28184]: mds.? [v2:172.21.15.76:6837/3793834090,v1:172.21.15.76:6839/3793834090] up:boot 2024-04-27T00:01:22.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:22 smithi072 ceph-mon[28184]: mds.? [v2:172.21.15.76:6836/658096945,v1:172.21.15.76:6838/658096945] up:boot 2024-04-27T00:01:22.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:22 smithi072 ceph-mon[28184]: mds.? [v2:172.21.15.136:6836/3492391768,v1:172.21.15.136:6838/3492391768] up:boot 2024-04-27T00:01:22.792 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:22 smithi072 ceph-mon[28184]: mds.? [v2:172.21.15.136:6837/538718993,v1:172.21.15.136:6839/538718993] up:boot 2024-04-27T00:01:22.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:22 smithi072 ceph-mon[28184]: mds.? [v2:172.21.15.72:6838/1943593401,v1:172.21.15.72:6839/1943593401] up:boot 2024-04-27T00:01:22.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:22 smithi072 ceph-mon[28184]: mds.? [v2:172.21.15.72:6840/2227353510,v1:172.21.15.72:6841/2227353510] up:boot 2024-04-27T00:01:22.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:22 smithi072 ceph-mon[28184]: fsmap 12 up:standby 2024-04-27T00:01:22.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:22 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-27T00:01:22.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:22 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-27T00:01:22.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:22 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-27T00:01:22.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:22 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-27T00:01:22.793 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:22 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-27T00:01:22.794 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:22 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-27T00:01:22.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:22.913+0000 7f6d5ce00640 1 -- 172.21.15.72:0/3972588868 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f6d5802dad0 con 0x7f6d58103350 2024-04-27T00:01:23.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:22 smithi136 ceph-mon[33050]: pgmap v874: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-27T00:01:23.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:22 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/785120440' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-27T00:01:23.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:22 smithi136 ceph-mon[33050]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-27T00:01:23.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:22 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:01:23.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:22 smithi136 ceph-mon[33050]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-27T00:01:23.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:22 smithi136 ceph-mon[33050]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-27T00:01:23.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:22 smithi136 ceph-mon[33050]: Cluster is now healthy 2024-04-27T00:01:23.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:22 smithi136 ceph-mon[33050]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-27T00:01:23.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:22 smithi136 ceph-mon[33050]: mds.? [v2:172.21.15.76:6837/3793834090,v1:172.21.15.76:6839/3793834090] up:boot 2024-04-27T00:01:23.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:22 smithi136 ceph-mon[33050]: mds.? [v2:172.21.15.76:6836/658096945,v1:172.21.15.76:6838/658096945] up:boot 2024-04-27T00:01:23.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:22 smithi136 ceph-mon[33050]: mds.? [v2:172.21.15.136:6836/3492391768,v1:172.21.15.136:6838/3492391768] up:boot 2024-04-27T00:01:23.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:22 smithi136 ceph-mon[33050]: mds.? [v2:172.21.15.136:6837/538718993,v1:172.21.15.136:6839/538718993] up:boot 2024-04-27T00:01:23.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:22 smithi136 ceph-mon[33050]: mds.? [v2:172.21.15.72:6838/1943593401,v1:172.21.15.72:6839/1943593401] up:boot 2024-04-27T00:01:23.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:22 smithi136 ceph-mon[33050]: mds.? [v2:172.21.15.72:6840/2227353510,v1:172.21.15.72:6841/2227353510] up:boot 2024-04-27T00:01:23.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:22 smithi136 ceph-mon[33050]: fsmap 12 up:standby 2024-04-27T00:01:23.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:22 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-27T00:01:23.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:22 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-27T00:01:23.049 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:22 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-27T00:01:23.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:22 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-27T00:01:23.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:22 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-27T00:01:23.050 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:22 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-27T00:01:23.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:22 smithi076 ceph-mon[32774]: pgmap v874: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-27T00:01:23.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:22 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/785120440' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-27T00:01:23.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:22 smithi076 ceph-mon[32774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-27T00:01:23.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:22 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-27T00:01:23.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:22 smithi076 ceph-mon[32774]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-27T00:01:23.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:22 smithi076 ceph-mon[32774]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-27T00:01:23.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:22 smithi076 ceph-mon[32774]: Cluster is now healthy 2024-04-27T00:01:23.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:22 smithi076 ceph-mon[32774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-27T00:01:23.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:22 smithi076 ceph-mon[32774]: mds.? [v2:172.21.15.76:6837/3793834090,v1:172.21.15.76:6839/3793834090] up:boot 2024-04-27T00:01:23.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:22 smithi076 ceph-mon[32774]: mds.? [v2:172.21.15.76:6836/658096945,v1:172.21.15.76:6838/658096945] up:boot 2024-04-27T00:01:23.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:22 smithi076 ceph-mon[32774]: mds.? [v2:172.21.15.136:6836/3492391768,v1:172.21.15.136:6838/3492391768] up:boot 2024-04-27T00:01:23.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:22 smithi076 ceph-mon[32774]: mds.? [v2:172.21.15.136:6837/538718993,v1:172.21.15.136:6839/538718993] up:boot 2024-04-27T00:01:23.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:22 smithi076 ceph-mon[32774]: mds.? [v2:172.21.15.72:6838/1943593401,v1:172.21.15.72:6839/1943593401] up:boot 2024-04-27T00:01:23.178 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:22 smithi076 ceph-mon[32774]: mds.? [v2:172.21.15.72:6840/2227353510,v1:172.21.15.72:6841/2227353510] up:boot 2024-04-27T00:01:23.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:22 smithi076 ceph-mon[32774]: fsmap 12 up:standby 2024-04-27T00:01:23.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:22 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-27T00:01:23.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:22 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-27T00:01:23.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:22 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-27T00:01:23.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:22 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-27T00:01:23.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:22 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-27T00:01:23.179 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:22 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-27T00:01:23.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.718+0000 7f6d4fe00640 1 -- 172.21.15.72:0/3972588868 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v95) ==== 181+0+0 (secure 0 0 0) 0x7f6d4809cb30 con 0x7f6d58103350 2024-04-27T00:01:23.720 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_metadata' removed 2024-04-27T00:01:23.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.723+0000 7f6d5ce00640 1 -- 172.21.15.72:0/3972588868 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6d3c0a3cc0 msgr2=0x7f6d3c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:23.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.723+0000 7f6d5ce00640 1 --2- 172.21.15.72:0/3972588868 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6d3c0a3cc0 0x7f6d3c0a6180 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f6d5806a560 tx=0x7f6d44005f90 comp rx=0 tx=0).stop 2024-04-27T00:01:23.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.723+0000 7f6d5ce00640 1 -- 172.21.15.72:0/3972588868 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d58103350 msgr2=0x7f6d5819ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:23.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.723+0000 7f6d5ce00640 1 --2- 172.21.15.72:0/3972588868 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d58103350 0x7f6d5819ac30 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6d4800e9f0 tx=0x7f6d4800edf0 comp rx=0 tx=0).stop 2024-04-27T00:01:23.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.723+0000 7f6d5ce00640 1 -- 172.21.15.72:0/3972588868 shutdown_connections 2024-04-27T00:01:23.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.723+0000 7f6d5ce00640 1 --2- 172.21.15.72:0/3972588868 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f6d3c0a3cc0 0x7f6d3c0a6180 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:23.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.723+0000 7f6d5ce00640 1 --2- 172.21.15.72:0/3972588868 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6d58105c80 0x7f6d5819b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:23.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.723+0000 7f6d5ce00640 1 --2- 172.21.15.72:0/3972588868 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6d581085b0 0x7f6d581a21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:23.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.723+0000 7f6d5ce00640 1 --2- 172.21.15.72:0/3972588868 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d58103350 0x7f6d5819ac30 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:23.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.723+0000 7f6d5ce00640 1 -- 172.21.15.72:0/3972588868 >> 172.21.15.72:0/3972588868 conn(0x7f6d580fcca0 msgr2=0x7f6d58102a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:23.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.724+0000 7f6d5ce00640 1 -- 172.21.15.72:0/3972588868 shutdown_connections 2024-04-27T00:01:23.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.724+0000 7f6d5ce00640 1 -- 172.21.15.72:0/3972588868 wait complete. 2024-04-27T00:01:23.752 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-27T00:01:23.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.940+0000 7f2d2a000640 1 -- 172.21.15.72:0/4024267789 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d24103b60 msgr2=0x7f2d24103fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:23.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.940+0000 7f2d2a000640 1 --2- 172.21.15.72:0/4024267789 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d24103b60 0x7f2d24103fe0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f2d10009900 tx=0x7f2d1002a410 comp rx=0 tx=0).stop 2024-04-27T00:01:23.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.940+0000 7f2d2a000640 1 -- 172.21.15.72:0/4024267789 shutdown_connections 2024-04-27T00:01:23.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.940+0000 7f2d2a000640 1 --2- 172.21.15.72:0/4024267789 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d24104520 0x7f2d2410adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:23.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.940+0000 7f2d2a000640 1 --2- 172.21.15.72:0/4024267789 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d24103b60 0x7f2d24103fe0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:23.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.940+0000 7f2d2a000640 1 --2- 172.21.15.72:0/4024267789 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d24102970 0x7f2d24102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:23.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.940+0000 7f2d2a000640 1 -- 172.21.15.72:0/4024267789 >> 172.21.15.72:0/4024267789 conn(0x7f2d240fdee0 msgr2=0x7f2d24100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:23.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.940+0000 7f2d2a000640 1 -- 172.21.15.72:0/4024267789 shutdown_connections 2024-04-27T00:01:23.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.940+0000 7f2d2a000640 1 -- 172.21.15.72:0/4024267789 wait complete. 2024-04-27T00:01:23.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.941+0000 7f2d2a000640 1 Processor -- start 2024-04-27T00:01:23.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.941+0000 7f2d2a000640 1 -- start start 2024-04-27T00:01:23.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.941+0000 7f2d2a000640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d24102970 0x7f2d240766e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:23.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.941+0000 7f2d2a000640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d24103b60 0x7f2d24074d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:23.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.941+0000 7f2d2a000640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d24104520 0x7f2d24075260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:23.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.941+0000 7f2d2a000640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f2d2410e4a0 con 0x7f2d24102970 2024-04-27T00:01:23.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.941+0000 7f2d2a000640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f2d2410e320 con 0x7f2d24104520 2024-04-27T00:01:23.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.941+0000 7f2d2a000640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f2d2410e620 con 0x7f2d24103b60 2024-04-27T00:01:23.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.941+0000 7f2d23400640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d24102970 0x7f2d240766e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:23.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.941+0000 7f2d23e00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d24104520 0x7f2d24075260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:23.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.941+0000 7f2d23400640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d24102970 0x7f2d240766e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43414/0 (socket says 172.21.15.72:43414) 2024-04-27T00:01:23.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.941+0000 7f2d23400640 1 -- 172.21.15.72:0/1019119186 learned_addr learned my addr 172.21.15.72:0/1019119186 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-27T00:01:23.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.942+0000 7f2d22a00640 1 --2- 172.21.15.72:0/1019119186 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d24103b60 0x7f2d24074d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:23.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.942+0000 7f2d23400640 1 -- 172.21.15.72:0/1019119186 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d24103b60 msgr2=0x7f2d24074d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:23.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.942+0000 7f2d23400640 1 --2- 172.21.15.72:0/1019119186 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d24103b60 0x7f2d24074d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:23.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.942+0000 7f2d23400640 1 -- 172.21.15.72:0/1019119186 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d24104520 msgr2=0x7f2d24075260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:23.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.942+0000 7f2d23400640 1 --2- 172.21.15.72:0/1019119186 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d24104520 0x7f2d24075260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:23.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.942+0000 7f2d23400640 1 -- 172.21.15.72:0/1019119186 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d24075af0 con 0x7f2d24102970 2024-04-27T00:01:23.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.942+0000 7f2d23e00640 1 --2- 172.21.15.72:0/1019119186 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d24104520 0x7f2d24075260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-27T00:01:23.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.942+0000 7f2d23400640 1 --2- 172.21.15.72:0/1019119186 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d24102970 0x7f2d240766e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f2d1400e9c0 tx=0x7f2d1400edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:23.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.942+0000 7f2d1be00640 1 -- 172.21.15.72:0/1019119186 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d1400ccc0 con 0x7f2d24102970 2024-04-27T00:01:23.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.942+0000 7f2d1be00640 1 -- 172.21.15.72:0/1019119186 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d140035f0 con 0x7f2d24102970 2024-04-27T00:01:23.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.942+0000 7f2d1be00640 1 -- 172.21.15.72:0/1019119186 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d1400ce60 con 0x7f2d24102970 2024-04-27T00:01:23.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.942+0000 7f2d2a000640 1 -- 172.21.15.72:0/1019119186 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d241a2d20 con 0x7f2d24102970 2024-04-27T00:01:23.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.943+0000 7f2d2a000640 1 -- 172.21.15.72:0/1019119186 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d241a31c0 con 0x7f2d24102970 2024-04-27T00:01:23.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.944+0000 7f2d2a000640 1 -- 172.21.15.72:0/1019119186 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d24109e00 con 0x7f2d24102970 2024-04-27T00:01:23.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.946+0000 7f2d1be00640 1 -- 172.21.15.72:0/1019119186 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f2d14004000 con 0x7f2d24102970 2024-04-27T00:01:23.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.947+0000 7f2d1be00640 1 --2- 172.21.15.72:0/1019119186 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f2d080a3cc0 0x7f2d080a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:23.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.947+0000 7f2d1be00640 1 -- 172.21.15.72:0/1019119186 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8970+0+0 (secure 0 0 0) 0x7f2d140d0e00 con 0x7f2d24102970 2024-04-27T00:01:23.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.947+0000 7f2d22a00640 1 --2- 172.21.15.72:0/1019119186 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f2d080a3cc0 0x7f2d080a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:23.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.948+0000 7f2d22a00640 1 --2- 172.21.15.72:0/1019119186 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f2d080a3cc0 0x7f2d080a6180 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f2d24076450 tx=0x7f2d100314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:23.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:23.952+0000 7f2d1be00640 1 -- 172.21.15.72:0/1019119186 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f2d1409d340 con 0x7f2d24102970 2024-04-27T00:01:24.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:23 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/3972588868' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T00:01:24.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:23 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T00:01:24.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:23 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T00:01:24.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:23 smithi072 ceph-mon[28184]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-27T00:01:24.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:23 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/3972588868' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T00:01:24.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:23 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T00:01:24.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:23 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T00:01:24.048 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:23 smithi136 ceph-mon[33050]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-27T00:01:24.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.157+0000 7f2d2a000640 1 -- 172.21.15.72:0/1019119186 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f2d240635d0 con 0x7f2d24102970 2024-04-27T00:01:24.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:23 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/3972588868' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T00:01:24.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:23 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-27T00:01:24.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:23 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-27T00:01:24.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:23 smithi076 ceph-mon[32774]: from='mgr.14152 172.21.15.72:0/3601006557' entity='mgr.x' 2024-04-27T00:01:24.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.731+0000 7f2d1be00640 1 -- 172.21.15.72:0/1019119186 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v96) ==== 169+0+0 (secure 0 0 0) 0x7f2d1409cc40 con 0x7f2d24102970 2024-04-27T00:01:24.733 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_data' removed 2024-04-27T00:01:24.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.736+0000 7f2d2a000640 1 -- 172.21.15.72:0/1019119186 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f2d080a3cc0 msgr2=0x7f2d080a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:24.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.736+0000 7f2d2a000640 1 --2- 172.21.15.72:0/1019119186 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f2d080a3cc0 0x7f2d080a6180 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f2d24076450 tx=0x7f2d100314a0 comp rx=0 tx=0).stop 2024-04-27T00:01:24.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.736+0000 7f2d2a000640 1 -- 172.21.15.72:0/1019119186 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d24102970 msgr2=0x7f2d240766e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:24.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.736+0000 7f2d2a000640 1 --2- 172.21.15.72:0/1019119186 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d24102970 0x7f2d240766e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f2d1400e9c0 tx=0x7f2d1400edc0 comp rx=0 tx=0).stop 2024-04-27T00:01:24.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.737+0000 7f2d2a000640 1 -- 172.21.15.72:0/1019119186 shutdown_connections 2024-04-27T00:01:24.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.737+0000 7f2d2a000640 1 --2- 172.21.15.72:0/1019119186 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f2d080a3cc0 0x7f2d080a6180 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:24.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.737+0000 7f2d2a000640 1 --2- 172.21.15.72:0/1019119186 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2d24104520 0x7f2d24075260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:24.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.737+0000 7f2d2a000640 1 --2- 172.21.15.72:0/1019119186 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d24103b60 0x7f2d24074d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:24.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.737+0000 7f2d2a000640 1 --2- 172.21.15.72:0/1019119186 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d24102970 0x7f2d240766e0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:24.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.737+0000 7f2d2a000640 1 -- 172.21.15.72:0/1019119186 >> 172.21.15.72:0/1019119186 conn(0x7f2d240fdee0 msgr2=0x7f2d24106fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:24.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.737+0000 7f2d2a000640 1 -- 172.21.15.72:0/1019119186 shutdown_connections 2024-04-27T00:01:24.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.738+0000 7f2d2a000640 1 -- 172.21.15.72:0/1019119186 wait complete. 2024-04-27T00:01:24.762 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-04-27T00:01:24.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.956+0000 7f02cbe00640 1 -- 172.21.15.72:0/3644449184 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f02c4105d80 msgr2=0x7f02c4106200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:24.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.956+0000 7f02cbe00640 1 --2- 172.21.15.72:0/3644449184 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f02c4105d80 0x7f02c4106200 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f02b0009860 tx=0x7f02b002a340 comp rx=0 tx=0).stop 2024-04-27T00:01:24.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.957+0000 7f02cbe00640 1 -- 172.21.15.72:0/3644449184 shutdown_connections 2024-04-27T00:01:24.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.957+0000 7f02cbe00640 1 --2- 172.21.15.72:0/3644449184 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f02c4105d80 0x7f02c4106200 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:24.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.957+0000 7f02cbe00640 1 --2- 172.21.15.72:0/3644449184 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02c4106740 0x7f02c410cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:24.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.957+0000 7f02cbe00640 1 --2- 172.21.15.72:0/3644449184 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f02c4104b90 0x7f02c4104fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:24.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.957+0000 7f02cbe00640 1 -- 172.21.15.72:0/3644449184 >> 172.21.15.72:0/3644449184 conn(0x7f02c4100120 msgr2=0x7f02c4102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:24.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.957+0000 7f02cbe00640 1 -- 172.21.15.72:0/3644449184 shutdown_connections 2024-04-27T00:01:24.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.957+0000 7f02cbe00640 1 -- 172.21.15.72:0/3644449184 wait complete. 2024-04-27T00:01:24.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.958+0000 7f02cbe00640 1 Processor -- start 2024-04-27T00:01:24.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.958+0000 7f02cbe00640 1 -- start start 2024-04-27T00:01:24.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.959+0000 7f02cbe00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02c4104b90 0x7f02c419adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:24.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.959+0000 7f02cbe00640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f02c4105d80 0x7f02c419b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:24.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.959+0000 7f02cbe00640 1 --2- >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f02c4106740 0x7f02c41a2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:24.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.959+0000 7f02cbe00640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f02c41106a0 con 0x7f02c4104b90 2024-04-27T00:01:24.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.959+0000 7f02cbe00640 1 -- --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 -- 0x7f02c4110520 con 0x7f02c4106740 2024-04-27T00:01:24.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.959+0000 7f02cbe00640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 -- 0x7f02c4110820 con 0x7f02c4105d80 2024-04-27T00:01:24.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.959+0000 7f02c9400640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02c4104b90 0x7f02c419adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:24.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.959+0000 7f02c9400640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02c4104b90 0x7f02c419adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43422/0 (socket says 172.21.15.72:43422) 2024-04-27T00:01:24.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.959+0000 7f02c9400640 1 -- 172.21.15.72:0/3935225276 learned_addr learned my addr 172.21.15.72:0/3935225276 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-27T00:01:24.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.959+0000 7f02c9e00640 1 --2- 172.21.15.72:0/3935225276 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f02c4106740 0x7f02c41a2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:24.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.959+0000 7f02c8a00640 1 --2- 172.21.15.72:0/3935225276 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f02c4105d80 0x7f02c419b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:24.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.959+0000 7f02c9400640 1 -- 172.21.15.72:0/3935225276 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f02c4105d80 msgr2=0x7f02c419b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:24.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.959+0000 7f02c9400640 1 --2- 172.21.15.72:0/3935225276 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f02c4105d80 0x7f02c419b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:24.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.959+0000 7f02c9400640 1 -- 172.21.15.72:0/3935225276 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f02c4106740 msgr2=0x7f02c41a2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:24.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.959+0000 7f02c9400640 1 --2- 172.21.15.72:0/3935225276 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f02c4106740 0x7f02c41a2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:24.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.959+0000 7f02c9400640 1 -- 172.21.15.72:0/3935225276 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02c41a29e0 con 0x7f02c4104b90 2024-04-27T00:01:24.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.959+0000 7f02c8a00640 1 --2- 172.21.15.72:0/3935225276 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f02c4105d80 0x7f02c419b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-27T00:01:24.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.960+0000 7f02c9400640 1 --2- 172.21.15.72:0/3935225276 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02c4104b90 0x7f02c419adc0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f02b800e9a0 tx=0x7f02b800eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:24.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.960+0000 7f02c2000640 1 -- 172.21.15.72:0/3935225276 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02b800ccc0 con 0x7f02c4104b90 2024-04-27T00:01:24.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.960+0000 7f02c2000640 1 -- 172.21.15.72:0/3935225276 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02b80108e0 con 0x7f02c4104b90 2024-04-27T00:01:24.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.960+0000 7f02cbe00640 1 -- 172.21.15.72:0/3935225276 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02c41a2cd0 con 0x7f02c4104b90 2024-04-27T00:01:24.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.960+0000 7f02c2000640 1 -- 172.21.15.72:0/3935225276 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02b8018980 con 0x7f02c4104b90 2024-04-27T00:01:24.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.960+0000 7f02cbe00640 1 -- 172.21.15.72:0/3935225276 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02c41a31a0 con 0x7f02c4104b90 2024-04-27T00:01:24.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.961+0000 7f02cbe00640 1 -- 172.21.15.72:0/3935225276 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02c410bf80 con 0x7f02c4104b90 2024-04-27T00:01:24.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.963+0000 7f02c2000640 1 -- 172.21.15.72:0/3935225276 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f02b801f020 con 0x7f02c4104b90 2024-04-27T00:01:24.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.965+0000 7f02c2000640 1 --2- 172.21.15.72:0/3935225276 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f02ac0a3c40 0x7f02ac0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-27T00:01:24.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.965+0000 7f02c2000640 1 -- 172.21.15.72:0/3935225276 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8324+0+0 (secure 0 0 0) 0x7f02b80d07b0 con 0x7f02c4104b90 2024-04-27T00:01:24.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.965+0000 7f02c8a00640 1 --2- 172.21.15.72:0/3935225276 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f02ac0a3c40 0x7f02ac0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-27T00:01:24.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.966+0000 7f02c8a00640 1 --2- 172.21.15.72:0/3935225276 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f02ac0a3c40 0x7f02ac0a6100 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f02b002a930 tx=0x7f02b0030450 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-27T00:01:24.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:24.969+0000 7f02c2000640 1 -- 172.21.15.72:0/3935225276 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f02b809cf70 con 0x7f02c4104b90 2024-04-27T00:01:25.040 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:24 smithi072 ceph-mon[28184]: pgmap v876: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2024-04-27T00:01:25.040 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:24 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/3972588868' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-27T00:01:25.040 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:24 smithi072 ceph-mon[28184]: osdmap e95: 12 total, 12 up, 12 in 2024-04-27T00:01:25.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:24 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/1019119186' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T00:01:25.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:24 smithi136 ceph-mon[33050]: pgmap v876: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2024-04-27T00:01:25.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:24 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/3972588868' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-27T00:01:25.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:24 smithi136 ceph-mon[33050]: osdmap e95: 12 total, 12 up, 12 in 2024-04-27T00:01:25.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:24 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/1019119186' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T00:01:25.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:24 smithi076 ceph-mon[32774]: pgmap v876: 129 pgs: 129 active+clean; 639 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2024-04-27T00:01:25.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:24 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/3972588868' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-27T00:01:25.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:24 smithi076 ceph-mon[32774]: osdmap e95: 12 total, 12 up, 12 in 2024-04-27T00:01:25.177 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:24 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/1019119186' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T00:01:25.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:25.183+0000 7f02cbe00640 1 -- 172.21.15.72:0/3935225276 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f02c402dad0 con 0x7f02c4104b90 2024-04-27T00:01:25.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:25.742+0000 7f02c2000640 1 -- 172.21.15.72:0/3935225276 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v97) ==== 178+0+0 (secure 0 0 0) 0x7f02b809c870 con 0x7f02c4104b90 2024-04-27T00:01:25.746 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_data_ec' removed 2024-04-27T00:01:25.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:25.749+0000 7f02cbe00640 1 -- 172.21.15.72:0/3935225276 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f02ac0a3c40 msgr2=0x7f02ac0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:25.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:25.749+0000 7f02cbe00640 1 --2- 172.21.15.72:0/3935225276 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f02ac0a3c40 0x7f02ac0a6100 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f02b002a930 tx=0x7f02b0030450 comp rx=0 tx=0).stop 2024-04-27T00:01:25.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:25.749+0000 7f02cbe00640 1 -- 172.21.15.72:0/3935225276 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02c4104b90 msgr2=0x7f02c419adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-27T00:01:25.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:25.749+0000 7f02cbe00640 1 --2- 172.21.15.72:0/3935225276 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02c4104b90 0x7f02c419adc0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f02b800e9a0 tx=0x7f02b800eda0 comp rx=0 tx=0).stop 2024-04-27T00:01:25.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:25.749+0000 7f02cbe00640 1 -- 172.21.15.72:0/3935225276 shutdown_connections 2024-04-27T00:01:25.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:25.749+0000 7f02cbe00640 1 --2- 172.21.15.72:0/3935225276 >> [v2:172.21.15.72:6800/4281177709,v1:172.21.15.72:6801/4281177709] conn(0x7f02ac0a3c40 0x7f02ac0a6100 secure :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f02b002a930 tx=0x7f02b0030450 comp rx=0 tx=0).stop 2024-04-27T00:01:25.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:25.749+0000 7f02cbe00640 1 --2- 172.21.15.72:0/3935225276 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f02c4106740 0x7f02c41a2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:25.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:25.749+0000 7f02cbe00640 1 --2- 172.21.15.72:0/3935225276 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f02c4105d80 0x7f02c419b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:25.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:25.749+0000 7f02cbe00640 1 --2- 172.21.15.72:0/3935225276 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02c4104b90 0x7f02c419adc0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-27T00:01:25.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:25.749+0000 7f02cbe00640 1 -- 172.21.15.72:0/3935225276 >> 172.21.15.72:0/3935225276 conn(0x7f02c4100120 msgr2=0x7f02c4109150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-27T00:01:25.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:25.750+0000 7f02cbe00640 1 -- 172.21.15.72:0/3935225276 shutdown_connections 2024-04-27T00:01:25.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-27T00:01:25.750+0000 7f02cbe00640 1 -- 172.21.15.72:0/3935225276 wait complete. 2024-04-27T00:01:25.776 DEBUG:teuthology.orchestra.run.smithi072:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-27T00:01:25.843 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-27T00:01:25.875 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-27T00:01:25.906 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-27T00:01:25.907 DEBUG:teuthology.orchestra.run.smithi072:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-27T00:01:25.962 DEBUG:teuthology.orchestra.run.smithi076:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-27T00:01:25.979 DEBUG:teuthology.orchestra.run.smithi136:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-27T00:01:25.996 INFO:tasks.cephadm:Stopping all daemons... 2024-04-27T00:01:25.997 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-27T00:01:25.997 DEBUG:teuthology.orchestra.run.smithi072:> sudo systemctl stop ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mon.a 2024-04-27T00:01:26.040 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:25 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/1019119186' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-27T00:01:26.040 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:25 smithi072 ceph-mon[28184]: osdmap e96: 12 total, 12 up, 12 in 2024-04-27T00:01:26.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:25 smithi072 ceph-mon[28184]: from='client.? 172.21.15.72:0/3935225276' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T00:01:26.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:25 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/1019119186' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-27T00:01:26.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:25 smithi136 ceph-mon[33050]: osdmap e96: 12 total, 12 up, 12 in 2024-04-27T00:01:26.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:25 smithi136 ceph-mon[33050]: from='client.? 172.21.15.72:0/3935225276' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T00:01:26.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:25 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/1019119186' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-27T00:01:26.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:25 smithi076 ceph-mon[32774]: osdmap e96: 12 total, 12 up, 12 in 2024-04-27T00:01:26.176 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:25 smithi076 ceph-mon[32774]: from='client.? 172.21.15.72:0/3935225276' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-27T00:01:26.542 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:26 smithi072 systemd[1]: Stopping Ceph mon.a for 3c8453ae-0425-11ef-bc93-c7b262605968... 2024-04-27T00:01:27.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:26 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mon-a[28180]: 2024-04-27T00:01:26.558+0000 7f536a000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:01:27.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:26 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mon-a[28180]: 2024-04-27T00:01:26.558+0000 7f536a000700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-27T00:01:27.041 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:26 smithi072 podman[84590]: 2024-04-27 00:01:26.755612767 +0000 UTC m=+0.360967014 container died a100b14c4c0236a1b57ef9f0e10d64dcbfd011e79faf68b0a303924f27d2349b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mon-a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS) 2024-04-27T00:01:27.791 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:27 smithi072 podman[84590]: 2024-04-27 00:01:27.369575433 +0000 UTC m=+0.974929670 container cleanup a100b14c4c0236a1b57ef9f0e10d64dcbfd011e79faf68b0a303924f27d2349b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mon-a, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD) 2024-04-27T00:01:27.791 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:27 smithi072 bash[84590]: ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mon-a 2024-04-27T00:01:27.791 INFO:journalctl@ceph.mon.a.smithi072.stdout:Apr 27 00:01:27 smithi072 podman[84604]: 2024-04-27 00:01:27.582997679 +0000 UTC m=+0.807110013 container remove a100b14c4c0236a1b57ef9f0e10d64dcbfd011e79faf68b0a303924f27d2349b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mon-a, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-27T00:01:28.048 DEBUG:teuthology.orchestra.run.smithi072:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mon.a.service' 2024-04-27T00:01:28.218 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:01:28.218 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-27T00:01:28.219 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-27T00:01:28.219 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mon.b 2024-04-27T00:01:28.534 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:28 smithi076 systemd[1]: Stopping Ceph mon.b for 3c8453ae-0425-11ef-bc93-c7b262605968... 2024-04-27T00:01:28.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:28 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mon-b[32770]: 2024-04-27T00:01:28.532+0000 7f0e4bab4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:01:28.926 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:28 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mon-b[32770]: 2024-04-27T00:01:28.532+0000 7f0e4bab4700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-27T00:01:28.927 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:28 smithi076 podman[66763]: 2024-04-27 00:01:28.706258895 +0000 UTC m=+0.303356534 container died 470c048cdec7ab5d3ee3267ed1b3b765f829a6135ebe2e39b308013a6ff9f05c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mon-b, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-27T00:01:29.426 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:29 smithi076 podman[66763]: 2024-04-27 00:01:29.233289723 +0000 UTC m=+0.830387413 container cleanup 470c048cdec7ab5d3ee3267ed1b3b765f829a6135ebe2e39b308013a6ff9f05c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mon-b, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-27T00:01:29.427 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:29 smithi076 bash[66763]: ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mon-b 2024-04-27T00:01:29.717 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mon.b.service' 2024-04-27T00:01:29.737 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:29 smithi076 podman[66777]: 2024-04-27 00:01:29.468214442 +0000 UTC m=+0.762557424 container remove 470c048cdec7ab5d3ee3267ed1b3b765f829a6135ebe2e39b308013a6ff9f05c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5) 2024-04-27T00:01:29.737 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:29 smithi076 systemd[1]: ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mon.b.service: Deactivated successfully. 2024-04-27T00:01:29.737 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:29 smithi076 systemd[1]: Stopped Ceph mon.b for 3c8453ae-0425-11ef-bc93-c7b262605968. 2024-04-27T00:01:29.737 INFO:journalctl@ceph.mon.b.smithi076.stdout:Apr 27 00:01:29 smithi076 systemd[1]: ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mon.b.service: Consumed 14.892s CPU time. 2024-04-27T00:01:29.763 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:01:29.764 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-27T00:01:29.764 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-27T00:01:29.764 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mon.c 2024-04-27T00:01:30.047 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:29 smithi136 systemd[1]: Stopping Ceph mon.c for 3c8453ae-0425-11ef-bc93-c7b262605968... 2024-04-27T00:01:30.442 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:30 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mon-c[33033]: 2024-04-27T00:01:30.074+0000 7fa37e530700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:01:30.443 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:30 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mon-c[33033]: 2024-04-27T00:01:30.074+0000 7fa37e530700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-27T00:01:30.443 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:30 smithi136 podman[67562]: 2024-04-27 00:01:30.156521955 +0000 UTC m=+0.201078941 container died dcf9bcf7ae9cc96e6a1c35346fd3c5c540d8d994a5083f1523ae91ab0ae4429c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mon-c, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-27T00:01:30.797 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:30 smithi136 podman[67562]: 2024-04-27 00:01:30.642140899 +0000 UTC m=+0.686697886 container cleanup dcf9bcf7ae9cc96e6a1c35346fd3c5c540d8d994a5083f1523ae91ab0ae4429c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mon-c, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-27T00:01:30.798 INFO:journalctl@ceph.mon.c.smithi136.stdout:Apr 27 00:01:30 smithi136 bash[67562]: ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mon-c 2024-04-27T00:01:31.005 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mon.c.service' 2024-04-27T00:01:31.058 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:01:31.059 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-27T00:01:31.059 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-27T00:01:31.059 DEBUG:teuthology.orchestra.run.smithi072:> sudo systemctl stop ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mgr.x 2024-04-27T00:01:31.541 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 27 00:01:31 smithi072 systemd[1]: Stopping Ceph mgr.x for 3c8453ae-0425-11ef-bc93-c7b262605968... 2024-04-27T00:01:32.011 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 27 00:01:31 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-27T00:01:31.576+0000 7f6f52200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:01:32.011 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 27 00:01:31 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x[28410]: 2024-04-27T00:01:31.576+0000 7f6f52200700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-27T00:01:32.011 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 27 00:01:31 smithi072 podman[84711]: 2024-04-27 00:01:31.694403854 +0000 UTC m=+0.263743964 container died bf3f10c9dfc7bd1ca4360ee6d8f423b51a4151a8408504c678a0165747df2e6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True) 2024-04-27T00:01:32.291 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 27 00:01:32 smithi072 podman[84711]: 2024-04-27 00:01:32.169806629 +0000 UTC m=+0.739146784 container cleanup bf3f10c9dfc7bd1ca4360ee6d8f423b51a4151a8408504c678a0165747df2e6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-27T00:01:32.291 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 27 00:01:32 smithi072 bash[84711]: ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x 2024-04-27T00:01:32.791 INFO:journalctl@ceph.mgr.x.smithi072.stdout:Apr 27 00:01:32 smithi072 podman[84725]: 2024-04-27 00:01:32.449634154 +0000 UTC m=+0.746817972 container remove bf3f10c9dfc7bd1ca4360ee6d8f423b51a4151a8408504c678a0165747df2e6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-x, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424) 2024-04-27T00:01:32.905 DEBUG:teuthology.orchestra.run.smithi072:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-27T00:01:33.011 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:01:33.011 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-27T00:01:33.012 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-27T00:01:33.012 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mgr.y 2024-04-27T00:01:33.426 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Apr 27 00:01:33 smithi076 systemd[1]: Stopping Ceph mgr.y for 3c8453ae-0425-11ef-bc93-c7b262605968... 2024-04-27T00:01:33.906 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Apr 27 00:01:33 smithi076 podman[66887]: 2024-04-27 00:01:33.575124096 +0000 UTC m=+0.148822319 container died d9487d6301b771feed4ac2949c89c78b484986dc91858d334e02cde4cfcf3bec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-y, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424) 2024-04-27T00:01:34.176 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Apr 27 00:01:34 smithi076 podman[66887]: 2024-04-27 00:01:34.013339078 +0000 UTC m=+0.587037303 container cleanup d9487d6301b771feed4ac2949c89c78b484986dc91858d334e02cde4cfcf3bec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-y, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True) 2024-04-27T00:01:34.176 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Apr 27 00:01:34 smithi076 bash[66887]: ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-y 2024-04-27T00:01:34.477 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-27T00:01:34.511 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:01:34.511 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-27T00:01:34.511 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-27T00:01:34.511 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mgr.z 2024-04-27T00:01:34.797 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 27 00:01:34 smithi136 systemd[1]: Stopping Ceph mgr.z for 3c8453ae-0425-11ef-bc93-c7b262605968... 2024-04-27T00:01:35.156 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 27 00:01:34 smithi136 podman[67687]: 2024-04-27 00:01:34.859117432 +0000 UTC m=+0.159319412 container died b4eaa4a5921344176d754f359f1bdd2a459792f696ae258c2e3c22abc1888ba3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-z, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0) 2024-04-27T00:01:35.457 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 27 00:01:35 smithi136 podman[67687]: 2024-04-27 00:01:35.272032316 +0000 UTC m=+0.572234303 container cleanup b4eaa4a5921344176d754f359f1bdd2a459792f696ae258c2e3c22abc1888ba3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-z, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-27T00:01:35.457 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 27 00:01:35 smithi136 bash[67687]: ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-z 2024-04-27T00:01:35.699 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mgr.z.service' 2024-04-27T00:01:35.719 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 27 00:01:35 smithi136 podman[67701]: 2024-04-27 00:01:35.456835886 +0000 UTC m=+0.594403318 container remove b4eaa4a5921344176d754f359f1bdd2a459792f696ae258c2e3c22abc1888ba3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mgr-z, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-27T00:01:35.720 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 27 00:01:35 smithi136 systemd[1]: ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mgr.z.service: Main process exited, code=exited, status=143/n/a 2024-04-27T00:01:35.720 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 27 00:01:35 smithi136 systemd[1]: ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mgr.z.service: Failed with result 'exit-code'. 2024-04-27T00:01:35.720 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 27 00:01:35 smithi136 systemd[1]: Stopped Ceph mgr.z for 3c8453ae-0425-11ef-bc93-c7b262605968. 2024-04-27T00:01:35.720 INFO:journalctl@ceph.mgr.z.smithi136.stdout:Apr 27 00:01:35 smithi136 systemd[1]: ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mgr.z.service: Consumed 14.525s CPU time. 2024-04-27T00:01:35.732 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:01:35.732 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-27T00:01:35.733 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-27T00:01:35.733 DEBUG:teuthology.orchestra.run.smithi072:> sudo systemctl stop ceph-3c8453ae-0425-11ef-bc93-c7b262605968@osd.0 2024-04-27T00:01:36.277 INFO:journalctl@ceph.osd.0.smithi072.stdout:Apr 27 00:01:35 smithi072 systemd[1]: Stopping Ceph osd.0 for 3c8453ae-0425-11ef-bc93-c7b262605968... 2024-04-27T00:01:36.541 INFO:journalctl@ceph.osd.0.smithi072.stdout:Apr 27 00:01:36 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-0[40236]: 2024-04-27T00:01:36.276+0000 7f3022c00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:01:36.541 INFO:journalctl@ceph.osd.0.smithi072.stdout:Apr 27 00:01:36 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-0[40236]: 2024-04-27T00:01:36.276+0000 7f3022c00700 -1 osd.0 97 *** Got signal Terminated *** 2024-04-27T00:01:36.541 INFO:journalctl@ceph.osd.0.smithi072.stdout:Apr 27 00:01:36 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-0[40236]: 2024-04-27T00:01:36.276+0000 7f3022c00700 -1 osd.0 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T00:01:42.178 INFO:journalctl@ceph.osd.0.smithi072.stdout:Apr 27 00:01:41 smithi072 podman[84831]: 2024-04-27 00:01:41.846322239 +0000 UTC m=+5.725330583 container died 4d99bcfa9a7b104ca2078208627e9164c632b591c3bb48f64790b83373de739e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True) 2024-04-27T00:01:42.529 INFO:journalctl@ceph.osd.0.smithi072.stdout:Apr 27 00:01:42 smithi072 podman[84831]: 2024-04-27 00:01:42.323378576 +0000 UTC m=+6.202386940 container cleanup 4d99bcfa9a7b104ca2078208627e9164c632b591c3bb48f64790b83373de739e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-27T00:01:42.529 INFO:journalctl@ceph.osd.0.smithi072.stdout:Apr 27 00:01:42 smithi072 bash[84831]: ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-0 2024-04-27T00:01:42.790 INFO:journalctl@ceph.osd.0.smithi072.stdout:Apr 27 00:01:42 smithi072 podman[84849]: 2024-04-27 00:01:42.527587134 +0000 UTC m=+0.669214012 container remove 4d99bcfa9a7b104ca2078208627e9164c632b591c3bb48f64790b83373de739e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-0, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-27T00:01:43.541 INFO:journalctl@ceph.osd.0.smithi072.stdout:Apr 27 00:01:43 smithi072 podman[84935]: 2024-04-27 00:01:43.187800232 +0000 UTC m=+0.036300286 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-27T00:01:43.541 INFO:journalctl@ceph.osd.0.smithi072.stdout:Apr 27 00:01:43 smithi072 podman[84935]: 2024-04-27 00:01:43.313025581 +0000 UTC m=+0.161525659 container create aa759517c3a91df36707de08177aa201c69b043592f06755b1d797093743ce55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux ) 2024-04-27T00:01:44.041 INFO:journalctl@ceph.osd.0.smithi072.stdout:Apr 27 00:01:43 smithi072 podman[84935]: 2024-04-27 00:01:43.63124286 +0000 UTC m=+0.479742951 container init aa759517c3a91df36707de08177aa201c69b043592f06755b1d797093743ce55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-27T00:01:44.041 INFO:journalctl@ceph.osd.0.smithi072.stdout:Apr 27 00:01:43 smithi072 podman[84935]: 2024-04-27 00:01:43.642029136 +0000 UTC m=+0.490529237 container start aa759517c3a91df36707de08177aa201c69b043592f06755b1d797093743ce55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-27T00:01:44.041 INFO:journalctl@ceph.osd.0.smithi072.stdout:Apr 27 00:01:43 smithi072 podman[84935]: 2024-04-27 00:01:43.719871941 +0000 UTC m=+0.568372029 container attach aa759517c3a91df36707de08177aa201c69b043592f06755b1d797093743ce55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-27T00:01:44.472 INFO:journalctl@ceph.osd.0.smithi072.stdout:Apr 27 00:01:44 smithi072 podman[84935]: 2024-04-27 00:01:44.100915727 +0000 UTC m=+0.949415824 container died aa759517c3a91df36707de08177aa201c69b043592f06755b1d797093743ce55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-27T00:01:44.936 DEBUG:teuthology.orchestra.run.smithi072:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@osd.0.service' 2024-04-27T00:01:45.044 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:01:45.045 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-27T00:01:45.045 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-27T00:01:45.045 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-3c8453ae-0425-11ef-bc93-c7b262605968@osd.1 2024-04-27T00:01:45.360 INFO:journalctl@ceph.osd.1.smithi076.stdout:Apr 27 00:01:45 smithi076 systemd[1]: Stopping Ceph osd.1 for 3c8453ae-0425-11ef-bc93-c7b262605968... 2024-04-27T00:01:45.676 INFO:journalctl@ceph.osd.1.smithi076.stdout:Apr 27 00:01:45 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-1[37364]: 2024-04-27T00:01:45.357+0000 7f4b6ca5e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:01:45.676 INFO:journalctl@ceph.osd.1.smithi076.stdout:Apr 27 00:01:45 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-1[37364]: 2024-04-27T00:01:45.357+0000 7f4b6ca5e700 -1 osd.1 97 *** Got signal Terminated *** 2024-04-27T00:01:45.676 INFO:journalctl@ceph.osd.1.smithi076.stdout:Apr 27 00:01:45 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-1[37364]: 2024-04-27T00:01:45.357+0000 7f4b6ca5e700 -1 osd.1 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T00:01:50.926 INFO:journalctl@ceph.osd.1.smithi076.stdout:Apr 27 00:01:50 smithi076 podman[67010]: 2024-04-27 00:01:50.668486854 +0000 UTC m=+5.432532494 container died 41515053043a75ea4668de2849f1c4517aeb959bafda6c335eace7d27403f437 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-1, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-27T00:01:51.344 INFO:journalctl@ceph.osd.1.smithi076.stdout:Apr 27 00:01:51 smithi076 podman[67010]: 2024-04-27 00:01:51.076296461 +0000 UTC m=+5.840342091 container cleanup 41515053043a75ea4668de2849f1c4517aeb959bafda6c335eace7d27403f437 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-1, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-27T00:01:51.345 INFO:journalctl@ceph.osd.1.smithi076.stdout:Apr 27 00:01:51 smithi076 bash[67010]: ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-1 2024-04-27T00:01:51.676 INFO:journalctl@ceph.osd.1.smithi076.stdout:Apr 27 00:01:51 smithi076 podman[67027]: 2024-04-27 00:01:51.344275794 +0000 UTC m=+0.673668767 container remove 41515053043a75ea4668de2849f1c4517aeb959bafda6c335eace7d27403f437 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD) 2024-04-27T00:01:52.031 INFO:journalctl@ceph.osd.1.smithi076.stdout:Apr 27 00:01:51 smithi076 podman[67119]: 2024-04-27 00:01:51.653302177 +0000 UTC m=+0.018139503 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-27T00:01:52.031 INFO:journalctl@ceph.osd.1.smithi076.stdout:Apr 27 00:01:51 smithi076 podman[67119]: 2024-04-27 00:01:51.770229991 +0000 UTC m=+0.135067363 container create b2454602e5b3f3203be0aa42b8e5b7cd60466b9257c084369a6b96ef253c3db5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-1-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T00:01:52.031 INFO:journalctl@ceph.osd.1.smithi076.stdout:Apr 27 00:01:52 smithi076 podman[67119]: 2024-04-27 00:01:52.030988784 +0000 UTC m=+0.395826098 container init b2454602e5b3f3203be0aa42b8e5b7cd60466b9257c084369a6b96ef253c3db5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-27T00:01:52.299 INFO:journalctl@ceph.osd.1.smithi076.stdout:Apr 27 00:01:52 smithi076 podman[67119]: 2024-04-27 00:01:52.03460994 +0000 UTC m=+0.399447258 container start b2454602e5b3f3203be0aa42b8e5b7cd60466b9257c084369a6b96ef253c3db5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-27T00:01:52.299 INFO:journalctl@ceph.osd.1.smithi076.stdout:Apr 27 00:01:52 smithi076 podman[67119]: 2024-04-27 00:01:52.131232932 +0000 UTC m=+0.496070259 container attach b2454602e5b3f3203be0aa42b8e5b7cd60466b9257c084369a6b96ef253c3db5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424) 2024-04-27T00:01:52.676 INFO:journalctl@ceph.osd.1.smithi076.stdout:Apr 27 00:01:52 smithi076 podman[67119]: 2024-04-27 00:01:52.339849765 +0000 UTC m=+0.704687085 container died b2454602e5b3f3203be0aa42b8e5b7cd60466b9257c084369a6b96ef253c3db5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424) 2024-04-27T00:01:53.057 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@osd.1.service' 2024-04-27T00:01:53.096 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:01:53.097 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-27T00:01:53.097 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-27T00:01:53.097 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-3c8453ae-0425-11ef-bc93-c7b262605968@osd.2 2024-04-27T00:01:53.442 INFO:journalctl@ceph.osd.2.smithi136.stdout:Apr 27 00:01:53 smithi136 systemd[1]: Stopping Ceph osd.2 for 3c8453ae-0425-11ef-bc93-c7b262605968... 2024-04-27T00:01:53.797 INFO:journalctl@ceph.osd.2.smithi136.stdout:Apr 27 00:01:53 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-2[38195]: 2024-04-27T00:01:53.441+0000 7fbd5ec93700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:01:53.797 INFO:journalctl@ceph.osd.2.smithi136.stdout:Apr 27 00:01:53 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-2[38195]: 2024-04-27T00:01:53.441+0000 7fbd5ec93700 -1 osd.2 97 *** Got signal Terminated *** 2024-04-27T00:01:53.798 INFO:journalctl@ceph.osd.2.smithi136.stdout:Apr 27 00:01:53 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-2[38195]: 2024-04-27T00:01:53.441+0000 7fbd5ec93700 -1 osd.2 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T00:01:59.160 INFO:journalctl@ceph.osd.2.smithi136.stdout:Apr 27 00:01:58 smithi136 podman[67814]: 2024-04-27 00:01:58.813325758 +0000 UTC m=+5.528538247 container died 8f0a51f4f780cebcc39b36f1d82cb2510b6d98119e970e80f557b371bf893ae5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T00:01:59.436 INFO:journalctl@ceph.osd.2.smithi136.stdout:Apr 27 00:01:59 smithi136 podman[67814]: 2024-04-27 00:01:59.251163115 +0000 UTC m=+5.966375597 container cleanup 8f0a51f4f780cebcc39b36f1d82cb2510b6d98119e970e80f557b371bf893ae5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-27T00:01:59.436 INFO:journalctl@ceph.osd.2.smithi136.stdout:Apr 27 00:01:59 smithi136 bash[67814]: ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-2 2024-04-27T00:01:59.797 INFO:journalctl@ceph.osd.2.smithi136.stdout:Apr 27 00:01:59 smithi136 podman[67843]: 2024-04-27 00:01:59.436149352 +0000 UTC m=+0.618610965 container remove 8f0a51f4f780cebcc39b36f1d82cb2510b6d98119e970e80f557b371bf893ae5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2) 2024-04-27T00:02:00.131 INFO:journalctl@ceph.osd.2.smithi136.stdout:Apr 27 00:01:59 smithi136 podman[67937]: 2024-04-27 00:01:59.765380903 +0000 UTC m=+0.017969753 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-27T00:02:00.131 INFO:journalctl@ceph.osd.2.smithi136.stdout:Apr 27 00:01:59 smithi136 podman[67937]: 2024-04-27 00:01:59.895777291 +0000 UTC m=+0.148366141 container create c7a0472d2428f7ad46b7b7f0611ae3a149484d81fd6d8aa2666730130d979fe0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-27T00:02:00.431 INFO:journalctl@ceph.osd.2.smithi136.stdout:Apr 27 00:02:00 smithi136 podman[67937]: 2024-04-27 00:02:00.13162854 +0000 UTC m=+0.384217390 container init c7a0472d2428f7ad46b7b7f0611ae3a149484d81fd6d8aa2666730130d979fe0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2) 2024-04-27T00:02:00.431 INFO:journalctl@ceph.osd.2.smithi136.stdout:Apr 27 00:02:00 smithi136 podman[67937]: 2024-04-27 00:02:00.134796808 +0000 UTC m=+0.387385659 container start c7a0472d2428f7ad46b7b7f0611ae3a149484d81fd6d8aa2666730130d979fe0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS) 2024-04-27T00:02:00.431 INFO:journalctl@ceph.osd.2.smithi136.stdout:Apr 27 00:02:00 smithi136 podman[67937]: 2024-04-27 00:02:00.231581862 +0000 UTC m=+0.484170735 container attach c7a0472d2428f7ad46b7b7f0611ae3a149484d81fd6d8aa2666730130d979fe0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T00:02:00.776 INFO:journalctl@ceph.osd.2.smithi136.stdout:Apr 27 00:02:00 smithi136 podman[67937]: 2024-04-27 00:02:00.431234309 +0000 UTC m=+0.683823163 container died c7a0472d2428f7ad46b7b7f0611ae3a149484d81fd6d8aa2666730130d979fe0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-2-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-27T00:02:01.051 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@osd.2.service' 2024-04-27T00:02:01.092 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:02:01.092 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-27T00:02:01.092 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-27T00:02:01.093 DEBUG:teuthology.orchestra.run.smithi072:> sudo systemctl stop ceph-3c8453ae-0425-11ef-bc93-c7b262605968@osd.3 2024-04-27T00:02:01.511 INFO:journalctl@ceph.osd.3.smithi072.stdout:Apr 27 00:02:01 smithi072 systemd[1]: Stopping Ceph osd.3 for 3c8453ae-0425-11ef-bc93-c7b262605968... 2024-04-27T00:02:01.791 INFO:journalctl@ceph.osd.3.smithi072.stdout:Apr 27 00:02:01 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-3[47161]: 2024-04-27T00:02:01.632+0000 7fe516a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:02:01.791 INFO:journalctl@ceph.osd.3.smithi072.stdout:Apr 27 00:02:01 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-3[47161]: 2024-04-27T00:02:01.632+0000 7fe516a00700 -1 osd.3 97 *** Got signal Terminated *** 2024-04-27T00:02:01.791 INFO:journalctl@ceph.osd.3.smithi072.stdout:Apr 27 00:02:01 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-3[47161]: 2024-04-27T00:02:01.632+0000 7fe516a00700 -1 osd.3 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T00:02:03.790 INFO:journalctl@ceph.osd.3.smithi072.stdout:Apr 27 00:02:03 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-3[47161]: 2024-04-27T00:02:03.500+0000 7fe50d200700 -1 osd.3 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:39.540692+0000 front 2024-04-27T00:01:39.540773+0000 (oldest deadline 2024-04-27T00:02:03.040802+0000) 2024-04-27T00:02:04.790 INFO:journalctl@ceph.osd.3.smithi072.stdout:Apr 27 00:02:04 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-3[47161]: 2024-04-27T00:02:04.513+0000 7fe50d200700 -1 osd.3 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:39.540692+0000 front 2024-04-27T00:01:39.540773+0000 (oldest deadline 2024-04-27T00:02:03.040802+0000) 2024-04-27T00:02:05.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:05 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:05.070+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:05.790 INFO:journalctl@ceph.osd.3.smithi072.stdout:Apr 27 00:02:05 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-3[47161]: 2024-04-27T00:02:05.538+0000 7fe50d200700 -1 osd.3 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:39.540692+0000 front 2024-04-27T00:01:39.540773+0000 (oldest deadline 2024-04-27T00:02:03.040802+0000) 2024-04-27T00:02:05.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:05 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:05.536+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:06.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:06 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:06.059+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:06.790 INFO:journalctl@ceph.osd.3.smithi072.stdout:Apr 27 00:02:06 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-3[47161]: 2024-04-27T00:02:06.508+0000 7fe50d200700 -1 osd.3 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:39.540692+0000 front 2024-04-27T00:01:39.540773+0000 (oldest deadline 2024-04-27T00:02:03.040802+0000) 2024-04-27T00:02:06.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:06 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:06.529+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:07.291 INFO:journalctl@ceph.osd.3.smithi072.stdout:Apr 27 00:02:07 smithi072 podman[85039]: 2024-04-27 00:02:07.014578333 +0000 UTC m=+5.533348809 container died c0270c8d18d038013655bffc585fd0626a554495b8059a4987b1b41578792268 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-3, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-27T00:02:07.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:07 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:07.083+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:07.692 INFO:journalctl@ceph.osd.3.smithi072.stdout:Apr 27 00:02:07 smithi072 podman[85039]: 2024-04-27 00:02:07.522329026 +0000 UTC m=+6.041099547 container cleanup c0270c8d18d038013655bffc585fd0626a554495b8059a4987b1b41578792268 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-3, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=) 2024-04-27T00:02:07.692 INFO:journalctl@ceph.osd.3.smithi072.stdout:Apr 27 00:02:07 smithi072 bash[85039]: ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-3 2024-04-27T00:02:07.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:07 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:07.514+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:08.041 INFO:journalctl@ceph.osd.3.smithi072.stdout:Apr 27 00:02:07 smithi072 podman[85070]: 2024-04-27 00:02:07.691354744 +0000 UTC m=+0.657363098 container remove c0270c8d18d038013655bffc585fd0626a554495b8059a4987b1b41578792268 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-3, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux ) 2024-04-27T00:02:08.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:08 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:08.060+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:08.791 INFO:journalctl@ceph.osd.3.smithi072.stdout:Apr 27 00:02:08 smithi072 podman[85159]: 2024-04-27 00:02:08.411007396 +0000 UTC m=+0.039150298 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-27T00:02:08.791 INFO:journalctl@ceph.osd.3.smithi072.stdout:Apr 27 00:02:08 smithi072 podman[85159]: 2024-04-27 00:02:08.528071905 +0000 UTC m=+0.156214836 container create 9e576e87f01e30fb60eb51a1d1c2875a28a405ced867374810cc7b69cf4441c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T00:02:08.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:08 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:08.511+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:09.291 INFO:journalctl@ceph.osd.3.smithi072.stdout:Apr 27 00:02:08 smithi072 podman[85159]: 2024-04-27 00:02:08.875707505 +0000 UTC m=+0.503850422 container init 9e576e87f01e30fb60eb51a1d1c2875a28a405ced867374810cc7b69cf4441c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-3-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T00:02:09.291 INFO:journalctl@ceph.osd.3.smithi072.stdout:Apr 27 00:02:08 smithi072 podman[85159]: 2024-04-27 00:02:08.887060731 +0000 UTC m=+0.515203614 container start 9e576e87f01e30fb60eb51a1d1c2875a28a405ced867374810cc7b69cf4441c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-27T00:02:09.291 INFO:journalctl@ceph.osd.3.smithi072.stdout:Apr 27 00:02:08 smithi072 podman[85159]: 2024-04-27 00:02:08.966255575 +0000 UTC m=+0.594398493 container attach 9e576e87f01e30fb60eb51a1d1c2875a28a405ced867374810cc7b69cf4441c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=) 2024-04-27T00:02:09.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:09 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:09.062+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:09.694 INFO:journalctl@ceph.osd.3.smithi072.stdout:Apr 27 00:02:09 smithi072 podman[85159]: 2024-04-27 00:02:09.337139755 +0000 UTC m=+0.965282672 container died 9e576e87f01e30fb60eb51a1d1c2875a28a405ced867374810cc7b69cf4441c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T00:02:09.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:09 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:09.482+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:10.084 DEBUG:teuthology.orchestra.run.smithi072:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@osd.3.service' 2024-04-27T00:02:10.192 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:02:10.192 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-27T00:02:10.193 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-27T00:02:10.193 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-3c8453ae-0425-11ef-bc93-c7b262605968@osd.4 2024-04-27T00:02:10.477 INFO:journalctl@ceph.osd.4.smithi076.stdout:Apr 27 00:02:10 smithi076 systemd[1]: Stopping Ceph osd.4 for 3c8453ae-0425-11ef-bc93-c7b262605968... 2024-04-27T00:02:10.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:10 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:10.105+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:10.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:10 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:10.476+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:10.926 INFO:journalctl@ceph.osd.4.smithi076.stdout:Apr 27 00:02:10 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-4[43878]: 2024-04-27T00:02:10.512+0000 7f087d718700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:02:10.926 INFO:journalctl@ceph.osd.4.smithi076.stdout:Apr 27 00:02:10 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-4[43878]: 2024-04-27T00:02:10.512+0000 7f087d718700 -1 osd.4 97 *** Got signal Terminated *** 2024-04-27T00:02:10.926 INFO:journalctl@ceph.osd.4.smithi076.stdout:Apr 27 00:02:10 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-4[43878]: 2024-04-27T00:02:10.512+0000 7f087d718700 -1 osd.4 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T00:02:11.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:11 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:11.087+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:11.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:11 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:11.485+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:11.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:11 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:11.286+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.718315+0000 front 2024-04-27T00:01:46.718291+0000 (oldest deadline 2024-04-27T00:02:10.818574+0000) 2024-04-27T00:02:12.291 INFO:journalctl@ceph.osd.6.smithi072.stdout:Apr 27 00:02:11 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-6[54140]: 2024-04-27T00:02:11.982+0000 7fa49e600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:48.374435+0000 front 2024-04-27T00:01:48.374372+0000 (oldest deadline 2024-04-27T00:02:11.874608+0000) 2024-04-27T00:02:12.517 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:12 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:12.245+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.718315+0000 front 2024-04-27T00:01:46.718291+0000 (oldest deadline 2024-04-27T00:02:10.818574+0000) 2024-04-27T00:02:12.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:12 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:12.080+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:12.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:12 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:12.517+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:12.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:12 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:12.517+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:02:13.291 INFO:journalctl@ceph.osd.6.smithi072.stdout:Apr 27 00:02:12 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-6[54140]: 2024-04-27T00:02:12.989+0000 7fa49e600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:48.374435+0000 front 2024-04-27T00:01:48.374372+0000 (oldest deadline 2024-04-27T00:02:11.874608+0000) 2024-04-27T00:02:13.292 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:12 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:12.894+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.482851+0000 front 2024-04-27T00:01:46.482649+0000 (oldest deadline 2024-04-27T00:02:12.382365+0000) 2024-04-27T00:02:13.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:13 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:13.124+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:13.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:13 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:13.275+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.718315+0000 front 2024-04-27T00:01:46.718291+0000 (oldest deadline 2024-04-27T00:02:10.818574+0000) 2024-04-27T00:02:13.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:13 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:13.492+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:13.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:13 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:13.492+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:02:14.227 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:13 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:13.869+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.482851+0000 front 2024-04-27T00:01:46.482649+0000 (oldest deadline 2024-04-27T00:02:12.382365+0000) 2024-04-27T00:02:14.227 INFO:journalctl@ceph.osd.6.smithi072.stdout:Apr 27 00:02:13 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-6[54140]: 2024-04-27T00:02:13.954+0000 7fa49e600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:48.374435+0000 front 2024-04-27T00:01:48.374372+0000 (oldest deadline 2024-04-27T00:02:11.874608+0000) 2024-04-27T00:02:14.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:14 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:14.158+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:14.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:14 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:14.257+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.718315+0000 front 2024-04-27T00:01:46.718291+0000 (oldest deadline 2024-04-27T00:02:10.818574+0000) 2024-04-27T00:02:14.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:14 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:14.494+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:14.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:14 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:14.494+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:02:15.047 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:14 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:14.693+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:50.039612+0000 front 2024-04-27T00:01:50.039500+0000 (oldest deadline 2024-04-27T00:02:14.139555+0000) 2024-04-27T00:02:15.291 INFO:journalctl@ceph.osd.6.smithi072.stdout:Apr 27 00:02:14 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-6[54140]: 2024-04-27T00:02:14.995+0000 7fa49e600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:48.374435+0000 front 2024-04-27T00:01:48.374372+0000 (oldest deadline 2024-04-27T00:02:11.874608+0000) 2024-04-27T00:02:15.292 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:14 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:14.894+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.482851+0000 front 2024-04-27T00:01:46.482649+0000 (oldest deadline 2024-04-27T00:02:12.382365+0000) 2024-04-27T00:02:15.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:15 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:15.150+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:15.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:15 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:15.214+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.718315+0000 front 2024-04-27T00:01:46.718291+0000 (oldest deadline 2024-04-27T00:02:10.818574+0000) 2024-04-27T00:02:15.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:15 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:15.450+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:15.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:15 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:15.450+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:02:16.047 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 27 00:02:15 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-5[44688]: 2024-04-27T00:02:15.595+0000 7f6f36c76700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:50.177081+0000 front 2024-04-27T00:01:50.177009+0000 (oldest deadline 2024-04-27T00:02:15.477043+0000) 2024-04-27T00:02:16.047 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:15 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:15.645+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:50.039612+0000 front 2024-04-27T00:01:50.039500+0000 (oldest deadline 2024-04-27T00:02:14.139555+0000) 2024-04-27T00:02:16.222 INFO:journalctl@ceph.osd.4.smithi076.stdout:Apr 27 00:02:15 smithi076 podman[67238]: 2024-04-27 00:02:15.955887207 +0000 UTC m=+5.574459444 container died 4149ad404ea11136d0e3ed7b77901f76b13a245b33020eb80f1ecea167f39e9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-4, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True) 2024-04-27T00:02:16.291 INFO:journalctl@ceph.osd.6.smithi072.stdout:Apr 27 00:02:15 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-6[54140]: 2024-04-27T00:02:15.988+0000 7fa49e600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:48.374435+0000 front 2024-04-27T00:01:48.374372+0000 (oldest deadline 2024-04-27T00:02:11.874608+0000) 2024-04-27T00:02:16.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:15 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:15.853+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.482851+0000 front 2024-04-27T00:01:46.482649+0000 (oldest deadline 2024-04-27T00:02:12.382365+0000) 2024-04-27T00:02:16.487 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:16 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:16.233+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.718315+0000 front 2024-04-27T00:01:46.718291+0000 (oldest deadline 2024-04-27T00:02:10.818574+0000) 2024-04-27T00:02:16.488 INFO:journalctl@ceph.osd.4.smithi076.stdout:Apr 27 00:02:16 smithi076 podman[67238]: 2024-04-27 00:02:16.323104935 +0000 UTC m=+5.941677162 container cleanup 4149ad404ea11136d0e3ed7b77901f76b13a245b33020eb80f1ecea167f39e9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-4, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-27T00:02:16.488 INFO:journalctl@ceph.osd.4.smithi076.stdout:Apr 27 00:02:16 smithi076 bash[67238]: ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-4 2024-04-27T00:02:16.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:16 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:16.169+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:16.926 INFO:journalctl@ceph.osd.4.smithi076.stdout:Apr 27 00:02:16 smithi076 podman[67256]: 2024-04-27 00:02:16.622423807 +0000 UTC m=+0.664867875 container remove 4149ad404ea11136d0e3ed7b77901f76b13a245b33020eb80f1ecea167f39e9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-4, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T00:02:16.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:16 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:16.486+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:16.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:16 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:16.486+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:02:17.047 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 27 00:02:16 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-5[44688]: 2024-04-27T00:02:16.573+0000 7f6f36c76700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:50.177081+0000 front 2024-04-27T00:01:50.177009+0000 (oldest deadline 2024-04-27T00:02:15.477043+0000) 2024-04-27T00:02:17.047 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:16 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:16.685+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:50.039612+0000 front 2024-04-27T00:01:50.039500+0000 (oldest deadline 2024-04-27T00:02:14.139555+0000) 2024-04-27T00:02:17.271 INFO:journalctl@ceph.osd.4.smithi076.stdout:Apr 27 00:02:16 smithi076 podman[67342]: 2024-04-27 00:02:16.845537101 +0000 UTC m=+0.018174636 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-27T00:02:17.271 INFO:journalctl@ceph.osd.4.smithi076.stdout:Apr 27 00:02:16 smithi076 podman[67342]: 2024-04-27 00:02:16.965376987 +0000 UTC m=+0.138014532 container create f6b28855b7729817edcfa6fce5070266e285dae664da0e05fc523006d3ae3f82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=) 2024-04-27T00:02:17.271 INFO:journalctl@ceph.osd.4.smithi076.stdout:Apr 27 00:02:17 smithi076 podman[67342]: 2024-04-27 00:02:17.17581675 +0000 UTC m=+0.348454253 container init f6b28855b7729817edcfa6fce5070266e285dae664da0e05fc523006d3ae3f82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-27T00:02:17.272 INFO:journalctl@ceph.osd.4.smithi076.stdout:Apr 27 00:02:17 smithi076 podman[67342]: 2024-04-27 00:02:17.178954665 +0000 UTC m=+0.351592168 container start f6b28855b7729817edcfa6fce5070266e285dae664da0e05fc523006d3ae3f82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-27T00:02:17.290 INFO:journalctl@ceph.osd.6.smithi072.stdout:Apr 27 00:02:16 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-6[54140]: 2024-04-27T00:02:16.991+0000 7fa49e600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:48.374435+0000 front 2024-04-27T00:01:48.374372+0000 (oldest deadline 2024-04-27T00:02:11.874608+0000) 2024-04-27T00:02:17.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:16 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:16.823+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.482851+0000 front 2024-04-27T00:01:46.482649+0000 (oldest deadline 2024-04-27T00:02:12.382365+0000) 2024-04-27T00:02:17.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:17 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:17.184+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:17.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:17 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:17.270+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.718315+0000 front 2024-04-27T00:01:46.718291+0000 (oldest deadline 2024-04-27T00:02:10.818574+0000) 2024-04-27T00:02:17.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:17 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:17.454+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:17.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:17 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:17.454+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:02:17.677 INFO:journalctl@ceph.osd.4.smithi076.stdout:Apr 27 00:02:17 smithi076 podman[67342]: 2024-04-27 00:02:17.275798033 +0000 UTC m=+0.448435565 container attach f6b28855b7729817edcfa6fce5070266e285dae664da0e05fc523006d3ae3f82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5) 2024-04-27T00:02:17.677 INFO:journalctl@ceph.osd.4.smithi076.stdout:Apr 27 00:02:17 smithi076 podman[67342]: 2024-04-27 00:02:17.471504019 +0000 UTC m=+0.644141524 container died f6b28855b7729817edcfa6fce5070266e285dae664da0e05fc523006d3ae3f82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.build-date=20240424) 2024-04-27T00:02:18.047 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 27 00:02:17 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-5[44688]: 2024-04-27T00:02:17.581+0000 7f6f36c76700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:50.177081+0000 front 2024-04-27T00:01:50.177009+0000 (oldest deadline 2024-04-27T00:02:15.477043+0000) 2024-04-27T00:02:18.047 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:17 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:17.709+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:50.039612+0000 front 2024-04-27T00:01:50.039500+0000 (oldest deadline 2024-04-27T00:02:14.139555+0000) 2024-04-27T00:02:18.152 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@osd.4.service' 2024-04-27T00:02:18.190 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:02:18.190 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-27T00:02:18.191 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-27T00:02:18.191 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-3c8453ae-0425-11ef-bc93-c7b262605968@osd.5 2024-04-27T00:02:18.290 INFO:journalctl@ceph.osd.6.smithi072.stdout:Apr 27 00:02:17 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-6[54140]: 2024-04-27T00:02:17.972+0000 7fa49e600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:48.374435+0000 front 2024-04-27T00:01:48.374372+0000 (oldest deadline 2024-04-27T00:02:11.874608+0000) 2024-04-27T00:02:18.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:17 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:17.790+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.482851+0000 front 2024-04-27T00:01:46.482649+0000 (oldest deadline 2024-04-27T00:02:12.382365+0000) 2024-04-27T00:02:18.426 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:18 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:18.308+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.718315+0000 front 2024-04-27T00:01:46.718291+0000 (oldest deadline 2024-04-27T00:02:10.818574+0000) 2024-04-27T00:02:18.547 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 27 00:02:18 smithi136 systemd[1]: Stopping Ceph osd.5 for 3c8453ae-0425-11ef-bc93-c7b262605968... 2024-04-27T00:02:18.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:18 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:18.167+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:18.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:18 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:18.444+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:18.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:18 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:18.444+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:02:19.047 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:18 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:18.667+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:50.039612+0000 front 2024-04-27T00:01:50.039500+0000 (oldest deadline 2024-04-27T00:02:14.139555+0000) 2024-04-27T00:02:19.047 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 27 00:02:18 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-5[44688]: 2024-04-27T00:02:18.561+0000 7f6f36c76700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:50.177081+0000 front 2024-04-27T00:01:50.177009+0000 (oldest deadline 2024-04-27T00:02:15.477043+0000) 2024-04-27T00:02:19.047 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 27 00:02:18 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-5[44688]: 2024-04-27T00:02:18.573+0000 7f6f3e68a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:02:19.048 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 27 00:02:18 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-5[44688]: 2024-04-27T00:02:18.573+0000 7f6f3e68a700 -1 osd.5 97 *** Got signal Terminated *** 2024-04-27T00:02:19.048 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 27 00:02:18 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-5[44688]: 2024-04-27T00:02:18.573+0000 7f6f3e68a700 -1 osd.5 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T00:02:19.291 INFO:journalctl@ceph.osd.6.smithi072.stdout:Apr 27 00:02:19 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-6[54140]: 2024-04-27T00:02:19.021+0000 7fa49e600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:48.374435+0000 front 2024-04-27T00:01:48.374372+0000 (oldest deadline 2024-04-27T00:02:11.874608+0000) 2024-04-27T00:02:19.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:18 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:18.825+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.482851+0000 front 2024-04-27T00:01:46.482649+0000 (oldest deadline 2024-04-27T00:02:12.382365+0000) 2024-04-27T00:02:19.521 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:19 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:19.180+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:19.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:19 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:19.404+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:19.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:19 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:19.404+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:02:19.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:19 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:19.308+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.718315+0000 front 2024-04-27T00:01:46.718291+0000 (oldest deadline 2024-04-27T00:02:10.818574+0000) 2024-04-27T00:02:19.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:19 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:19.308+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:57.220259+0000 front 2024-04-27T00:01:57.220356+0000 (oldest deadline 2024-04-27T00:02:18.920342+0000) 2024-04-27T00:02:19.797 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 27 00:02:19 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-5[44688]: 2024-04-27T00:02:19.519+0000 7f6f36c76700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:50.177081+0000 front 2024-04-27T00:01:50.177009+0000 (oldest deadline 2024-04-27T00:02:15.477043+0000) 2024-04-27T00:02:19.797 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:19 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:19.685+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:50.039612+0000 front 2024-04-27T00:01:50.039500+0000 (oldest deadline 2024-04-27T00:02:14.139555+0000) 2024-04-27T00:02:20.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:19 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:19.844+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.482851+0000 front 2024-04-27T00:01:46.482649+0000 (oldest deadline 2024-04-27T00:02:12.382365+0000) 2024-04-27T00:02:20.291 INFO:journalctl@ceph.osd.6.smithi072.stdout:Apr 27 00:02:19 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-6[54140]: 2024-04-27T00:02:19.990+0000 7fa49e600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:48.374435+0000 front 2024-04-27T00:01:48.374372+0000 (oldest deadline 2024-04-27T00:02:11.874608+0000) 2024-04-27T00:02:20.291 INFO:journalctl@ceph.osd.6.smithi072.stdout:Apr 27 00:02:19 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-6[54140]: 2024-04-27T00:02:19.990+0000 7fa49e600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.677534+0000 front 2024-04-27T00:01:54.677465+0000 (oldest deadline 2024-04-27T00:02:19.977697+0000) 2024-04-27T00:02:20.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:20 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:20.191+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:20.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:20 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:20.354+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.718315+0000 front 2024-04-27T00:01:46.718291+0000 (oldest deadline 2024-04-27T00:02:10.818574+0000) 2024-04-27T00:02:20.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:20 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:20.354+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:57.220259+0000 front 2024-04-27T00:01:57.220356+0000 (oldest deadline 2024-04-27T00:02:18.920342+0000) 2024-04-27T00:02:20.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:20 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:20.380+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:20.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:20 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:20.380+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:02:21.047 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 27 00:02:20 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-5[44688]: 2024-04-27T00:02:20.547+0000 7f6f36c76700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:50.177081+0000 front 2024-04-27T00:01:50.177009+0000 (oldest deadline 2024-04-27T00:02:15.477043+0000) 2024-04-27T00:02:21.048 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:20 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:20.729+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:50.039612+0000 front 2024-04-27T00:01:50.039500+0000 (oldest deadline 2024-04-27T00:02:14.139555+0000) 2024-04-27T00:02:21.291 INFO:journalctl@ceph.osd.6.smithi072.stdout:Apr 27 00:02:20 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-6[54140]: 2024-04-27T00:02:20.950+0000 7fa49e600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:48.374435+0000 front 2024-04-27T00:01:48.374372+0000 (oldest deadline 2024-04-27T00:02:11.874608+0000) 2024-04-27T00:02:21.291 INFO:journalctl@ceph.osd.6.smithi072.stdout:Apr 27 00:02:20 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-6[54140]: 2024-04-27T00:02:20.950+0000 7fa49e600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.677534+0000 front 2024-04-27T00:01:54.677465+0000 (oldest deadline 2024-04-27T00:02:19.977697+0000) 2024-04-27T00:02:21.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:20 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:20.847+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.482851+0000 front 2024-04-27T00:01:46.482649+0000 (oldest deadline 2024-04-27T00:02:12.382365+0000) 2024-04-27T00:02:21.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:20 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:20.847+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:56.484639+0000 front 2024-04-27T00:01:56.484700+0000 (oldest deadline 2024-04-27T00:02:20.584890+0000) 2024-04-27T00:02:21.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:21 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:21.198+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:21.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:21 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:21.198+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:02:21.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:21 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:21.305+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.718315+0000 front 2024-04-27T00:01:46.718291+0000 (oldest deadline 2024-04-27T00:02:10.818574+0000) 2024-04-27T00:02:21.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:21 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:21.305+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:57.220259+0000 front 2024-04-27T00:01:57.220356+0000 (oldest deadline 2024-04-27T00:02:18.920342+0000) 2024-04-27T00:02:21.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:21 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:21.400+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:21.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:21 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:21.400+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:02:21.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:21 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:21.400+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.306812+0000 front 2024-04-27T00:01:55.306787+0000 (oldest deadline 2024-04-27T00:02:21.206885+0000) 2024-04-27T00:02:22.047 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:21 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:21.691+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:50.039612+0000 front 2024-04-27T00:01:50.039500+0000 (oldest deadline 2024-04-27T00:02:14.139555+0000) 2024-04-27T00:02:22.047 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 27 00:02:21 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-5[44688]: 2024-04-27T00:02:21.581+0000 7f6f36c76700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:50.177081+0000 front 2024-04-27T00:01:50.177009+0000 (oldest deadline 2024-04-27T00:02:15.477043+0000) 2024-04-27T00:02:22.048 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 27 00:02:21 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-5[44688]: 2024-04-27T00:02:21.581+0000 7f6f36c76700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.477306+0000 front 2024-04-27T00:01:55.477323+0000 (oldest deadline 2024-04-27T00:02:21.377654+0000) 2024-04-27T00:02:22.291 INFO:journalctl@ceph.osd.6.smithi072.stdout:Apr 27 00:02:21 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-6[54140]: 2024-04-27T00:02:21.969+0000 7fa49e600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:48.374435+0000 front 2024-04-27T00:01:48.374372+0000 (oldest deadline 2024-04-27T00:02:11.874608+0000) 2024-04-27T00:02:22.291 INFO:journalctl@ceph.osd.6.smithi072.stdout:Apr 27 00:02:21 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-6[54140]: 2024-04-27T00:02:21.969+0000 7fa49e600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.677534+0000 front 2024-04-27T00:01:54.677465+0000 (oldest deadline 2024-04-27T00:02:19.977697+0000) 2024-04-27T00:02:22.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:21 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:21.831+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.482851+0000 front 2024-04-27T00:01:46.482649+0000 (oldest deadline 2024-04-27T00:02:12.382365+0000) 2024-04-27T00:02:22.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:21 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:21.831+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:56.484639+0000 front 2024-04-27T00:01:56.484700+0000 (oldest deadline 2024-04-27T00:02:20.584890+0000) 2024-04-27T00:02:22.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:22 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:22.244+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:22.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:22 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:22.244+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:02:22.577 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:22 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:22.360+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:22.577 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:22 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:22.360+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:02:22.577 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:22 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:22.360+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.306812+0000 front 2024-04-27T00:01:55.306787+0000 (oldest deadline 2024-04-27T00:02:21.206885+0000) 2024-04-27T00:02:22.577 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:22 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:22.282+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.718315+0000 front 2024-04-27T00:01:46.718291+0000 (oldest deadline 2024-04-27T00:02:10.818574+0000) 2024-04-27T00:02:22.578 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:22 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:22.282+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:57.220259+0000 front 2024-04-27T00:01:57.220356+0000 (oldest deadline 2024-04-27T00:02:18.920342+0000) 2024-04-27T00:02:23.047 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 27 00:02:22 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-5[44688]: 2024-04-27T00:02:22.601+0000 7f6f36c76700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:50.177081+0000 front 2024-04-27T00:01:50.177009+0000 (oldest deadline 2024-04-27T00:02:15.477043+0000) 2024-04-27T00:02:23.047 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 27 00:02:22 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-5[44688]: 2024-04-27T00:02:22.601+0000 7f6f36c76700 -1 osd.5 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.477306+0000 front 2024-04-27T00:01:55.477323+0000 (oldest deadline 2024-04-27T00:02:21.377654+0000) 2024-04-27T00:02:23.048 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:22 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:22.688+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:50.039612+0000 front 2024-04-27T00:01:50.039500+0000 (oldest deadline 2024-04-27T00:02:14.139555+0000) 2024-04-27T00:02:23.048 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:22 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:22.688+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.840368+0000 front 2024-04-27T00:01:55.840396+0000 (oldest deadline 2024-04-27T00:02:21.740426+0000) 2024-04-27T00:02:23.291 INFO:journalctl@ceph.osd.6.smithi072.stdout:Apr 27 00:02:22 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-6[54140]: 2024-04-27T00:02:22.960+0000 7fa49e600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:48.374435+0000 front 2024-04-27T00:01:48.374372+0000 (oldest deadline 2024-04-27T00:02:11.874608+0000) 2024-04-27T00:02:23.291 INFO:journalctl@ceph.osd.6.smithi072.stdout:Apr 27 00:02:22 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-6[54140]: 2024-04-27T00:02:22.960+0000 7fa49e600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.677534+0000 front 2024-04-27T00:01:54.677465+0000 (oldest deadline 2024-04-27T00:02:19.977697+0000) 2024-04-27T00:02:23.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:22 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:22.790+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.482851+0000 front 2024-04-27T00:01:46.482649+0000 (oldest deadline 2024-04-27T00:02:12.382365+0000) 2024-04-27T00:02:23.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:22 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:22.790+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:56.484639+0000 front 2024-04-27T00:01:56.484700+0000 (oldest deadline 2024-04-27T00:02:20.584890+0000) 2024-04-27T00:02:23.321 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:23 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:23.250+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:23.321 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:23 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:23.250+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:02:23.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:23 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:23.250+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.718315+0000 front 2024-04-27T00:01:46.718291+0000 (oldest deadline 2024-04-27T00:02:10.818574+0000) 2024-04-27T00:02:23.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:23 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:23.250+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:57.220259+0000 front 2024-04-27T00:01:57.220356+0000 (oldest deadline 2024-04-27T00:02:18.920342+0000) 2024-04-27T00:02:23.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:23 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:23.316+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:23.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:23 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:23.316+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:02:23.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:23 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:23.316+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.306812+0000 front 2024-04-27T00:01:55.306787+0000 (oldest deadline 2024-04-27T00:02:21.206885+0000) 2024-04-27T00:02:24.009 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:23 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:23.704+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:50.039612+0000 front 2024-04-27T00:01:50.039500+0000 (oldest deadline 2024-04-27T00:02:14.139555+0000) 2024-04-27T00:02:24.009 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:23 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:23.704+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.840368+0000 front 2024-04-27T00:01:55.840396+0000 (oldest deadline 2024-04-27T00:02:21.740426+0000) 2024-04-27T00:02:24.040 INFO:journalctl@ceph.osd.6.smithi072.stdout:Apr 27 00:02:23 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-6[54140]: 2024-04-27T00:02:23.978+0000 7fa49e600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:48.374435+0000 front 2024-04-27T00:01:48.374372+0000 (oldest deadline 2024-04-27T00:02:11.874608+0000) 2024-04-27T00:02:24.041 INFO:journalctl@ceph.osd.6.smithi072.stdout:Apr 27 00:02:23 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-6[54140]: 2024-04-27T00:02:23.978+0000 7fa49e600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.677534+0000 front 2024-04-27T00:01:54.677465+0000 (oldest deadline 2024-04-27T00:02:19.977697+0000) 2024-04-27T00:02:24.041 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:23 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:23.787+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.482851+0000 front 2024-04-27T00:01:46.482649+0000 (oldest deadline 2024-04-27T00:02:12.382365+0000) 2024-04-27T00:02:24.041 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:23 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:23.787+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:56.484639+0000 front 2024-04-27T00:01:56.484700+0000 (oldest deadline 2024-04-27T00:02:20.584890+0000) 2024-04-27T00:02:24.264 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 27 00:02:24 smithi136 podman[68045]: 2024-04-27 00:02:24.008333278 +0000 UTC m=+5.625797183 container died 544c2273c27d88ee33a6c956592af14e9abf54eb45b21e1b1d56af02ddfcbf00 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-27T00:02:24.547 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 27 00:02:24 smithi136 podman[68045]: 2024-04-27 00:02:24.395595891 +0000 UTC m=+6.013059783 container cleanup 544c2273c27d88ee33a6c956592af14e9abf54eb45b21e1b1d56af02ddfcbf00 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-5, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424) 2024-04-27T00:02:24.547 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 27 00:02:24 smithi136 bash[68045]: ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-5 2024-04-27T00:02:24.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:24 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:24.288+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:24.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:24 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:24.288+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:02:24.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:24 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:24.298+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.718315+0000 front 2024-04-27T00:01:46.718291+0000 (oldest deadline 2024-04-27T00:02:10.818574+0000) 2024-04-27T00:02:24.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:24 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:24.298+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:57.220259+0000 front 2024-04-27T00:01:57.220356+0000 (oldest deadline 2024-04-27T00:02:18.920342+0000) 2024-04-27T00:02:24.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:24 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:24.304+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:24.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:24 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:24.304+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:02:24.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:24 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:24.304+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.306812+0000 front 2024-04-27T00:01:55.306787+0000 (oldest deadline 2024-04-27T00:02:21.206885+0000) 2024-04-27T00:02:25.002 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 27 00:02:24 smithi136 podman[68072]: 2024-04-27 00:02:24.672628373 +0000 UTC m=+0.656550126 container remove 544c2273c27d88ee33a6c956592af14e9abf54eb45b21e1b1d56af02ddfcbf00 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-5, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T00:02:25.003 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:24 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:24.748+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:50.039612+0000 front 2024-04-27T00:01:50.039500+0000 (oldest deadline 2024-04-27T00:02:14.139555+0000) 2024-04-27T00:02:25.003 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:24 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:24.748+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.840368+0000 front 2024-04-27T00:01:55.840396+0000 (oldest deadline 2024-04-27T00:02:21.740426+0000) 2024-04-27T00:02:25.267 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 27 00:02:25 smithi136 podman[68164]: 2024-04-27 00:02:24.905654286 +0000 UTC m=+0.018050806 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-27T00:02:25.268 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 27 00:02:25 smithi136 podman[68164]: 2024-04-27 00:02:25.015549099 +0000 UTC m=+0.127945664 container create 397c4772a15da35da077e50c83fffb2e0f6100e594f1a7e7eaa06698f842f0eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-27T00:02:25.290 INFO:journalctl@ceph.osd.6.smithi072.stdout:Apr 27 00:02:24 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-6[54140]: 2024-04-27T00:02:24.995+0000 7fa49e600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:48.374435+0000 front 2024-04-27T00:01:48.374372+0000 (oldest deadline 2024-04-27T00:02:11.874608+0000) 2024-04-27T00:02:25.291 INFO:journalctl@ceph.osd.6.smithi072.stdout:Apr 27 00:02:24 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-6[54140]: 2024-04-27T00:02:24.995+0000 7fa49e600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.677534+0000 front 2024-04-27T00:01:54.677465+0000 (oldest deadline 2024-04-27T00:02:19.977697+0000) 2024-04-27T00:02:25.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:24 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:24.790+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.482851+0000 front 2024-04-27T00:01:46.482649+0000 (oldest deadline 2024-04-27T00:02:12.382365+0000) 2024-04-27T00:02:25.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:24 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:24.790+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:56.484639+0000 front 2024-04-27T00:01:56.484700+0000 (oldest deadline 2024-04-27T00:02:20.584890+0000) 2024-04-27T00:02:25.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:25 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:25.302+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:25.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:25 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:25.302+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:02:25.548 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 27 00:02:25 smithi136 podman[68164]: 2024-04-27 00:02:25.267917816 +0000 UTC m=+0.380314335 container init 397c4772a15da35da077e50c83fffb2e0f6100e594f1a7e7eaa06698f842f0eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-5-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424) 2024-04-27T00:02:25.548 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 27 00:02:25 smithi136 podman[68164]: 2024-04-27 00:02:25.271152481 +0000 UTC m=+0.383549001 container start 397c4772a15da35da077e50c83fffb2e0f6100e594f1a7e7eaa06698f842f0eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-27T00:02:25.548 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 27 00:02:25 smithi136 podman[68164]: 2024-04-27 00:02:25.376340657 +0000 UTC m=+0.488737185 container attach 397c4772a15da35da077e50c83fffb2e0f6100e594f1a7e7eaa06698f842f0eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True) 2024-04-27T00:02:25.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:25 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:25.251+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.718315+0000 front 2024-04-27T00:01:46.718291+0000 (oldest deadline 2024-04-27T00:02:10.818574+0000) 2024-04-27T00:02:25.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:25 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:25.251+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:57.220259+0000 front 2024-04-27T00:01:57.220356+0000 (oldest deadline 2024-04-27T00:02:18.920342+0000) 2024-04-27T00:02:25.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:25 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:25.313+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:25.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:25 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:25.313+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:02:25.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:25 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:25.313+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.306812+0000 front 2024-04-27T00:01:55.306787+0000 (oldest deadline 2024-04-27T00:02:21.206885+0000) 2024-04-27T00:02:25.912 INFO:journalctl@ceph.osd.5.smithi136.stdout:Apr 27 00:02:25 smithi136 podman[68191]: 2024-04-27 00:02:25.596060418 +0000 UTC m=+0.018781273 container died 397c4772a15da35da077e50c83fffb2e0f6100e594f1a7e7eaa06698f842f0eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-5-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424) 2024-04-27T00:02:25.913 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:25 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:25.724+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:50.039612+0000 front 2024-04-27T00:01:50.039500+0000 (oldest deadline 2024-04-27T00:02:14.139555+0000) 2024-04-27T00:02:25.913 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:25 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:25.724+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.840368+0000 front 2024-04-27T00:01:55.840396+0000 (oldest deadline 2024-04-27T00:02:21.740426+0000) 2024-04-27T00:02:26.255 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@osd.5.service' 2024-04-27T00:02:26.291 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:02:26.291 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-27T00:02:26.291 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-27T00:02:26.291 DEBUG:teuthology.orchestra.run.smithi072:> sudo systemctl stop ceph-3c8453ae-0425-11ef-bc93-c7b262605968@osd.6 2024-04-27T00:02:26.292 INFO:journalctl@ceph.osd.6.smithi072.stdout:Apr 27 00:02:25 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-6[54140]: 2024-04-27T00:02:25.983+0000 7fa49e600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:48.374435+0000 front 2024-04-27T00:01:48.374372+0000 (oldest deadline 2024-04-27T00:02:11.874608+0000) 2024-04-27T00:02:26.292 INFO:journalctl@ceph.osd.6.smithi072.stdout:Apr 27 00:02:25 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-6[54140]: 2024-04-27T00:02:25.983+0000 7fa49e600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.677534+0000 front 2024-04-27T00:01:54.677465+0000 (oldest deadline 2024-04-27T00:02:19.977697+0000) 2024-04-27T00:02:26.293 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:25 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:25.811+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.482851+0000 front 2024-04-27T00:01:46.482649+0000 (oldest deadline 2024-04-27T00:02:12.382365+0000) 2024-04-27T00:02:26.293 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:25 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:25.811+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:56.484639+0000 front 2024-04-27T00:01:56.484700+0000 (oldest deadline 2024-04-27T00:02:20.584890+0000) 2024-04-27T00:02:26.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:26 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:26.292+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:26.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:26 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:26.292+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:02:26.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:26 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:26.237+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.718315+0000 front 2024-04-27T00:01:46.718291+0000 (oldest deadline 2024-04-27T00:02:10.818574+0000) 2024-04-27T00:02:26.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:26 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:26.237+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:57.220259+0000 front 2024-04-27T00:01:57.220356+0000 (oldest deadline 2024-04-27T00:02:18.920342+0000) 2024-04-27T00:02:26.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:26 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:26.279+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:26.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:26 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:26.279+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:02:26.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:26 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:26.279+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.306812+0000 front 2024-04-27T00:01:55.306787+0000 (oldest deadline 2024-04-27T00:02:21.206885+0000) 2024-04-27T00:02:26.790 INFO:journalctl@ceph.osd.6.smithi072.stdout:Apr 27 00:02:26 smithi072 systemd[1]: Stopping Ceph osd.6 for 3c8453ae-0425-11ef-bc93-c7b262605968... 2024-04-27T00:02:27.047 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:26 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:26.676+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:50.039612+0000 front 2024-04-27T00:01:50.039500+0000 (oldest deadline 2024-04-27T00:02:14.139555+0000) 2024-04-27T00:02:27.047 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:26 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:26.676+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.840368+0000 front 2024-04-27T00:01:55.840396+0000 (oldest deadline 2024-04-27T00:02:21.740426+0000) 2024-04-27T00:02:27.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:26 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:26.821+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.482851+0000 front 2024-04-27T00:01:46.482649+0000 (oldest deadline 2024-04-27T00:02:12.382365+0000) 2024-04-27T00:02:27.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:26 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:26.821+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:56.484639+0000 front 2024-04-27T00:01:56.484700+0000 (oldest deadline 2024-04-27T00:02:20.584890+0000) 2024-04-27T00:02:27.291 INFO:journalctl@ceph.osd.6.smithi072.stdout:Apr 27 00:02:26 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-6[54140]: 2024-04-27T00:02:26.888+0000 7fa4a7e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:02:27.291 INFO:journalctl@ceph.osd.6.smithi072.stdout:Apr 27 00:02:26 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-6[54140]: 2024-04-27T00:02:26.888+0000 7fa4a7e00700 -1 osd.6 97 *** Got signal Terminated *** 2024-04-27T00:02:27.292 INFO:journalctl@ceph.osd.6.smithi072.stdout:Apr 27 00:02:26 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-6[54140]: 2024-04-27T00:02:26.888+0000 7fa4a7e00700 -1 osd.6 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T00:02:27.292 INFO:journalctl@ceph.osd.6.smithi072.stdout:Apr 27 00:02:26 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-6[54140]: 2024-04-27T00:02:26.940+0000 7fa49e600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:48.374435+0000 front 2024-04-27T00:01:48.374372+0000 (oldest deadline 2024-04-27T00:02:11.874608+0000) 2024-04-27T00:02:27.292 INFO:journalctl@ceph.osd.6.smithi072.stdout:Apr 27 00:02:26 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-6[54140]: 2024-04-27T00:02:26.940+0000 7fa49e600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.677534+0000 front 2024-04-27T00:01:54.677465+0000 (oldest deadline 2024-04-27T00:02:19.977697+0000) 2024-04-27T00:02:27.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:27 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:27.257+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:27.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:27 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:27.257+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:02:27.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:27 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:27.259+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.718315+0000 front 2024-04-27T00:01:46.718291+0000 (oldest deadline 2024-04-27T00:02:10.818574+0000) 2024-04-27T00:02:27.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:27 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:27.259+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:57.220259+0000 front 2024-04-27T00:01:57.220356+0000 (oldest deadline 2024-04-27T00:02:18.920342+0000) 2024-04-27T00:02:27.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:27 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:27.326+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:27.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:27 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:27.326+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:02:27.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:27 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:27.326+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.306812+0000 front 2024-04-27T00:01:55.306787+0000 (oldest deadline 2024-04-27T00:02:21.206885+0000) 2024-04-27T00:02:28.040 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:27 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:27.776+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.482851+0000 front 2024-04-27T00:01:46.482649+0000 (oldest deadline 2024-04-27T00:02:12.382365+0000) 2024-04-27T00:02:28.041 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:27 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:27.776+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:56.484639+0000 front 2024-04-27T00:01:56.484700+0000 (oldest deadline 2024-04-27T00:02:20.584890+0000) 2024-04-27T00:02:28.041 INFO:journalctl@ceph.osd.6.smithi072.stdout:Apr 27 00:02:27 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-6[54140]: 2024-04-27T00:02:27.911+0000 7fa49e600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:48.374435+0000 front 2024-04-27T00:01:48.374372+0000 (oldest deadline 2024-04-27T00:02:11.874608+0000) 2024-04-27T00:02:28.041 INFO:journalctl@ceph.osd.6.smithi072.stdout:Apr 27 00:02:27 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-6[54140]: 2024-04-27T00:02:27.911+0000 7fa49e600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.677534+0000 front 2024-04-27T00:01:54.677465+0000 (oldest deadline 2024-04-27T00:02:19.977697+0000) 2024-04-27T00:02:28.047 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:27 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:27.722+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:50.039612+0000 front 2024-04-27T00:01:50.039500+0000 (oldest deadline 2024-04-27T00:02:14.139555+0000) 2024-04-27T00:02:28.047 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:27 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:27.722+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.840368+0000 front 2024-04-27T00:01:55.840396+0000 (oldest deadline 2024-04-27T00:02:21.740426+0000) 2024-04-27T00:02:28.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:28 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:28.270+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:28.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:28 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:28.270+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:02:28.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:28 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:28.270+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.693019+0000 front 2024-04-27T00:02:03.693029+0000 (oldest deadline 2024-04-27T00:02:27.792790+0000) 2024-04-27T00:02:28.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:28 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:28.250+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.718315+0000 front 2024-04-27T00:01:46.718291+0000 (oldest deadline 2024-04-27T00:02:10.818574+0000) 2024-04-27T00:02:28.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:28 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:28.250+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:57.220259+0000 front 2024-04-27T00:01:57.220356+0000 (oldest deadline 2024-04-27T00:02:18.920342+0000) 2024-04-27T00:02:28.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:28 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:28.250+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:05.723455+0000 front 2024-04-27T00:02:05.723579+0000 (oldest deadline 2024-04-27T00:02:27.423213+0000) 2024-04-27T00:02:28.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:28 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:28.345+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:28.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:28 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:28.345+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:02:28.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:28 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:28.345+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.306812+0000 front 2024-04-27T00:01:55.306787+0000 (oldest deadline 2024-04-27T00:02:21.206885+0000) 2024-04-27T00:02:29.040 INFO:journalctl@ceph.osd.6.smithi072.stdout:Apr 27 00:02:28 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-6[54140]: 2024-04-27T00:02:28.952+0000 7fa49e600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:48.374435+0000 front 2024-04-27T00:01:48.374372+0000 (oldest deadline 2024-04-27T00:02:11.874608+0000) 2024-04-27T00:02:29.041 INFO:journalctl@ceph.osd.6.smithi072.stdout:Apr 27 00:02:28 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-6[54140]: 2024-04-27T00:02:28.952+0000 7fa49e600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.677534+0000 front 2024-04-27T00:01:54.677465+0000 (oldest deadline 2024-04-27T00:02:19.977697+0000) 2024-04-27T00:02:29.041 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:28 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:28.750+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.482851+0000 front 2024-04-27T00:01:46.482649+0000 (oldest deadline 2024-04-27T00:02:12.382365+0000) 2024-04-27T00:02:29.041 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:28 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:28.750+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:56.484639+0000 front 2024-04-27T00:01:56.484700+0000 (oldest deadline 2024-04-27T00:02:20.584890+0000) 2024-04-27T00:02:29.047 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:28 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:28.735+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:50.039612+0000 front 2024-04-27T00:01:50.039500+0000 (oldest deadline 2024-04-27T00:02:14.139555+0000) 2024-04-27T00:02:29.047 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:28 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:28.735+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.840368+0000 front 2024-04-27T00:01:55.840396+0000 (oldest deadline 2024-04-27T00:02:21.740426+0000) 2024-04-27T00:02:29.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:29 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:29.261+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:29.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:29 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:29.261+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:02:29.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:29 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:29.261+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.693019+0000 front 2024-04-27T00:02:03.693029+0000 (oldest deadline 2024-04-27T00:02:27.792790+0000) 2024-04-27T00:02:29.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:29 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:29.297+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.718315+0000 front 2024-04-27T00:01:46.718291+0000 (oldest deadline 2024-04-27T00:02:10.818574+0000) 2024-04-27T00:02:29.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:29 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:29.297+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:57.220259+0000 front 2024-04-27T00:01:57.220356+0000 (oldest deadline 2024-04-27T00:02:18.920342+0000) 2024-04-27T00:02:29.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:29 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:29.297+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:05.723455+0000 front 2024-04-27T00:02:05.723579+0000 (oldest deadline 2024-04-27T00:02:27.423213+0000) 2024-04-27T00:02:29.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:29 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:29.363+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:29.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:29 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:29.363+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:02:29.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:29 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:29.363+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.306812+0000 front 2024-04-27T00:01:55.306787+0000 (oldest deadline 2024-04-27T00:02:21.206885+0000) 2024-04-27T00:02:30.041 INFO:journalctl@ceph.osd.6.smithi072.stdout:Apr 27 00:02:29 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-6[54140]: 2024-04-27T00:02:29.928+0000 7fa49e600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:48.374435+0000 front 2024-04-27T00:01:48.374372+0000 (oldest deadline 2024-04-27T00:02:11.874608+0000) 2024-04-27T00:02:30.041 INFO:journalctl@ceph.osd.6.smithi072.stdout:Apr 27 00:02:29 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-6[54140]: 2024-04-27T00:02:29.928+0000 7fa49e600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.677534+0000 front 2024-04-27T00:01:54.677465+0000 (oldest deadline 2024-04-27T00:02:19.977697+0000) 2024-04-27T00:02:30.041 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:29 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:29.734+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.482851+0000 front 2024-04-27T00:01:46.482649+0000 (oldest deadline 2024-04-27T00:02:12.382365+0000) 2024-04-27T00:02:30.041 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:29 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:29.734+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:56.484639+0000 front 2024-04-27T00:01:56.484700+0000 (oldest deadline 2024-04-27T00:02:20.584890+0000) 2024-04-27T00:02:30.041 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:29 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:29.734+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.487199+0000 front 2024-04-27T00:02:03.487670+0000 (oldest deadline 2024-04-27T00:02:28.787386+0000) 2024-04-27T00:02:30.047 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:29 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:29.698+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:50.039612+0000 front 2024-04-27T00:01:50.039500+0000 (oldest deadline 2024-04-27T00:02:14.139555+0000) 2024-04-27T00:02:30.047 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:29 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:29.698+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.840368+0000 front 2024-04-27T00:01:55.840396+0000 (oldest deadline 2024-04-27T00:02:21.740426+0000) 2024-04-27T00:02:30.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:30 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:30.265+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:30.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:30 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:30.265+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:02:30.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:30 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:30.265+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.693019+0000 front 2024-04-27T00:02:03.693029+0000 (oldest deadline 2024-04-27T00:02:27.792790+0000) 2024-04-27T00:02:30.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:30 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:30.259+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.718315+0000 front 2024-04-27T00:01:46.718291+0000 (oldest deadline 2024-04-27T00:02:10.818574+0000) 2024-04-27T00:02:30.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:30 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:30.259+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:57.220259+0000 front 2024-04-27T00:01:57.220356+0000 (oldest deadline 2024-04-27T00:02:18.920342+0000) 2024-04-27T00:02:30.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:30 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:30.259+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:05.723455+0000 front 2024-04-27T00:02:05.723579+0000 (oldest deadline 2024-04-27T00:02:27.423213+0000) 2024-04-27T00:02:30.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:30 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:30.360+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:30.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:30 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:30.360+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:02:30.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:30 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:30.360+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.306812+0000 front 2024-04-27T00:01:55.306787+0000 (oldest deadline 2024-04-27T00:02:21.206885+0000) 2024-04-27T00:02:30.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:30 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:30.360+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.408538+0000 front 2024-04-27T00:02:06.408413+0000 (oldest deadline 2024-04-27T00:02:29.908238+0000) 2024-04-27T00:02:30.955 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:30 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:30.699+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.482851+0000 front 2024-04-27T00:01:46.482649+0000 (oldest deadline 2024-04-27T00:02:12.382365+0000) 2024-04-27T00:02:30.955 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:30 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:30.699+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:56.484639+0000 front 2024-04-27T00:01:56.484700+0000 (oldest deadline 2024-04-27T00:02:20.584890+0000) 2024-04-27T00:02:30.955 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:30 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:30.699+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.487199+0000 front 2024-04-27T00:02:03.487670+0000 (oldest deadline 2024-04-27T00:02:28.787386+0000) 2024-04-27T00:02:31.047 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:30 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:30.701+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:50.039612+0000 front 2024-04-27T00:01:50.039500+0000 (oldest deadline 2024-04-27T00:02:14.139555+0000) 2024-04-27T00:02:31.047 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:30 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:30.701+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.840368+0000 front 2024-04-27T00:01:55.840396+0000 (oldest deadline 2024-04-27T00:02:21.740426+0000) 2024-04-27T00:02:31.291 INFO:journalctl@ceph.osd.6.smithi072.stdout:Apr 27 00:02:30 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-6[54140]: 2024-04-27T00:02:30.954+0000 7fa49e600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:48.374435+0000 front 2024-04-27T00:01:48.374372+0000 (oldest deadline 2024-04-27T00:02:11.874608+0000) 2024-04-27T00:02:31.291 INFO:journalctl@ceph.osd.6.smithi072.stdout:Apr 27 00:02:30 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-6[54140]: 2024-04-27T00:02:30.954+0000 7fa49e600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.677534+0000 front 2024-04-27T00:01:54.677465+0000 (oldest deadline 2024-04-27T00:02:19.977697+0000) 2024-04-27T00:02:31.291 INFO:journalctl@ceph.osd.6.smithi072.stdout:Apr 27 00:02:30 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-6[54140]: 2024-04-27T00:02:30.954+0000 7fa49e600700 -1 osd.6 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:02.879509+0000 front 2024-04-27T00:02:02.879553+0000 (oldest deadline 2024-04-27T00:02:30.480587+0000) 2024-04-27T00:02:31.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:31 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:31.284+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:31.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:31 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:31.284+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:02:31.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:31 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:31.284+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.693019+0000 front 2024-04-27T00:02:03.693029+0000 (oldest deadline 2024-04-27T00:02:27.792790+0000) 2024-04-27T00:02:31.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:31 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:31.245+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.718315+0000 front 2024-04-27T00:01:46.718291+0000 (oldest deadline 2024-04-27T00:02:10.818574+0000) 2024-04-27T00:02:31.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:31 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:31.245+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:57.220259+0000 front 2024-04-27T00:01:57.220356+0000 (oldest deadline 2024-04-27T00:02:18.920342+0000) 2024-04-27T00:02:31.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:31 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:31.245+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:05.723455+0000 front 2024-04-27T00:02:05.723579+0000 (oldest deadline 2024-04-27T00:02:27.423213+0000) 2024-04-27T00:02:31.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:31 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:31.350+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:31.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:31 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:31.350+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:02:31.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:31 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:31.350+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.306812+0000 front 2024-04-27T00:01:55.306787+0000 (oldest deadline 2024-04-27T00:02:21.206885+0000) 2024-04-27T00:02:31.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:31 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:31.350+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.408538+0000 front 2024-04-27T00:02:06.408413+0000 (oldest deadline 2024-04-27T00:02:29.908238+0000) 2024-04-27T00:02:32.004 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:31 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:31.697+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.482851+0000 front 2024-04-27T00:01:46.482649+0000 (oldest deadline 2024-04-27T00:02:12.382365+0000) 2024-04-27T00:02:32.004 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:31 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:31.697+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:56.484639+0000 front 2024-04-27T00:01:56.484700+0000 (oldest deadline 2024-04-27T00:02:20.584890+0000) 2024-04-27T00:02:32.004 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:31 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:31.697+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.487199+0000 front 2024-04-27T00:02:03.487670+0000 (oldest deadline 2024-04-27T00:02:28.787386+0000) 2024-04-27T00:02:32.047 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:31 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:31.741+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:50.039612+0000 front 2024-04-27T00:01:50.039500+0000 (oldest deadline 2024-04-27T00:02:14.139555+0000) 2024-04-27T00:02:32.047 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:31 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:31.741+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.840368+0000 front 2024-04-27T00:01:55.840396+0000 (oldest deadline 2024-04-27T00:02:21.740426+0000) 2024-04-27T00:02:32.047 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:31 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:31.741+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.342128+0000 front 2024-04-27T00:02:06.342039+0000 (oldest deadline 2024-04-27T00:02:31.641818+0000) 2024-04-27T00:02:32.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:32 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:32.257+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:32.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:32 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:32.257+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:02:32.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:32 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:32.257+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.693019+0000 front 2024-04-27T00:02:03.693029+0000 (oldest deadline 2024-04-27T00:02:27.792790+0000) 2024-04-27T00:02:32.597 INFO:journalctl@ceph.osd.6.smithi072.stdout:Apr 27 00:02:32 smithi072 podman[85267]: 2024-04-27 00:02:32.306523939 +0000 UTC m=+5.578161077 container died 680f476a0f1d73fa90690805eee2b1ef417c083c588e3ea1430860e77ee5460c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-27T00:02:32.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:32 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:32.250+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.718315+0000 front 2024-04-27T00:01:46.718291+0000 (oldest deadline 2024-04-27T00:02:10.818574+0000) 2024-04-27T00:02:32.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:32 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:32.250+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:57.220259+0000 front 2024-04-27T00:01:57.220356+0000 (oldest deadline 2024-04-27T00:02:18.920342+0000) 2024-04-27T00:02:32.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:32 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:32.250+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:05.723455+0000 front 2024-04-27T00:02:05.723579+0000 (oldest deadline 2024-04-27T00:02:27.423213+0000) 2024-04-27T00:02:32.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:32 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:32.326+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:32.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:32 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:32.326+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:02:32.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:32 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:32.326+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.306812+0000 front 2024-04-27T00:01:55.306787+0000 (oldest deadline 2024-04-27T00:02:21.206885+0000) 2024-04-27T00:02:32.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:32 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:32.326+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.408538+0000 front 2024-04-27T00:02:06.408413+0000 (oldest deadline 2024-04-27T00:02:29.908238+0000) 2024-04-27T00:02:32.991 INFO:journalctl@ceph.osd.6.smithi072.stdout:Apr 27 00:02:32 smithi072 podman[85267]: 2024-04-27 00:02:32.708430879 +0000 UTC m=+5.980068025 container cleanup 680f476a0f1d73fa90690805eee2b1ef417c083c588e3ea1430860e77ee5460c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-6, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-27T00:02:32.991 INFO:journalctl@ceph.osd.6.smithi072.stdout:Apr 27 00:02:32 smithi072 bash[85267]: ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-6 2024-04-27T00:02:32.992 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:32 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:32.660+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.482851+0000 front 2024-04-27T00:01:46.482649+0000 (oldest deadline 2024-04-27T00:02:12.382365+0000) 2024-04-27T00:02:32.992 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:32 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:32.660+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:56.484639+0000 front 2024-04-27T00:01:56.484700+0000 (oldest deadline 2024-04-27T00:02:20.584890+0000) 2024-04-27T00:02:32.992 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:32 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:32.660+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.487199+0000 front 2024-04-27T00:02:03.487670+0000 (oldest deadline 2024-04-27T00:02:28.787386+0000) 2024-04-27T00:02:33.047 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:32 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:32.727+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:50.039612+0000 front 2024-04-27T00:01:50.039500+0000 (oldest deadline 2024-04-27T00:02:14.139555+0000) 2024-04-27T00:02:33.047 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:32 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:32.727+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.840368+0000 front 2024-04-27T00:01:55.840396+0000 (oldest deadline 2024-04-27T00:02:21.740426+0000) 2024-04-27T00:02:33.047 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:32 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:32.727+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.342128+0000 front 2024-04-27T00:02:06.342039+0000 (oldest deadline 2024-04-27T00:02:31.641818+0000) 2024-04-27T00:02:33.290 INFO:journalctl@ceph.osd.6.smithi072.stdout:Apr 27 00:02:33 smithi072 podman[85296]: 2024-04-27 00:02:33.087972533 +0000 UTC m=+0.768587349 container remove 680f476a0f1d73fa90690805eee2b1ef417c083c588e3ea1430860e77ee5460c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-6, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-27T00:02:33.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:33 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:33.222+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:33.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:33 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:33.222+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:02:33.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:33 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:33.222+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.693019+0000 front 2024-04-27T00:02:03.693029+0000 (oldest deadline 2024-04-27T00:02:27.792790+0000) 2024-04-27T00:02:33.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:33 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:33.290+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.718315+0000 front 2024-04-27T00:01:46.718291+0000 (oldest deadline 2024-04-27T00:02:10.818574+0000) 2024-04-27T00:02:33.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:33 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:33.290+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:57.220259+0000 front 2024-04-27T00:01:57.220356+0000 (oldest deadline 2024-04-27T00:02:18.920342+0000) 2024-04-27T00:02:33.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:33 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:33.290+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:05.723455+0000 front 2024-04-27T00:02:05.723579+0000 (oldest deadline 2024-04-27T00:02:27.423213+0000) 2024-04-27T00:02:33.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:33 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:33.287+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:33.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:33 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:33.287+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:02:33.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:33 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:33.287+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.306812+0000 front 2024-04-27T00:01:55.306787+0000 (oldest deadline 2024-04-27T00:02:21.206885+0000) 2024-04-27T00:02:33.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:33 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:33.287+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.408538+0000 front 2024-04-27T00:02:06.408413+0000 (oldest deadline 2024-04-27T00:02:29.908238+0000) 2024-04-27T00:02:34.004 INFO:journalctl@ceph.osd.6.smithi072.stdout:Apr 27 00:02:33 smithi072 podman[85384]: 2024-04-27 00:02:33.607380807 +0000 UTC m=+0.034364226 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-27T00:02:34.004 INFO:journalctl@ceph.osd.6.smithi072.stdout:Apr 27 00:02:33 smithi072 podman[85384]: 2024-04-27 00:02:33.706398271 +0000 UTC m=+0.133381721 container create 7390ea07666e02f58345762e116f3e6373fabe27454f7bff85d3fddd13b94a2a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True) 2024-04-27T00:02:34.004 INFO:journalctl@ceph.osd.6.smithi072.stdout:Apr 27 00:02:34 smithi072 podman[85384]: 2024-04-27 00:02:34.003188955 +0000 UTC m=+0.430172415 container init 7390ea07666e02f58345762e116f3e6373fabe27454f7bff85d3fddd13b94a2a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=) 2024-04-27T00:02:34.004 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:33 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:33.681+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.482851+0000 front 2024-04-27T00:01:46.482649+0000 (oldest deadline 2024-04-27T00:02:12.382365+0000) 2024-04-27T00:02:34.004 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:33 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:33.681+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:56.484639+0000 front 2024-04-27T00:01:56.484700+0000 (oldest deadline 2024-04-27T00:02:20.584890+0000) 2024-04-27T00:02:34.004 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:33 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:33.681+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.487199+0000 front 2024-04-27T00:02:03.487670+0000 (oldest deadline 2024-04-27T00:02:28.787386+0000) 2024-04-27T00:02:34.047 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:33 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:33.765+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:50.039612+0000 front 2024-04-27T00:01:50.039500+0000 (oldest deadline 2024-04-27T00:02:14.139555+0000) 2024-04-27T00:02:34.047 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:33 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:33.765+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.840368+0000 front 2024-04-27T00:01:55.840396+0000 (oldest deadline 2024-04-27T00:02:21.740426+0000) 2024-04-27T00:02:34.047 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:33 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:33.765+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.342128+0000 front 2024-04-27T00:02:06.342039+0000 (oldest deadline 2024-04-27T00:02:31.641818+0000) 2024-04-27T00:02:34.291 INFO:journalctl@ceph.osd.6.smithi072.stdout:Apr 27 00:02:34 smithi072 podman[85384]: 2024-04-27 00:02:34.015049525 +0000 UTC m=+0.442032946 container start 7390ea07666e02f58345762e116f3e6373fabe27454f7bff85d3fddd13b94a2a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-6-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-27T00:02:34.291 INFO:journalctl@ceph.osd.6.smithi072.stdout:Apr 27 00:02:34 smithi072 podman[85384]: 2024-04-27 00:02:34.094239468 +0000 UTC m=+0.521222908 container attach 7390ea07666e02f58345762e116f3e6373fabe27454f7bff85d3fddd13b94a2a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-6-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True) 2024-04-27T00:02:34.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:34 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:34.219+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:34.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:34 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:34.219+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:02:34.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:34 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:34.219+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.693019+0000 front 2024-04-27T00:02:03.693029+0000 (oldest deadline 2024-04-27T00:02:27.792790+0000) 2024-04-27T00:02:34.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:34 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:34.268+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.718315+0000 front 2024-04-27T00:01:46.718291+0000 (oldest deadline 2024-04-27T00:02:10.818574+0000) 2024-04-27T00:02:34.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:34 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:34.268+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:57.220259+0000 front 2024-04-27T00:01:57.220356+0000 (oldest deadline 2024-04-27T00:02:18.920342+0000) 2024-04-27T00:02:34.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:34 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:34.268+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:05.723455+0000 front 2024-04-27T00:02:05.723579+0000 (oldest deadline 2024-04-27T00:02:27.423213+0000) 2024-04-27T00:02:34.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:34 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:34.310+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:34.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:34 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:34.310+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:02:34.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:34 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:34.310+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.306812+0000 front 2024-04-27T00:01:55.306787+0000 (oldest deadline 2024-04-27T00:02:21.206885+0000) 2024-04-27T00:02:34.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:34 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:34.310+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.408538+0000 front 2024-04-27T00:02:06.408413+0000 (oldest deadline 2024-04-27T00:02:29.908238+0000) 2024-04-27T00:02:34.790 INFO:journalctl@ceph.osd.6.smithi072.stdout:Apr 27 00:02:34 smithi072 podman[85384]: 2024-04-27 00:02:34.474545471 +0000 UTC m=+0.901528884 container died 7390ea07666e02f58345762e116f3e6373fabe27454f7bff85d3fddd13b94a2a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T00:02:34.791 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:34 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:34.659+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.482851+0000 front 2024-04-27T00:01:46.482649+0000 (oldest deadline 2024-04-27T00:02:12.382365+0000) 2024-04-27T00:02:34.791 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:34 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:34.659+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:56.484639+0000 front 2024-04-27T00:01:56.484700+0000 (oldest deadline 2024-04-27T00:02:20.584890+0000) 2024-04-27T00:02:34.791 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:34 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:34.659+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.487199+0000 front 2024-04-27T00:02:03.487670+0000 (oldest deadline 2024-04-27T00:02:28.787386+0000) 2024-04-27T00:02:35.047 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:34 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:34.772+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:50.039612+0000 front 2024-04-27T00:01:50.039500+0000 (oldest deadline 2024-04-27T00:02:14.139555+0000) 2024-04-27T00:02:35.047 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:34 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:34.772+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.840368+0000 front 2024-04-27T00:01:55.840396+0000 (oldest deadline 2024-04-27T00:02:21.740426+0000) 2024-04-27T00:02:35.047 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:34 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:34.772+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.342128+0000 front 2024-04-27T00:02:06.342039+0000 (oldest deadline 2024-04-27T00:02:31.641818+0000) 2024-04-27T00:02:35.256 DEBUG:teuthology.orchestra.run.smithi072:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@osd.6.service' 2024-04-27T00:02:35.371 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:02:35.371 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-27T00:02:35.371 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-27T00:02:35.372 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-3c8453ae-0425-11ef-bc93-c7b262605968@osd.7 2024-04-27T00:02:35.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:35 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:35.250+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:35.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:35 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:35.250+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:02:35.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:35 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:35.250+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.693019+0000 front 2024-04-27T00:02:03.693029+0000 (oldest deadline 2024-04-27T00:02:27.792790+0000) 2024-04-27T00:02:35.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:35 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:35.221+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.718315+0000 front 2024-04-27T00:01:46.718291+0000 (oldest deadline 2024-04-27T00:02:10.818574+0000) 2024-04-27T00:02:35.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:35 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:35.221+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:57.220259+0000 front 2024-04-27T00:01:57.220356+0000 (oldest deadline 2024-04-27T00:02:18.920342+0000) 2024-04-27T00:02:35.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:35 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:35.221+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:05.723455+0000 front 2024-04-27T00:02:05.723579+0000 (oldest deadline 2024-04-27T00:02:27.423213+0000) 2024-04-27T00:02:35.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:35 smithi076 systemd[1]: Stopping Ceph osd.7 for 3c8453ae-0425-11ef-bc93-c7b262605968... 2024-04-27T00:02:35.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:35 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:35.335+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:35.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:35 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:35.335+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:02:35.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:35 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:35.335+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.306812+0000 front 2024-04-27T00:01:55.306787+0000 (oldest deadline 2024-04-27T00:02:21.206885+0000) 2024-04-27T00:02:35.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:35 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:35.335+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.408538+0000 front 2024-04-27T00:02:06.408413+0000 (oldest deadline 2024-04-27T00:02:29.908238+0000) 2024-04-27T00:02:36.041 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:35 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:35.652+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.482851+0000 front 2024-04-27T00:01:46.482649+0000 (oldest deadline 2024-04-27T00:02:12.382365+0000) 2024-04-27T00:02:36.041 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:35 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:35.652+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:56.484639+0000 front 2024-04-27T00:01:56.484700+0000 (oldest deadline 2024-04-27T00:02:20.584890+0000) 2024-04-27T00:02:36.041 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:35 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:35.652+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.487199+0000 front 2024-04-27T00:02:03.487670+0000 (oldest deadline 2024-04-27T00:02:28.787386+0000) 2024-04-27T00:02:36.176 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:35 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:35.681+0000 7fb9af424700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:02:36.176 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:35 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:35.681+0000 7fb9af424700 -1 osd.7 97 *** Got signal Terminated *** 2024-04-27T00:02:36.176 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:35 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:35.681+0000 7fb9af424700 -1 osd.7 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T00:02:36.220 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:35 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:35.796+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:50.039612+0000 front 2024-04-27T00:01:50.039500+0000 (oldest deadline 2024-04-27T00:02:14.139555+0000) 2024-04-27T00:02:36.220 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:35 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:35.796+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.840368+0000 front 2024-04-27T00:01:55.840396+0000 (oldest deadline 2024-04-27T00:02:21.740426+0000) 2024-04-27T00:02:36.220 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:35 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:35.796+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.342128+0000 front 2024-04-27T00:02:06.342039+0000 (oldest deadline 2024-04-27T00:02:31.641818+0000) 2024-04-27T00:02:36.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:36 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:36.219+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:36.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:36 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:36.219+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:02:36.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:36 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:36.219+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.693019+0000 front 2024-04-27T00:02:03.693029+0000 (oldest deadline 2024-04-27T00:02:27.792790+0000) 2024-04-27T00:02:36.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:36 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:36.250+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.718315+0000 front 2024-04-27T00:01:46.718291+0000 (oldest deadline 2024-04-27T00:02:10.818574+0000) 2024-04-27T00:02:36.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:36 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:36.250+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:57.220259+0000 front 2024-04-27T00:01:57.220356+0000 (oldest deadline 2024-04-27T00:02:18.920342+0000) 2024-04-27T00:02:36.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:36 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:36.250+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:05.723455+0000 front 2024-04-27T00:02:05.723579+0000 (oldest deadline 2024-04-27T00:02:27.423213+0000) 2024-04-27T00:02:36.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:36 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:36.352+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:36.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:36 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:36.352+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:02:36.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:36 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:36.352+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.306812+0000 front 2024-04-27T00:01:55.306787+0000 (oldest deadline 2024-04-27T00:02:21.206885+0000) 2024-04-27T00:02:36.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:36 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:36.352+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.408538+0000 front 2024-04-27T00:02:06.408413+0000 (oldest deadline 2024-04-27T00:02:29.908238+0000) 2024-04-27T00:02:37.041 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:36 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:36.634+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.482851+0000 front 2024-04-27T00:01:46.482649+0000 (oldest deadline 2024-04-27T00:02:12.382365+0000) 2024-04-27T00:02:37.041 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:36 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:36.634+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:56.484639+0000 front 2024-04-27T00:01:56.484700+0000 (oldest deadline 2024-04-27T00:02:20.584890+0000) 2024-04-27T00:02:37.041 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:36 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:36.634+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.487199+0000 front 2024-04-27T00:02:03.487670+0000 (oldest deadline 2024-04-27T00:02:28.787386+0000) 2024-04-27T00:02:37.049 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:36 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:36.768+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:50.039612+0000 front 2024-04-27T00:01:50.039500+0000 (oldest deadline 2024-04-27T00:02:14.139555+0000) 2024-04-27T00:02:37.049 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:36 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:36.768+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.840368+0000 front 2024-04-27T00:01:55.840396+0000 (oldest deadline 2024-04-27T00:02:21.740426+0000) 2024-04-27T00:02:37.050 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:36 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:36.769+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.342128+0000 front 2024-04-27T00:02:06.342039+0000 (oldest deadline 2024-04-27T00:02:31.641818+0000) 2024-04-27T00:02:37.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:37 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:37.240+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:37.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:37 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:37.240+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:02:37.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:37 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:37.240+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.693019+0000 front 2024-04-27T00:02:03.693029+0000 (oldest deadline 2024-04-27T00:02:27.792790+0000) 2024-04-27T00:02:37.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:37 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:37.240+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.094253+0000 front 2024-04-27T00:02:13.094226+0000 (oldest deadline 2024-04-27T00:02:37.194152+0000) 2024-04-27T00:02:37.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:37 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:37.223+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.718315+0000 front 2024-04-27T00:01:46.718291+0000 (oldest deadline 2024-04-27T00:02:10.818574+0000) 2024-04-27T00:02:37.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:37 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:37.223+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:57.220259+0000 front 2024-04-27T00:01:57.220356+0000 (oldest deadline 2024-04-27T00:02:18.920342+0000) 2024-04-27T00:02:37.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:37 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:37.223+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:05.723455+0000 front 2024-04-27T00:02:05.723579+0000 (oldest deadline 2024-04-27T00:02:27.423213+0000) 2024-04-27T00:02:37.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:37 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:37.342+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:37.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:37 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:37.342+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:02:37.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:37 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:37.342+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.306812+0000 front 2024-04-27T00:01:55.306787+0000 (oldest deadline 2024-04-27T00:02:21.206885+0000) 2024-04-27T00:02:37.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:37 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:37.342+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.408538+0000 front 2024-04-27T00:02:06.408413+0000 (oldest deadline 2024-04-27T00:02:29.908238+0000) 2024-04-27T00:02:38.040 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:37 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:37.649+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.482851+0000 front 2024-04-27T00:01:46.482649+0000 (oldest deadline 2024-04-27T00:02:12.382365+0000) 2024-04-27T00:02:38.041 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:37 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:37.649+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:56.484639+0000 front 2024-04-27T00:01:56.484700+0000 (oldest deadline 2024-04-27T00:02:20.584890+0000) 2024-04-27T00:02:38.041 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:37 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:37.649+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.487199+0000 front 2024-04-27T00:02:03.487670+0000 (oldest deadline 2024-04-27T00:02:28.787386+0000) 2024-04-27T00:02:38.047 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:37 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:37.730+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:50.039612+0000 front 2024-04-27T00:01:50.039500+0000 (oldest deadline 2024-04-27T00:02:14.139555+0000) 2024-04-27T00:02:38.047 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:37 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:37.730+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.840368+0000 front 2024-04-27T00:01:55.840396+0000 (oldest deadline 2024-04-27T00:02:21.740426+0000) 2024-04-27T00:02:38.047 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:37 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:37.730+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.342128+0000 front 2024-04-27T00:02:06.342039+0000 (oldest deadline 2024-04-27T00:02:31.641818+0000) 2024-04-27T00:02:38.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:38 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:38.200+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:38.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:38 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:38.200+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:02:38.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:38 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:38.200+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.693019+0000 front 2024-04-27T00:02:03.693029+0000 (oldest deadline 2024-04-27T00:02:27.792790+0000) 2024-04-27T00:02:38.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:38 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:38.200+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.094253+0000 front 2024-04-27T00:02:13.094226+0000 (oldest deadline 2024-04-27T00:02:37.194152+0000) 2024-04-27T00:02:38.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:38 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:38.175+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.718315+0000 front 2024-04-27T00:01:46.718291+0000 (oldest deadline 2024-04-27T00:02:10.818574+0000) 2024-04-27T00:02:38.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:38 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:38.175+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:57.220259+0000 front 2024-04-27T00:01:57.220356+0000 (oldest deadline 2024-04-27T00:02:18.920342+0000) 2024-04-27T00:02:38.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:38 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:38.175+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:05.723455+0000 front 2024-04-27T00:02:05.723579+0000 (oldest deadline 2024-04-27T00:02:27.423213+0000) 2024-04-27T00:02:38.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:38 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:38.175+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:09.724185+0000 front 2024-04-27T00:02:09.724206+0000 (oldest deadline 2024-04-27T00:02:37.324897+0000) 2024-04-27T00:02:38.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:38 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:38.346+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:38.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:38 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:38.346+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:02:38.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:38 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:38.346+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.306812+0000 front 2024-04-27T00:01:55.306787+0000 (oldest deadline 2024-04-27T00:02:21.206885+0000) 2024-04-27T00:02:38.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:38 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:38.346+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.408538+0000 front 2024-04-27T00:02:06.408413+0000 (oldest deadline 2024-04-27T00:02:29.908238+0000) 2024-04-27T00:02:39.041 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:38 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:38.661+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.482851+0000 front 2024-04-27T00:01:46.482649+0000 (oldest deadline 2024-04-27T00:02:12.382365+0000) 2024-04-27T00:02:39.041 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:38 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:38.661+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:56.484639+0000 front 2024-04-27T00:01:56.484700+0000 (oldest deadline 2024-04-27T00:02:20.584890+0000) 2024-04-27T00:02:39.041 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:38 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:38.661+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.487199+0000 front 2024-04-27T00:02:03.487670+0000 (oldest deadline 2024-04-27T00:02:28.787386+0000) 2024-04-27T00:02:39.047 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:38 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:38.748+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:50.039612+0000 front 2024-04-27T00:01:50.039500+0000 (oldest deadline 2024-04-27T00:02:14.139555+0000) 2024-04-27T00:02:39.048 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:38 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:38.748+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.840368+0000 front 2024-04-27T00:01:55.840396+0000 (oldest deadline 2024-04-27T00:02:21.740426+0000) 2024-04-27T00:02:39.048 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:38 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:38.748+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.342128+0000 front 2024-04-27T00:02:06.342039+0000 (oldest deadline 2024-04-27T00:02:31.641818+0000) 2024-04-27T00:02:39.048 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:38 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:38.748+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:12.142808+0000 front 2024-04-27T00:02:12.142760+0000 (oldest deadline 2024-04-27T00:02:38.042721+0000) 2024-04-27T00:02:39.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:39 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:39.243+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:39.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:39 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:39.243+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:02:39.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:39 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:39.243+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.693019+0000 front 2024-04-27T00:02:03.693029+0000 (oldest deadline 2024-04-27T00:02:27.792790+0000) 2024-04-27T00:02:39.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:39 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:39.243+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.094253+0000 front 2024-04-27T00:02:13.094226+0000 (oldest deadline 2024-04-27T00:02:37.194152+0000) 2024-04-27T00:02:39.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:39 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:39.221+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.718315+0000 front 2024-04-27T00:01:46.718291+0000 (oldest deadline 2024-04-27T00:02:10.818574+0000) 2024-04-27T00:02:39.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:39 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:39.221+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:57.220259+0000 front 2024-04-27T00:01:57.220356+0000 (oldest deadline 2024-04-27T00:02:18.920342+0000) 2024-04-27T00:02:39.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:39 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:39.221+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:05.723455+0000 front 2024-04-27T00:02:05.723579+0000 (oldest deadline 2024-04-27T00:02:27.423213+0000) 2024-04-27T00:02:39.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:39 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:39.221+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:09.724185+0000 front 2024-04-27T00:02:09.724206+0000 (oldest deadline 2024-04-27T00:02:37.324897+0000) 2024-04-27T00:02:39.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:39 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:39.365+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:39.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:39 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:39.365+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:02:39.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:39 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:39.365+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.306812+0000 front 2024-04-27T00:01:55.306787+0000 (oldest deadline 2024-04-27T00:02:21.206885+0000) 2024-04-27T00:02:39.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:39 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:39.365+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.408538+0000 front 2024-04-27T00:02:06.408413+0000 (oldest deadline 2024-04-27T00:02:29.908238+0000) 2024-04-27T00:02:39.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:39 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:39.365+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:09.908699+0000 front 2024-04-27T00:02:09.908717+0000 (oldest deadline 2024-04-27T00:02:39.309343+0000) 2024-04-27T00:02:40.041 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:39 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:39.700+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.482851+0000 front 2024-04-27T00:01:46.482649+0000 (oldest deadline 2024-04-27T00:02:12.382365+0000) 2024-04-27T00:02:40.041 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:39 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:39.700+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:56.484639+0000 front 2024-04-27T00:01:56.484700+0000 (oldest deadline 2024-04-27T00:02:20.584890+0000) 2024-04-27T00:02:40.041 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:39 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:39.700+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.487199+0000 front 2024-04-27T00:02:03.487670+0000 (oldest deadline 2024-04-27T00:02:28.787386+0000) 2024-04-27T00:02:40.041 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:39 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:39.700+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.489607+0000 front 2024-04-27T00:02:13.489578+0000 (oldest deadline 2024-04-27T00:02:38.789771+0000) 2024-04-27T00:02:40.047 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:39 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:39.750+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:50.039612+0000 front 2024-04-27T00:01:50.039500+0000 (oldest deadline 2024-04-27T00:02:14.139555+0000) 2024-04-27T00:02:40.047 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:39 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:39.750+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.840368+0000 front 2024-04-27T00:01:55.840396+0000 (oldest deadline 2024-04-27T00:02:21.740426+0000) 2024-04-27T00:02:40.047 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:39 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:39.750+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.342128+0000 front 2024-04-27T00:02:06.342039+0000 (oldest deadline 2024-04-27T00:02:31.641818+0000) 2024-04-27T00:02:40.047 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:39 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:39.750+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:12.142808+0000 front 2024-04-27T00:02:12.142760+0000 (oldest deadline 2024-04-27T00:02:38.042721+0000) 2024-04-27T00:02:40.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:40 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:40.245+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:40.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:40 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:40.245+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:02:40.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:40 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:40.245+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.693019+0000 front 2024-04-27T00:02:03.693029+0000 (oldest deadline 2024-04-27T00:02:27.792790+0000) 2024-04-27T00:02:40.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:40 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:40.245+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.094253+0000 front 2024-04-27T00:02:13.094226+0000 (oldest deadline 2024-04-27T00:02:37.194152+0000) 2024-04-27T00:02:40.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:40 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:40.186+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.718315+0000 front 2024-04-27T00:01:46.718291+0000 (oldest deadline 2024-04-27T00:02:10.818574+0000) 2024-04-27T00:02:40.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:40 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:40.186+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:57.220259+0000 front 2024-04-27T00:01:57.220356+0000 (oldest deadline 2024-04-27T00:02:18.920342+0000) 2024-04-27T00:02:40.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:40 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:40.186+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:05.723455+0000 front 2024-04-27T00:02:05.723579+0000 (oldest deadline 2024-04-27T00:02:27.423213+0000) 2024-04-27T00:02:40.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:40 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7[50445]: 2024-04-27T00:02:40.186+0000 7fb9a7a10700 -1 osd.7 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:09.724185+0000 front 2024-04-27T00:02:09.724206+0000 (oldest deadline 2024-04-27T00:02:37.324897+0000) 2024-04-27T00:02:40.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:40 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:40.332+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:40.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:40 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:40.332+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:02:40.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:40 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:40.332+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.306812+0000 front 2024-04-27T00:01:55.306787+0000 (oldest deadline 2024-04-27T00:02:21.206885+0000) 2024-04-27T00:02:40.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:40 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:40.332+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.408538+0000 front 2024-04-27T00:02:06.408413+0000 (oldest deadline 2024-04-27T00:02:29.908238+0000) 2024-04-27T00:02:40.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:40 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:40.332+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:09.908699+0000 front 2024-04-27T00:02:09.908717+0000 (oldest deadline 2024-04-27T00:02:39.309343+0000) 2024-04-27T00:02:41.041 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:40 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:40.745+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.482851+0000 front 2024-04-27T00:01:46.482649+0000 (oldest deadline 2024-04-27T00:02:12.382365+0000) 2024-04-27T00:02:41.041 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:40 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:40.745+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:56.484639+0000 front 2024-04-27T00:01:56.484700+0000 (oldest deadline 2024-04-27T00:02:20.584890+0000) 2024-04-27T00:02:41.041 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:40 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:40.745+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.487199+0000 front 2024-04-27T00:02:03.487670+0000 (oldest deadline 2024-04-27T00:02:28.787386+0000) 2024-04-27T00:02:41.041 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:40 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:40.745+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.489607+0000 front 2024-04-27T00:02:13.489578+0000 (oldest deadline 2024-04-27T00:02:38.789771+0000) 2024-04-27T00:02:41.047 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:40 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:40.773+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:50.039612+0000 front 2024-04-27T00:01:50.039500+0000 (oldest deadline 2024-04-27T00:02:14.139555+0000) 2024-04-27T00:02:41.047 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:40 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:40.773+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.840368+0000 front 2024-04-27T00:01:55.840396+0000 (oldest deadline 2024-04-27T00:02:21.740426+0000) 2024-04-27T00:02:41.047 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:40 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:40.773+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.342128+0000 front 2024-04-27T00:02:06.342039+0000 (oldest deadline 2024-04-27T00:02:31.641818+0000) 2024-04-27T00:02:41.047 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:40 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:40.773+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:12.142808+0000 front 2024-04-27T00:02:12.142760+0000 (oldest deadline 2024-04-27T00:02:38.042721+0000) 2024-04-27T00:02:41.382 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:41 smithi076 podman[67459]: 2024-04-27 00:02:41.125449129 +0000 UTC m=+5.565224347 container died b046f7b13818248ca64f0ea1647563c541572154b42421f632afdd4aef372296 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS) 2024-04-27T00:02:41.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:41 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:41.208+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:41.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:41 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:41.208+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:02:41.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:41 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:41.208+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.693019+0000 front 2024-04-27T00:02:03.693029+0000 (oldest deadline 2024-04-27T00:02:27.792790+0000) 2024-04-27T00:02:41.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:41 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:41.208+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.094253+0000 front 2024-04-27T00:02:13.094226+0000 (oldest deadline 2024-04-27T00:02:37.194152+0000) 2024-04-27T00:02:41.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:41 smithi076 podman[67459]: 2024-04-27 00:02:41.514931714 +0000 UTC m=+5.954706934 container cleanup b046f7b13818248ca64f0ea1647563c541572154b42421f632afdd4aef372296 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-27T00:02:41.676 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:41 smithi076 bash[67459]: ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7 2024-04-27T00:02:41.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:41 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:41.381+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:41.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:41 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:41.381+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:02:41.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:41 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:41.381+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.306812+0000 front 2024-04-27T00:01:55.306787+0000 (oldest deadline 2024-04-27T00:02:21.206885+0000) 2024-04-27T00:02:41.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:41 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:41.381+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.408538+0000 front 2024-04-27T00:02:06.408413+0000 (oldest deadline 2024-04-27T00:02:29.908238+0000) 2024-04-27T00:02:41.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:41 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:41.381+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:09.908699+0000 front 2024-04-27T00:02:09.908717+0000 (oldest deadline 2024-04-27T00:02:39.309343+0000) 2024-04-27T00:02:42.057 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:41 smithi076 podman[67477]: 2024-04-27 00:02:41.766168236 +0000 UTC m=+0.632504960 container remove b046f7b13818248ca64f0ea1647563c541572154b42421f632afdd4aef372296 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T00:02:42.215 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:41 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:41.817+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:50.039612+0000 front 2024-04-27T00:01:50.039500+0000 (oldest deadline 2024-04-27T00:02:14.139555+0000) 2024-04-27T00:02:42.216 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:41 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:41.818+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.840368+0000 front 2024-04-27T00:01:55.840396+0000 (oldest deadline 2024-04-27T00:02:21.740426+0000) 2024-04-27T00:02:42.216 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:41 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:41.818+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.342128+0000 front 2024-04-27T00:02:06.342039+0000 (oldest deadline 2024-04-27T00:02:31.641818+0000) 2024-04-27T00:02:42.216 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:41 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:41.818+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:12.142808+0000 front 2024-04-27T00:02:12.142760+0000 (oldest deadline 2024-04-27T00:02:38.042721+0000) 2024-04-27T00:02:42.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:41 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:41.792+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.482851+0000 front 2024-04-27T00:01:46.482649+0000 (oldest deadline 2024-04-27T00:02:12.382365+0000) 2024-04-27T00:02:42.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:41 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:41.792+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:56.484639+0000 front 2024-04-27T00:01:56.484700+0000 (oldest deadline 2024-04-27T00:02:20.584890+0000) 2024-04-27T00:02:42.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:41 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:41.792+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.487199+0000 front 2024-04-27T00:02:03.487670+0000 (oldest deadline 2024-04-27T00:02:28.787386+0000) 2024-04-27T00:02:42.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:41 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:41.792+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.489607+0000 front 2024-04-27T00:02:13.489578+0000 (oldest deadline 2024-04-27T00:02:38.789771+0000) 2024-04-27T00:02:42.310 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:42 smithi076 podman[67567]: 2024-04-27 00:02:41.960115738 +0000 UTC m=+0.018060882 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-27T00:02:42.311 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:42 smithi076 podman[67567]: 2024-04-27 00:02:42.075191844 +0000 UTC m=+0.133136976 container create 4c906b0e32c4b847644a912b4246c16042b273529a7c002d1b393fd780b53260 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5) 2024-04-27T00:02:42.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:42 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:42.214+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:42.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:42 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:42.214+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:02:42.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:42 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:42.214+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.693019+0000 front 2024-04-27T00:02:03.693029+0000 (oldest deadline 2024-04-27T00:02:27.792790+0000) 2024-04-27T00:02:42.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:42 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:42.214+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.094253+0000 front 2024-04-27T00:02:13.094226+0000 (oldest deadline 2024-04-27T00:02:37.194152+0000) 2024-04-27T00:02:42.603 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:42 smithi076 podman[67567]: 2024-04-27 00:02:42.310939228 +0000 UTC m=+0.368884361 container init 4c906b0e32c4b847644a912b4246c16042b273529a7c002d1b393fd780b53260 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T00:02:42.603 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:42 smithi076 podman[67567]: 2024-04-27 00:02:42.314189846 +0000 UTC m=+0.372134979 container start 4c906b0e32c4b847644a912b4246c16042b273529a7c002d1b393fd780b53260 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T00:02:42.604 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:42 smithi076 podman[67567]: 2024-04-27 00:02:42.427792875 +0000 UTC m=+0.485738007 container attach 4c906b0e32c4b847644a912b4246c16042b273529a7c002d1b393fd780b53260 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2) 2024-04-27T00:02:42.604 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:42 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:42.414+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:42.604 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:42 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:42.414+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:02:42.604 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:42 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:42.414+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.306812+0000 front 2024-04-27T00:01:55.306787+0000 (oldest deadline 2024-04-27T00:02:21.206885+0000) 2024-04-27T00:02:42.604 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:42 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:42.414+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.408538+0000 front 2024-04-27T00:02:06.408413+0000 (oldest deadline 2024-04-27T00:02:29.908238+0000) 2024-04-27T00:02:42.605 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:42 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:42.414+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:09.908699+0000 front 2024-04-27T00:02:09.908717+0000 (oldest deadline 2024-04-27T00:02:39.309343+0000) 2024-04-27T00:02:42.926 INFO:journalctl@ceph.osd.7.smithi076.stdout:Apr 27 00:02:42 smithi076 podman[67567]: 2024-04-27 00:02:42.603242011 +0000 UTC m=+0.661187150 container died 4c906b0e32c4b847644a912b4246c16042b273529a7c002d1b393fd780b53260 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-27T00:02:43.111 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:42 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:42.802+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.482851+0000 front 2024-04-27T00:01:46.482649+0000 (oldest deadline 2024-04-27T00:02:12.382365+0000) 2024-04-27T00:02:43.112 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:42 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:42.802+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:56.484639+0000 front 2024-04-27T00:01:56.484700+0000 (oldest deadline 2024-04-27T00:02:20.584890+0000) 2024-04-27T00:02:43.112 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:42 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:42.802+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.487199+0000 front 2024-04-27T00:02:03.487670+0000 (oldest deadline 2024-04-27T00:02:28.787386+0000) 2024-04-27T00:02:43.112 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:42 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:42.802+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.489607+0000 front 2024-04-27T00:02:13.489578+0000 (oldest deadline 2024-04-27T00:02:38.789771+0000) 2024-04-27T00:02:43.252 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:42 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:42.832+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:50.039612+0000 front 2024-04-27T00:01:50.039500+0000 (oldest deadline 2024-04-27T00:02:14.139555+0000) 2024-04-27T00:02:43.253 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:42 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:42.832+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.840368+0000 front 2024-04-27T00:01:55.840396+0000 (oldest deadline 2024-04-27T00:02:21.740426+0000) 2024-04-27T00:02:43.253 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:42 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:42.832+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.342128+0000 front 2024-04-27T00:02:06.342039+0000 (oldest deadline 2024-04-27T00:02:31.641818+0000) 2024-04-27T00:02:43.253 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:42 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:42.832+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:12.142808+0000 front 2024-04-27T00:02:12.142760+0000 (oldest deadline 2024-04-27T00:02:38.042721+0000) 2024-04-27T00:02:43.375 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@osd.7.service' 2024-04-27T00:02:43.420 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:02:43.420 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-27T00:02:43.420 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-27T00:02:43.420 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-3c8453ae-0425-11ef-bc93-c7b262605968@osd.8 2024-04-27T00:02:43.547 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:43 smithi136 systemd[1]: Stopping Ceph osd.8 for 3c8453ae-0425-11ef-bc93-c7b262605968... 2024-04-27T00:02:43.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:43 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:43.251+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:43.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:43 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:43.251+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:02:43.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:43 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:43.251+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.693019+0000 front 2024-04-27T00:02:03.693029+0000 (oldest deadline 2024-04-27T00:02:27.792790+0000) 2024-04-27T00:02:43.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:43 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:43.251+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.094253+0000 front 2024-04-27T00:02:13.094226+0000 (oldest deadline 2024-04-27T00:02:37.194152+0000) 2024-04-27T00:02:43.638 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:43 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:43.382+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:43.638 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:43 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:43.382+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:02:43.638 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:43 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:43.382+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.306812+0000 front 2024-04-27T00:01:55.306787+0000 (oldest deadline 2024-04-27T00:02:21.206885+0000) 2024-04-27T00:02:43.638 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:43 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:43.382+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.408538+0000 front 2024-04-27T00:02:06.408413+0000 (oldest deadline 2024-04-27T00:02:29.908238+0000) 2024-04-27T00:02:43.639 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:43 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:43.382+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:09.908699+0000 front 2024-04-27T00:02:09.908717+0000 (oldest deadline 2024-04-27T00:02:39.309343+0000) 2024-04-27T00:02:44.047 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:43 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:43.730+0000 7f81082bb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:02:44.047 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:43 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:43.730+0000 7f81082bb700 -1 osd.8 97 *** Got signal Terminated *** 2024-04-27T00:02:44.047 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:43 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:43.730+0000 7f81082bb700 -1 osd.8 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T00:02:44.047 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:43 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:43.854+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:50.039612+0000 front 2024-04-27T00:01:50.039500+0000 (oldest deadline 2024-04-27T00:02:14.139555+0000) 2024-04-27T00:02:44.048 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:43 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:43.854+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.840368+0000 front 2024-04-27T00:01:55.840396+0000 (oldest deadline 2024-04-27T00:02:21.740426+0000) 2024-04-27T00:02:44.048 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:43 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:43.854+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.342128+0000 front 2024-04-27T00:02:06.342039+0000 (oldest deadline 2024-04-27T00:02:31.641818+0000) 2024-04-27T00:02:44.048 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:43 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:43.854+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:12.142808+0000 front 2024-04-27T00:02:12.142760+0000 (oldest deadline 2024-04-27T00:02:38.042721+0000) 2024-04-27T00:02:44.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:43 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:43.845+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.482851+0000 front 2024-04-27T00:01:46.482649+0000 (oldest deadline 2024-04-27T00:02:12.382365+0000) 2024-04-27T00:02:44.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:43 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:43.845+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:56.484639+0000 front 2024-04-27T00:01:56.484700+0000 (oldest deadline 2024-04-27T00:02:20.584890+0000) 2024-04-27T00:02:44.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:43 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:43.845+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.487199+0000 front 2024-04-27T00:02:03.487670+0000 (oldest deadline 2024-04-27T00:02:28.787386+0000) 2024-04-27T00:02:44.292 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:43 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:43.845+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.489607+0000 front 2024-04-27T00:02:13.489578+0000 (oldest deadline 2024-04-27T00:02:38.789771+0000) 2024-04-27T00:02:44.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:44 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:44.245+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:44.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:44 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:44.245+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:02:44.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:44 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:44.245+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.693019+0000 front 2024-04-27T00:02:03.693029+0000 (oldest deadline 2024-04-27T00:02:27.792790+0000) 2024-04-27T00:02:44.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:44 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:44.245+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.094253+0000 front 2024-04-27T00:02:13.094226+0000 (oldest deadline 2024-04-27T00:02:37.194152+0000) 2024-04-27T00:02:44.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:44 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:44.383+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:44.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:44 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:44.383+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:02:44.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:44 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:44.383+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.306812+0000 front 2024-04-27T00:01:55.306787+0000 (oldest deadline 2024-04-27T00:02:21.206885+0000) 2024-04-27T00:02:44.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:44 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:44.383+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.408538+0000 front 2024-04-27T00:02:06.408413+0000 (oldest deadline 2024-04-27T00:02:29.908238+0000) 2024-04-27T00:02:44.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:44 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:44.383+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:09.908699+0000 front 2024-04-27T00:02:09.908717+0000 (oldest deadline 2024-04-27T00:02:39.309343+0000) 2024-04-27T00:02:45.255 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:44 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:44.848+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:50.039612+0000 front 2024-04-27T00:01:50.039500+0000 (oldest deadline 2024-04-27T00:02:14.139555+0000) 2024-04-27T00:02:45.255 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:44 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:44.848+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.840368+0000 front 2024-04-27T00:01:55.840396+0000 (oldest deadline 2024-04-27T00:02:21.740426+0000) 2024-04-27T00:02:45.256 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:44 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:44.848+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.342128+0000 front 2024-04-27T00:02:06.342039+0000 (oldest deadline 2024-04-27T00:02:31.641818+0000) 2024-04-27T00:02:45.256 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:44 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:44.848+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:12.142808+0000 front 2024-04-27T00:02:12.142760+0000 (oldest deadline 2024-04-27T00:02:38.042721+0000) 2024-04-27T00:02:45.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:44 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:44.885+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.482851+0000 front 2024-04-27T00:01:46.482649+0000 (oldest deadline 2024-04-27T00:02:12.382365+0000) 2024-04-27T00:02:45.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:44 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:44.885+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:56.484639+0000 front 2024-04-27T00:01:56.484700+0000 (oldest deadline 2024-04-27T00:02:20.584890+0000) 2024-04-27T00:02:45.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:44 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:44.885+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.487199+0000 front 2024-04-27T00:02:03.487670+0000 (oldest deadline 2024-04-27T00:02:28.787386+0000) 2024-04-27T00:02:45.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:44 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:44.885+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.489607+0000 front 2024-04-27T00:02:13.489578+0000 (oldest deadline 2024-04-27T00:02:38.789771+0000) 2024-04-27T00:02:45.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:45 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:45.254+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:45.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:45 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:45.254+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:02:45.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:45 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:45.254+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.693019+0000 front 2024-04-27T00:02:03.693029+0000 (oldest deadline 2024-04-27T00:02:27.792790+0000) 2024-04-27T00:02:45.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:45 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:45.254+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.094253+0000 front 2024-04-27T00:02:13.094226+0000 (oldest deadline 2024-04-27T00:02:37.194152+0000) 2024-04-27T00:02:45.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:45 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:45.347+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:45.678 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:45 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:45.347+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:02:45.678 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:45 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:45.347+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.306812+0000 front 2024-04-27T00:01:55.306787+0000 (oldest deadline 2024-04-27T00:02:21.206885+0000) 2024-04-27T00:02:45.678 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:45 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:45.347+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.408538+0000 front 2024-04-27T00:02:06.408413+0000 (oldest deadline 2024-04-27T00:02:29.908238+0000) 2024-04-27T00:02:45.678 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:45 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:45.347+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:09.908699+0000 front 2024-04-27T00:02:09.908717+0000 (oldest deadline 2024-04-27T00:02:39.309343+0000) 2024-04-27T00:02:45.678 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:45 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:45.347+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:19.309951+0000 front 2024-04-27T00:02:19.309966+0000 (oldest deadline 2024-04-27T00:02:45.209887+0000) 2024-04-27T00:02:46.243 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:45 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:45.832+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:50.039612+0000 front 2024-04-27T00:01:50.039500+0000 (oldest deadline 2024-04-27T00:02:14.139555+0000) 2024-04-27T00:02:46.243 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:45 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:45.832+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.840368+0000 front 2024-04-27T00:01:55.840396+0000 (oldest deadline 2024-04-27T00:02:21.740426+0000) 2024-04-27T00:02:46.243 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:45 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:45.832+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.342128+0000 front 2024-04-27T00:02:06.342039+0000 (oldest deadline 2024-04-27T00:02:31.641818+0000) 2024-04-27T00:02:46.243 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:45 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:45.832+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:12.142808+0000 front 2024-04-27T00:02:12.142760+0000 (oldest deadline 2024-04-27T00:02:38.042721+0000) 2024-04-27T00:02:46.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:45 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:45.917+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.482851+0000 front 2024-04-27T00:01:46.482649+0000 (oldest deadline 2024-04-27T00:02:12.382365+0000) 2024-04-27T00:02:46.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:45 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:45.917+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:56.484639+0000 front 2024-04-27T00:01:56.484700+0000 (oldest deadline 2024-04-27T00:02:20.584890+0000) 2024-04-27T00:02:46.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:45 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:45.917+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.487199+0000 front 2024-04-27T00:02:03.487670+0000 (oldest deadline 2024-04-27T00:02:28.787386+0000) 2024-04-27T00:02:46.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:45 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:45.917+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.489607+0000 front 2024-04-27T00:02:13.489578+0000 (oldest deadline 2024-04-27T00:02:38.789771+0000) 2024-04-27T00:02:46.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:46 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:46.242+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:46.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:46 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:46.242+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:02:46.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:46 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:46.242+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.693019+0000 front 2024-04-27T00:02:03.693029+0000 (oldest deadline 2024-04-27T00:02:27.792790+0000) 2024-04-27T00:02:46.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:46 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:46.242+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.094253+0000 front 2024-04-27T00:02:13.094226+0000 (oldest deadline 2024-04-27T00:02:37.194152+0000) 2024-04-27T00:02:46.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:46 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:46.362+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:46.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:46 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:46.362+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:02:46.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:46 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:46.362+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.306812+0000 front 2024-04-27T00:01:55.306787+0000 (oldest deadline 2024-04-27T00:02:21.206885+0000) 2024-04-27T00:02:46.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:46 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:46.362+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.408538+0000 front 2024-04-27T00:02:06.408413+0000 (oldest deadline 2024-04-27T00:02:29.908238+0000) 2024-04-27T00:02:46.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:46 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:46.362+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:09.908699+0000 front 2024-04-27T00:02:09.908717+0000 (oldest deadline 2024-04-27T00:02:39.309343+0000) 2024-04-27T00:02:46.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:46 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:46.362+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:19.309951+0000 front 2024-04-27T00:02:19.309966+0000 (oldest deadline 2024-04-27T00:02:45.209887+0000) 2024-04-27T00:02:47.264 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:46 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:46.851+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:50.039612+0000 front 2024-04-27T00:01:50.039500+0000 (oldest deadline 2024-04-27T00:02:14.139555+0000) 2024-04-27T00:02:47.265 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:46 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:46.851+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.840368+0000 front 2024-04-27T00:01:55.840396+0000 (oldest deadline 2024-04-27T00:02:21.740426+0000) 2024-04-27T00:02:47.265 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:46 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:46.851+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.342128+0000 front 2024-04-27T00:02:06.342039+0000 (oldest deadline 2024-04-27T00:02:31.641818+0000) 2024-04-27T00:02:47.265 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:46 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:46.851+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:12.142808+0000 front 2024-04-27T00:02:12.142760+0000 (oldest deadline 2024-04-27T00:02:38.042721+0000) 2024-04-27T00:02:47.265 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:46 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:46.851+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:20.244409+0000 front 2024-04-27T00:02:20.244439+0000 (oldest deadline 2024-04-27T00:02:46.144586+0000) 2024-04-27T00:02:47.290 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:46 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:46.876+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.482851+0000 front 2024-04-27T00:01:46.482649+0000 (oldest deadline 2024-04-27T00:02:12.382365+0000) 2024-04-27T00:02:47.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:46 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:46.876+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:56.484639+0000 front 2024-04-27T00:01:56.484700+0000 (oldest deadline 2024-04-27T00:02:20.584890+0000) 2024-04-27T00:02:47.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:46 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:46.876+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.487199+0000 front 2024-04-27T00:02:03.487670+0000 (oldest deadline 2024-04-27T00:02:28.787386+0000) 2024-04-27T00:02:47.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:46 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:46.876+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.489607+0000 front 2024-04-27T00:02:13.489578+0000 (oldest deadline 2024-04-27T00:02:38.789771+0000) 2024-04-27T00:02:47.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:47 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:47.263+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:47.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:47 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:47.263+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:02:47.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:47 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:47.263+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.693019+0000 front 2024-04-27T00:02:03.693029+0000 (oldest deadline 2024-04-27T00:02:27.792790+0000) 2024-04-27T00:02:47.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:47 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:47.263+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.094253+0000 front 2024-04-27T00:02:13.094226+0000 (oldest deadline 2024-04-27T00:02:37.194152+0000) 2024-04-27T00:02:47.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:47 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:47.341+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:47.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:47 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:47.341+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:02:47.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:47 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:47.341+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.306812+0000 front 2024-04-27T00:01:55.306787+0000 (oldest deadline 2024-04-27T00:02:21.206885+0000) 2024-04-27T00:02:47.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:47 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:47.341+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.408538+0000 front 2024-04-27T00:02:06.408413+0000 (oldest deadline 2024-04-27T00:02:29.908238+0000) 2024-04-27T00:02:47.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:47 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:47.341+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:09.908699+0000 front 2024-04-27T00:02:09.908717+0000 (oldest deadline 2024-04-27T00:02:39.309343+0000) 2024-04-27T00:02:47.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:47 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:47.341+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:19.309951+0000 front 2024-04-27T00:02:19.309966+0000 (oldest deadline 2024-04-27T00:02:45.209887+0000) 2024-04-27T00:02:48.247 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:47 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:47.827+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:50.039612+0000 front 2024-04-27T00:01:50.039500+0000 (oldest deadline 2024-04-27T00:02:14.139555+0000) 2024-04-27T00:02:48.247 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:47 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:47.827+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.840368+0000 front 2024-04-27T00:01:55.840396+0000 (oldest deadline 2024-04-27T00:02:21.740426+0000) 2024-04-27T00:02:48.247 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:47 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:47.827+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.342128+0000 front 2024-04-27T00:02:06.342039+0000 (oldest deadline 2024-04-27T00:02:31.641818+0000) 2024-04-27T00:02:48.247 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:47 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:47.827+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:12.142808+0000 front 2024-04-27T00:02:12.142760+0000 (oldest deadline 2024-04-27T00:02:38.042721+0000) 2024-04-27T00:02:48.248 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:47 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8[51270]: 2024-04-27T00:02:47.827+0000 7f81048af700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:20.244409+0000 front 2024-04-27T00:02:20.244439+0000 (oldest deadline 2024-04-27T00:02:46.144586+0000) 2024-04-27T00:02:48.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:47 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:47.885+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.482851+0000 front 2024-04-27T00:01:46.482649+0000 (oldest deadline 2024-04-27T00:02:12.382365+0000) 2024-04-27T00:02:48.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:47 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:47.885+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:56.484639+0000 front 2024-04-27T00:01:56.484700+0000 (oldest deadline 2024-04-27T00:02:20.584890+0000) 2024-04-27T00:02:48.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:47 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:47.885+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.487199+0000 front 2024-04-27T00:02:03.487670+0000 (oldest deadline 2024-04-27T00:02:28.787386+0000) 2024-04-27T00:02:48.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:47 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:47.885+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.489607+0000 front 2024-04-27T00:02:13.489578+0000 (oldest deadline 2024-04-27T00:02:38.789771+0000) 2024-04-27T00:02:48.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:47 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:47.885+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:21.692039+0000 front 2024-04-27T00:02:21.691945+0000 (oldest deadline 2024-04-27T00:02:46.893035+0000) 2024-04-27T00:02:48.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:48 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:48.246+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:48.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:48 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:48.246+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:02:48.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:48 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:48.246+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.693019+0000 front 2024-04-27T00:02:03.693029+0000 (oldest deadline 2024-04-27T00:02:27.792790+0000) 2024-04-27T00:02:48.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:48 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:48.246+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.094253+0000 front 2024-04-27T00:02:13.094226+0000 (oldest deadline 2024-04-27T00:02:37.194152+0000) 2024-04-27T00:02:48.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:48 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:48.375+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:48.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:48 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:48.375+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:02:48.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:48 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:48.375+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.306812+0000 front 2024-04-27T00:01:55.306787+0000 (oldest deadline 2024-04-27T00:02:21.206885+0000) 2024-04-27T00:02:48.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:48 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:48.375+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.408538+0000 front 2024-04-27T00:02:06.408413+0000 (oldest deadline 2024-04-27T00:02:29.908238+0000) 2024-04-27T00:02:48.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:48 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:48.375+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:09.908699+0000 front 2024-04-27T00:02:09.908717+0000 (oldest deadline 2024-04-27T00:02:39.309343+0000) 2024-04-27T00:02:48.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:48 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:48.375+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:19.309951+0000 front 2024-04-27T00:02:19.309966+0000 (oldest deadline 2024-04-27T00:02:45.209887+0000) 2024-04-27T00:02:49.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:48 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:48.841+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.482851+0000 front 2024-04-27T00:01:46.482649+0000 (oldest deadline 2024-04-27T00:02:12.382365+0000) 2024-04-27T00:02:49.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:48 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:48.841+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:56.484639+0000 front 2024-04-27T00:01:56.484700+0000 (oldest deadline 2024-04-27T00:02:20.584890+0000) 2024-04-27T00:02:49.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:48 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:48.841+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.487199+0000 front 2024-04-27T00:02:03.487670+0000 (oldest deadline 2024-04-27T00:02:28.787386+0000) 2024-04-27T00:02:49.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:48 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:48.841+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.489607+0000 front 2024-04-27T00:02:13.489578+0000 (oldest deadline 2024-04-27T00:02:38.789771+0000) 2024-04-27T00:02:49.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:48 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:48.841+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:21.692039+0000 front 2024-04-27T00:02:21.691945+0000 (oldest deadline 2024-04-27T00:02:46.893035+0000) 2024-04-27T00:02:49.357 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:49 smithi136 podman[68273]: 2024-04-27 00:02:49.054915262 +0000 UTC m=+5.442517984 container died e41def0d4aa1164963af21cd9cbe47f75eb9f8c53a45f44f5aa4afeedb5290c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-27T00:02:49.357 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:49 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:49.284+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:49.357 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:49 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:49.285+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:02:49.357 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:49 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:49.285+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.693019+0000 front 2024-04-27T00:02:03.693029+0000 (oldest deadline 2024-04-27T00:02:27.792790+0000) 2024-04-27T00:02:49.357 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:49 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:49.285+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.094253+0000 front 2024-04-27T00:02:13.094226+0000 (oldest deadline 2024-04-27T00:02:37.194152+0000) 2024-04-27T00:02:49.358 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:49 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:49.285+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:22.995698+0000 front 2024-04-27T00:02:22.995787+0000 (oldest deadline 2024-04-27T00:02:48.895802+0000) 2024-04-27T00:02:49.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:49 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:49.389+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:49.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:49 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:49.389+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:02:49.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:49 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:49.389+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.306812+0000 front 2024-04-27T00:01:55.306787+0000 (oldest deadline 2024-04-27T00:02:21.206885+0000) 2024-04-27T00:02:49.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:49 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:49.389+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.408538+0000 front 2024-04-27T00:02:06.408413+0000 (oldest deadline 2024-04-27T00:02:29.908238+0000) 2024-04-27T00:02:49.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:49 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:49.389+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:09.908699+0000 front 2024-04-27T00:02:09.908717+0000 (oldest deadline 2024-04-27T00:02:39.309343+0000) 2024-04-27T00:02:49.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:49 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:49.389+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:19.309951+0000 front 2024-04-27T00:02:19.309966+0000 (oldest deadline 2024-04-27T00:02:45.209887+0000) 2024-04-27T00:02:49.682 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:49 smithi136 podman[68273]: 2024-04-27 00:02:49.472669599 +0000 UTC m=+5.860272303 container cleanup e41def0d4aa1164963af21cd9cbe47f75eb9f8c53a45f44f5aa4afeedb5290c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-27T00:02:49.682 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:49 smithi136 bash[68273]: ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8 2024-04-27T00:02:50.005 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:49 smithi136 podman[68291]: 2024-04-27 00:02:49.682347759 +0000 UTC m=+0.625027026 container remove e41def0d4aa1164963af21cd9cbe47f75eb9f8c53a45f44f5aa4afeedb5290c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True) 2024-04-27T00:02:50.271 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:50 smithi136 podman[68387]: 2024-04-27 00:02:49.908871035 +0000 UTC m=+0.017760827 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-27T00:02:50.272 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:50 smithi136 podman[68387]: 2024-04-27 00:02:50.033619668 +0000 UTC m=+0.142509448 container create c1484ac4a24531005073158451ff761177e1fbe4e74de2cf1f7208ed5209be32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8-deactivate, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-27T00:02:50.272 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:50 smithi136 podman[68387]: 2024-04-27 00:02:50.271827051 +0000 UTC m=+0.380716831 container init c1484ac4a24531005073158451ff761177e1fbe4e74de2cf1f7208ed5209be32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-27T00:02:50.272 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:50 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:50.252+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:50.272 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:50 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:50.252+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:02:50.272 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:50 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:50.252+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.693019+0000 front 2024-04-27T00:02:03.693029+0000 (oldest deadline 2024-04-27T00:02:27.792790+0000) 2024-04-27T00:02:50.273 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:50 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:50.252+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.094253+0000 front 2024-04-27T00:02:13.094226+0000 (oldest deadline 2024-04-27T00:02:37.194152+0000) 2024-04-27T00:02:50.273 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:50 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:50.252+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:22.995698+0000 front 2024-04-27T00:02:22.995787+0000 (oldest deadline 2024-04-27T00:02:48.895802+0000) 2024-04-27T00:02:50.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:49 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:49.800+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.482851+0000 front 2024-04-27T00:01:46.482649+0000 (oldest deadline 2024-04-27T00:02:12.382365+0000) 2024-04-27T00:02:50.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:49 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:49.800+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:56.484639+0000 front 2024-04-27T00:01:56.484700+0000 (oldest deadline 2024-04-27T00:02:20.584890+0000) 2024-04-27T00:02:50.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:49 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:49.800+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.487199+0000 front 2024-04-27T00:02:03.487670+0000 (oldest deadline 2024-04-27T00:02:28.787386+0000) 2024-04-27T00:02:50.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:49 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:49.800+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.489607+0000 front 2024-04-27T00:02:13.489578+0000 (oldest deadline 2024-04-27T00:02:38.789771+0000) 2024-04-27T00:02:50.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:49 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:49.800+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:21.692039+0000 front 2024-04-27T00:02:21.691945+0000 (oldest deadline 2024-04-27T00:02:46.893035+0000) 2024-04-27T00:02:50.547 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:50 smithi136 podman[68387]: 2024-04-27 00:02:50.275416837 +0000 UTC m=+0.384306619 container start c1484ac4a24531005073158451ff761177e1fbe4e74de2cf1f7208ed5209be32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-27T00:02:50.547 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:50 smithi136 podman[68387]: 2024-04-27 00:02:50.380237698 +0000 UTC m=+0.489127492 container attach c1484ac4a24531005073158451ff761177e1fbe4e74de2cf1f7208ed5209be32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T00:02:50.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:50 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:50.381+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:50.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:50 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:50.381+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:02:50.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:50 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:50.381+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.306812+0000 front 2024-04-27T00:01:55.306787+0000 (oldest deadline 2024-04-27T00:02:21.206885+0000) 2024-04-27T00:02:50.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:50 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:50.381+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.408538+0000 front 2024-04-27T00:02:06.408413+0000 (oldest deadline 2024-04-27T00:02:29.908238+0000) 2024-04-27T00:02:50.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:50 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:50.381+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:09.908699+0000 front 2024-04-27T00:02:09.908717+0000 (oldest deadline 2024-04-27T00:02:39.309343+0000) 2024-04-27T00:02:50.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:50 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:50.381+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:19.309951+0000 front 2024-04-27T00:02:19.309966+0000 (oldest deadline 2024-04-27T00:02:45.209887+0000) 2024-04-27T00:02:50.933 INFO:journalctl@ceph.osd.8.smithi136.stdout:Apr 27 00:02:50 smithi136 podman[68387]: 2024-04-27 00:02:50.563388675 +0000 UTC m=+0.672278449 container died c1484ac4a24531005073158451ff761177e1fbe4e74de2cf1f7208ed5209be32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-8-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, GIT_BRANCH=HEAD) 2024-04-27T00:02:51.240 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@osd.8.service' 2024-04-27T00:02:51.281 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:02:51.281 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-27T00:02:51.281 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-27T00:02:51.282 DEBUG:teuthology.orchestra.run.smithi072:> sudo systemctl stop ceph-3c8453ae-0425-11ef-bc93-c7b262605968@osd.9 2024-04-27T00:02:51.290 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:50 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:50.819+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.482851+0000 front 2024-04-27T00:01:46.482649+0000 (oldest deadline 2024-04-27T00:02:12.382365+0000) 2024-04-27T00:02:51.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:50 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:50.819+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:56.484639+0000 front 2024-04-27T00:01:56.484700+0000 (oldest deadline 2024-04-27T00:02:20.584890+0000) 2024-04-27T00:02:51.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:50 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:50.819+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.487199+0000 front 2024-04-27T00:02:03.487670+0000 (oldest deadline 2024-04-27T00:02:28.787386+0000) 2024-04-27T00:02:51.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:50 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:50.819+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.489607+0000 front 2024-04-27T00:02:13.489578+0000 (oldest deadline 2024-04-27T00:02:38.789771+0000) 2024-04-27T00:02:51.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:50 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:50.819+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:21.692039+0000 front 2024-04-27T00:02:21.691945+0000 (oldest deadline 2024-04-27T00:02:46.893035+0000) 2024-04-27T00:02:51.492 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:51 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:51.256+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:51.493 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:51 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:51.256+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:02:51.493 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:51 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:51.256+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.693019+0000 front 2024-04-27T00:02:03.693029+0000 (oldest deadline 2024-04-27T00:02:27.792790+0000) 2024-04-27T00:02:51.493 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:51 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:51.256+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.094253+0000 front 2024-04-27T00:02:13.094226+0000 (oldest deadline 2024-04-27T00:02:37.194152+0000) 2024-04-27T00:02:51.493 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:51 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:51.256+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:22.995698+0000 front 2024-04-27T00:02:22.995787+0000 (oldest deadline 2024-04-27T00:02:48.895802+0000) 2024-04-27T00:02:51.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:51 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:51.361+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:51.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:51 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:51.361+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:02:51.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:51 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:51.361+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.306812+0000 front 2024-04-27T00:01:55.306787+0000 (oldest deadline 2024-04-27T00:02:21.206885+0000) 2024-04-27T00:02:51.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:51 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:51.361+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.408538+0000 front 2024-04-27T00:02:06.408413+0000 (oldest deadline 2024-04-27T00:02:29.908238+0000) 2024-04-27T00:02:51.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:51 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:51.361+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:09.908699+0000 front 2024-04-27T00:02:09.908717+0000 (oldest deadline 2024-04-27T00:02:39.309343+0000) 2024-04-27T00:02:51.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:51 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:51.361+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:19.309951+0000 front 2024-04-27T00:02:19.309966+0000 (oldest deadline 2024-04-27T00:02:45.209887+0000) 2024-04-27T00:02:51.791 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:51 smithi072 systemd[1]: Stopping Ceph osd.9 for 3c8453ae-0425-11ef-bc93-c7b262605968... 2024-04-27T00:02:52.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:51 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:51.855+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.482851+0000 front 2024-04-27T00:01:46.482649+0000 (oldest deadline 2024-04-27T00:02:12.382365+0000) 2024-04-27T00:02:52.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:51 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:51.855+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:56.484639+0000 front 2024-04-27T00:01:56.484700+0000 (oldest deadline 2024-04-27T00:02:20.584890+0000) 2024-04-27T00:02:52.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:51 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:51.855+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.487199+0000 front 2024-04-27T00:02:03.487670+0000 (oldest deadline 2024-04-27T00:02:28.787386+0000) 2024-04-27T00:02:52.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:51 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:51.855+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.489607+0000 front 2024-04-27T00:02:13.489578+0000 (oldest deadline 2024-04-27T00:02:38.789771+0000) 2024-04-27T00:02:52.292 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:51 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:51.855+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:21.692039+0000 front 2024-04-27T00:02:21.691945+0000 (oldest deadline 2024-04-27T00:02:46.893035+0000) 2024-04-27T00:02:52.292 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:51 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:51.879+0000 7f51d6e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:02:52.292 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:51 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:51.880+0000 7f51d6e00700 -1 osd.9 97 *** Got signal Terminated *** 2024-04-27T00:02:52.292 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:51 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:51.880+0000 7f51d6e00700 -1 osd.9 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T00:02:52.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:52 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:52.247+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:52.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:52 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:52.247+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:02:52.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:52 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:52.247+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.693019+0000 front 2024-04-27T00:02:03.693029+0000 (oldest deadline 2024-04-27T00:02:27.792790+0000) 2024-04-27T00:02:52.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:52 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:52.247+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.094253+0000 front 2024-04-27T00:02:13.094226+0000 (oldest deadline 2024-04-27T00:02:37.194152+0000) 2024-04-27T00:02:52.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:52 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:52.247+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:22.995698+0000 front 2024-04-27T00:02:22.995787+0000 (oldest deadline 2024-04-27T00:02:48.895802+0000) 2024-04-27T00:02:52.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:52 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:52.358+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:52.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:52 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:52.358+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:02:52.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:52 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:52.358+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.306812+0000 front 2024-04-27T00:01:55.306787+0000 (oldest deadline 2024-04-27T00:02:21.206885+0000) 2024-04-27T00:02:52.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:52 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:52.358+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.408538+0000 front 2024-04-27T00:02:06.408413+0000 (oldest deadline 2024-04-27T00:02:29.908238+0000) 2024-04-27T00:02:52.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:52 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:52.358+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:09.908699+0000 front 2024-04-27T00:02:09.908717+0000 (oldest deadline 2024-04-27T00:02:39.309343+0000) 2024-04-27T00:02:52.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:52 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:52.358+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:19.309951+0000 front 2024-04-27T00:02:19.309966+0000 (oldest deadline 2024-04-27T00:02:45.209887+0000) 2024-04-27T00:02:53.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:52 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:52.880+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.482851+0000 front 2024-04-27T00:01:46.482649+0000 (oldest deadline 2024-04-27T00:02:12.382365+0000) 2024-04-27T00:02:53.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:52 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:52.880+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:56.484639+0000 front 2024-04-27T00:01:56.484700+0000 (oldest deadline 2024-04-27T00:02:20.584890+0000) 2024-04-27T00:02:53.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:52 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:52.880+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.487199+0000 front 2024-04-27T00:02:03.487670+0000 (oldest deadline 2024-04-27T00:02:28.787386+0000) 2024-04-27T00:02:53.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:52 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:52.880+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.489607+0000 front 2024-04-27T00:02:13.489578+0000 (oldest deadline 2024-04-27T00:02:38.789771+0000) 2024-04-27T00:02:53.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:52 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:52.880+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:21.692039+0000 front 2024-04-27T00:02:21.691945+0000 (oldest deadline 2024-04-27T00:02:46.893035+0000) 2024-04-27T00:02:53.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:53 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:53.247+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:53.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:53 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:53.247+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:02:53.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:53 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:53.247+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.693019+0000 front 2024-04-27T00:02:03.693029+0000 (oldest deadline 2024-04-27T00:02:27.792790+0000) 2024-04-27T00:02:53.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:53 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:53.247+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.094253+0000 front 2024-04-27T00:02:13.094226+0000 (oldest deadline 2024-04-27T00:02:37.194152+0000) 2024-04-27T00:02:53.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:53 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:53.247+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:22.995698+0000 front 2024-04-27T00:02:22.995787+0000 (oldest deadline 2024-04-27T00:02:48.895802+0000) 2024-04-27T00:02:53.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:53 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:53.337+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:53.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:53 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:53.337+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:02:53.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:53 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:53.337+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.306812+0000 front 2024-04-27T00:01:55.306787+0000 (oldest deadline 2024-04-27T00:02:21.206885+0000) 2024-04-27T00:02:53.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:53 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:53.337+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.408538+0000 front 2024-04-27T00:02:06.408413+0000 (oldest deadline 2024-04-27T00:02:29.908238+0000) 2024-04-27T00:02:53.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:53 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:53.337+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:09.908699+0000 front 2024-04-27T00:02:09.908717+0000 (oldest deadline 2024-04-27T00:02:39.309343+0000) 2024-04-27T00:02:53.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:53 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:53.337+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:19.309951+0000 front 2024-04-27T00:02:19.309966+0000 (oldest deadline 2024-04-27T00:02:45.209887+0000) 2024-04-27T00:02:54.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:53 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:53.922+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.482851+0000 front 2024-04-27T00:01:46.482649+0000 (oldest deadline 2024-04-27T00:02:12.382365+0000) 2024-04-27T00:02:54.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:53 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:53.922+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:56.484639+0000 front 2024-04-27T00:01:56.484700+0000 (oldest deadline 2024-04-27T00:02:20.584890+0000) 2024-04-27T00:02:54.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:53 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:53.922+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.487199+0000 front 2024-04-27T00:02:03.487670+0000 (oldest deadline 2024-04-27T00:02:28.787386+0000) 2024-04-27T00:02:54.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:53 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:53.922+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.489607+0000 front 2024-04-27T00:02:13.489578+0000 (oldest deadline 2024-04-27T00:02:38.789771+0000) 2024-04-27T00:02:54.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:53 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:53.922+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:21.692039+0000 front 2024-04-27T00:02:21.691945+0000 (oldest deadline 2024-04-27T00:02:46.893035+0000) 2024-04-27T00:02:54.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:54 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:54.266+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:54.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:54 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:54.266+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:02:54.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:54 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:54.266+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.693019+0000 front 2024-04-27T00:02:03.693029+0000 (oldest deadline 2024-04-27T00:02:27.792790+0000) 2024-04-27T00:02:54.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:54 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:54.266+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.094253+0000 front 2024-04-27T00:02:13.094226+0000 (oldest deadline 2024-04-27T00:02:37.194152+0000) 2024-04-27T00:02:54.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:54 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:54.266+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:22.995698+0000 front 2024-04-27T00:02:22.995787+0000 (oldest deadline 2024-04-27T00:02:48.895802+0000) 2024-04-27T00:02:54.637 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:54 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:54.331+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:54.637 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:54 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:54.331+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:02:54.637 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:54 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:54.331+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.306812+0000 front 2024-04-27T00:01:55.306787+0000 (oldest deadline 2024-04-27T00:02:21.206885+0000) 2024-04-27T00:02:54.637 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:54 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:54.331+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.408538+0000 front 2024-04-27T00:02:06.408413+0000 (oldest deadline 2024-04-27T00:02:29.908238+0000) 2024-04-27T00:02:54.637 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:54 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:54.331+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:09.908699+0000 front 2024-04-27T00:02:09.908717+0000 (oldest deadline 2024-04-27T00:02:39.309343+0000) 2024-04-27T00:02:54.638 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:54 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:54.331+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:19.309951+0000 front 2024-04-27T00:02:19.309966+0000 (oldest deadline 2024-04-27T00:02:45.209887+0000) 2024-04-27T00:02:55.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:54 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:54.947+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.482851+0000 front 2024-04-27T00:01:46.482649+0000 (oldest deadline 2024-04-27T00:02:12.382365+0000) 2024-04-27T00:02:55.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:54 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:54.947+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:56.484639+0000 front 2024-04-27T00:01:56.484700+0000 (oldest deadline 2024-04-27T00:02:20.584890+0000) 2024-04-27T00:02:55.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:54 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:54.947+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.487199+0000 front 2024-04-27T00:02:03.487670+0000 (oldest deadline 2024-04-27T00:02:28.787386+0000) 2024-04-27T00:02:55.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:54 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:54.947+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.489607+0000 front 2024-04-27T00:02:13.489578+0000 (oldest deadline 2024-04-27T00:02:38.789771+0000) 2024-04-27T00:02:55.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:54 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:54.947+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:21.692039+0000 front 2024-04-27T00:02:21.691945+0000 (oldest deadline 2024-04-27T00:02:46.893035+0000) 2024-04-27T00:02:55.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:55 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:55.232+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:55.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:55 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:55.232+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:02:55.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:55 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:55.232+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.693019+0000 front 2024-04-27T00:02:03.693029+0000 (oldest deadline 2024-04-27T00:02:27.792790+0000) 2024-04-27T00:02:55.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:55 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:55.232+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.094253+0000 front 2024-04-27T00:02:13.094226+0000 (oldest deadline 2024-04-27T00:02:37.194152+0000) 2024-04-27T00:02:55.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:55 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:55.232+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:22.995698+0000 front 2024-04-27T00:02:22.995787+0000 (oldest deadline 2024-04-27T00:02:48.895802+0000) 2024-04-27T00:02:55.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:55 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:55.284+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:55.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:55 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:55.284+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:02:55.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:55 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:55.284+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.306812+0000 front 2024-04-27T00:01:55.306787+0000 (oldest deadline 2024-04-27T00:02:21.206885+0000) 2024-04-27T00:02:55.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:55 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:55.284+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.408538+0000 front 2024-04-27T00:02:06.408413+0000 (oldest deadline 2024-04-27T00:02:29.908238+0000) 2024-04-27T00:02:55.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:55 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:55.284+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:09.908699+0000 front 2024-04-27T00:02:09.908717+0000 (oldest deadline 2024-04-27T00:02:39.309343+0000) 2024-04-27T00:02:55.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:55 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:55.284+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:19.309951+0000 front 2024-04-27T00:02:19.309966+0000 (oldest deadline 2024-04-27T00:02:45.209887+0000) 2024-04-27T00:02:56.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:55 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:55.972+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:46.482851+0000 front 2024-04-27T00:01:46.482649+0000 (oldest deadline 2024-04-27T00:02:12.382365+0000) 2024-04-27T00:02:56.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:55 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:55.972+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:56.484639+0000 front 2024-04-27T00:01:56.484700+0000 (oldest deadline 2024-04-27T00:02:20.584890+0000) 2024-04-27T00:02:56.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:55 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:55.972+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.487199+0000 front 2024-04-27T00:02:03.487670+0000 (oldest deadline 2024-04-27T00:02:28.787386+0000) 2024-04-27T00:02:56.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:55 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:55.972+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.489607+0000 front 2024-04-27T00:02:13.489578+0000 (oldest deadline 2024-04-27T00:02:38.789771+0000) 2024-04-27T00:02:56.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:55 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9[61142]: 2024-04-27T00:02:55.972+0000 7f51d2600700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:21.692039+0000 front 2024-04-27T00:02:21.691945+0000 (oldest deadline 2024-04-27T00:02:46.893035+0000) 2024-04-27T00:02:56.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:56 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:56.249+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:56.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:56 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:56.249+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:02:56.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:56 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:56.249+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.693019+0000 front 2024-04-27T00:02:03.693029+0000 (oldest deadline 2024-04-27T00:02:27.792790+0000) 2024-04-27T00:02:56.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:56 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:56.249+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.094253+0000 front 2024-04-27T00:02:13.094226+0000 (oldest deadline 2024-04-27T00:02:37.194152+0000) 2024-04-27T00:02:56.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:56 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:56.249+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:22.995698+0000 front 2024-04-27T00:02:22.995787+0000 (oldest deadline 2024-04-27T00:02:48.895802+0000) 2024-04-27T00:02:56.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:56 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:56.249+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6822 osd.6 since back 2024-04-27T00:02:30.497796+0000 front 2024-04-27T00:02:30.497664+0000 (oldest deadline 2024-04-27T00:02:55.797482+0000) 2024-04-27T00:02:56.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:56 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:56.316+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:56.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:56 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:56.316+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:02:56.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:56 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:56.316+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.306812+0000 front 2024-04-27T00:01:55.306787+0000 (oldest deadline 2024-04-27T00:02:21.206885+0000) 2024-04-27T00:02:56.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:56 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:56.316+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.408538+0000 front 2024-04-27T00:02:06.408413+0000 (oldest deadline 2024-04-27T00:02:29.908238+0000) 2024-04-27T00:02:56.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:56 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:56.316+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:09.908699+0000 front 2024-04-27T00:02:09.908717+0000 (oldest deadline 2024-04-27T00:02:39.309343+0000) 2024-04-27T00:02:56.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:56 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:56.316+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:19.309951+0000 front 2024-04-27T00:02:19.309966+0000 (oldest deadline 2024-04-27T00:02:45.209887+0000) 2024-04-27T00:02:57.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:57 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:57.216+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:57.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:57 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:57.216+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:02:57.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:57 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:57.216+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.693019+0000 front 2024-04-27T00:02:03.693029+0000 (oldest deadline 2024-04-27T00:02:27.792790+0000) 2024-04-27T00:02:57.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:57 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:57.216+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.094253+0000 front 2024-04-27T00:02:13.094226+0000 (oldest deadline 2024-04-27T00:02:37.194152+0000) 2024-04-27T00:02:57.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:57 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:57.216+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:22.995698+0000 front 2024-04-27T00:02:22.995787+0000 (oldest deadline 2024-04-27T00:02:48.895802+0000) 2024-04-27T00:02:57.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:57 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:57.216+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6822 osd.6 since back 2024-04-27T00:02:30.497796+0000 front 2024-04-27T00:02:30.497664+0000 (oldest deadline 2024-04-27T00:02:55.797482+0000) 2024-04-27T00:02:57.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:57 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:57.339+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:57.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:57 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:57.339+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:02:57.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:57 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:57.339+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.306812+0000 front 2024-04-27T00:01:55.306787+0000 (oldest deadline 2024-04-27T00:02:21.206885+0000) 2024-04-27T00:02:57.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:57 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:57.339+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.408538+0000 front 2024-04-27T00:02:06.408413+0000 (oldest deadline 2024-04-27T00:02:29.908238+0000) 2024-04-27T00:02:57.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:57 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:57.339+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:09.908699+0000 front 2024-04-27T00:02:09.908717+0000 (oldest deadline 2024-04-27T00:02:39.309343+0000) 2024-04-27T00:02:57.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:57 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:57.339+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:19.309951+0000 front 2024-04-27T00:02:19.309966+0000 (oldest deadline 2024-04-27T00:02:45.209887+0000) 2024-04-27T00:02:57.699 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:57 smithi072 podman[85487]: 2024-04-27 00:02:57.423632447 +0000 UTC m=+5.690313493 container died f07e5c61c056c7b998c3489c92b0721e1528ed1dd9484e1c7a7993a66b93ba0e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-27T00:02:58.041 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:57 smithi072 podman[85487]: 2024-04-27 00:02:57.8094708 +0000 UTC m=+6.076151878 container cleanup f07e5c61c056c7b998c3489c92b0721e1528ed1dd9484e1c7a7993a66b93ba0e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=) 2024-04-27T00:02:58.041 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:57 smithi072 bash[85487]: ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9 2024-04-27T00:02:58.540 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:58 smithi072 podman[85505]: 2024-04-27 00:02:58.161890237 +0000 UTC m=+0.731678363 container remove f07e5c61c056c7b998c3489c92b0721e1528ed1dd9484e1c7a7993a66b93ba0e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-27T00:02:58.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:58 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:58.209+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:58.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:58 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:58.209+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:02:58.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:58 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:58.209+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.693019+0000 front 2024-04-27T00:02:03.693029+0000 (oldest deadline 2024-04-27T00:02:27.792790+0000) 2024-04-27T00:02:58.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:58 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:58.209+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.094253+0000 front 2024-04-27T00:02:13.094226+0000 (oldest deadline 2024-04-27T00:02:37.194152+0000) 2024-04-27T00:02:58.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:58 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:58.209+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:22.995698+0000 front 2024-04-27T00:02:22.995787+0000 (oldest deadline 2024-04-27T00:02:48.895802+0000) 2024-04-27T00:02:58.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:58 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:58.209+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6822 osd.6 since back 2024-04-27T00:02:30.497796+0000 front 2024-04-27T00:02:30.497664+0000 (oldest deadline 2024-04-27T00:02:55.797482+0000) 2024-04-27T00:02:58.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:58 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:58.375+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:58.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:58 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:58.375+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:02:58.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:58 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:58.375+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.306812+0000 front 2024-04-27T00:01:55.306787+0000 (oldest deadline 2024-04-27T00:02:21.206885+0000) 2024-04-27T00:02:58.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:58 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:58.375+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.408538+0000 front 2024-04-27T00:02:06.408413+0000 (oldest deadline 2024-04-27T00:02:29.908238+0000) 2024-04-27T00:02:58.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:58 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:58.375+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:09.908699+0000 front 2024-04-27T00:02:09.908717+0000 (oldest deadline 2024-04-27T00:02:39.309343+0000) 2024-04-27T00:02:58.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:58 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:58.375+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:19.309951+0000 front 2024-04-27T00:02:19.309966+0000 (oldest deadline 2024-04-27T00:02:45.209887+0000) 2024-04-27T00:02:58.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:58 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:58.375+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6822 osd.6 since back 2024-04-27T00:02:31.611698+0000 front 2024-04-27T00:02:31.611584+0000 (oldest deadline 2024-04-27T00:02:57.511345+0000) 2024-04-27T00:02:59.031 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:58 smithi072 podman[85594]: 2024-04-27 00:02:58.681870866 +0000 UTC m=+0.034302751 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-27T00:02:59.032 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:58 smithi072 podman[85594]: 2024-04-27 00:02:58.856947719 +0000 UTC m=+0.209379659 container create d3bafe1f80a02640eefb961cf8e4fd38b9391bf061a15a0e0e7ddc44a67d5f23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T00:02:59.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:59 smithi072 podman[85594]: 2024-04-27 00:02:59.162364031 +0000 UTC m=+0.514795952 container init d3bafe1f80a02640eefb961cf8e4fd38b9391bf061a15a0e0e7ddc44a67d5f23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T00:02:59.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:59 smithi072 podman[85594]: 2024-04-27 00:02:59.173252882 +0000 UTC m=+0.525684794 container start d3bafe1f80a02640eefb961cf8e4fd38b9391bf061a15a0e0e7ddc44a67d5f23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T00:02:59.291 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:59 smithi072 podman[85594]: 2024-04-27 00:02:59.252835154 +0000 UTC m=+0.605267075 container attach d3bafe1f80a02640eefb961cf8e4fd38b9391bf061a15a0e0e7ddc44a67d5f23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5) 2024-04-27T00:02:59.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:59 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:59.236+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:02:59.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:59 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:59.236+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:02:59.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:59 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:59.236+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.693019+0000 front 2024-04-27T00:02:03.693029+0000 (oldest deadline 2024-04-27T00:02:27.792790+0000) 2024-04-27T00:02:59.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:59 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:59.236+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.094253+0000 front 2024-04-27T00:02:13.094226+0000 (oldest deadline 2024-04-27T00:02:37.194152+0000) 2024-04-27T00:02:59.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:59 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:59.236+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:22.995698+0000 front 2024-04-27T00:02:22.995787+0000 (oldest deadline 2024-04-27T00:02:48.895802+0000) 2024-04-27T00:02:59.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:02:59 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:02:59.236+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6822 osd.6 since back 2024-04-27T00:02:30.497796+0000 front 2024-04-27T00:02:30.497664+0000 (oldest deadline 2024-04-27T00:02:55.797482+0000) 2024-04-27T00:02:59.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:59 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:59.392+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:02:59.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:59 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:59.392+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:02:59.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:59 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:59.392+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.306812+0000 front 2024-04-27T00:01:55.306787+0000 (oldest deadline 2024-04-27T00:02:21.206885+0000) 2024-04-27T00:02:59.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:59 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:59.392+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.408538+0000 front 2024-04-27T00:02:06.408413+0000 (oldest deadline 2024-04-27T00:02:29.908238+0000) 2024-04-27T00:02:59.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:59 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:59.392+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:09.908699+0000 front 2024-04-27T00:02:09.908717+0000 (oldest deadline 2024-04-27T00:02:39.309343+0000) 2024-04-27T00:02:59.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:59 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:59.392+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:19.309951+0000 front 2024-04-27T00:02:19.309966+0000 (oldest deadline 2024-04-27T00:02:45.209887+0000) 2024-04-27T00:02:59.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:02:59 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:02:59.392+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6822 osd.6 since back 2024-04-27T00:02:31.611698+0000 front 2024-04-27T00:02:31.611584+0000 (oldest deadline 2024-04-27T00:02:57.511345+0000) 2024-04-27T00:02:59.692 INFO:journalctl@ceph.osd.9.smithi072.stdout:Apr 27 00:02:59 smithi072 podman[85594]: 2024-04-27 00:02:59.62366312 +0000 UTC m=+0.976095000 container died d3bafe1f80a02640eefb961cf8e4fd38b9391bf061a15a0e0e7ddc44a67d5f23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-9-deactivate, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-27T00:03:00.288 DEBUG:teuthology.orchestra.run.smithi072:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@osd.9.service' 2024-04-27T00:03:00.380 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:03:00.381 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-27T00:03:00.381 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-27T00:03:00.381 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-3c8453ae-0425-11ef-bc93-c7b262605968@osd.10 2024-04-27T00:03:00.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:00 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:00.205+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:03:00.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:00 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:00.205+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:03:00.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:00 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:00.205+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.693019+0000 front 2024-04-27T00:02:03.693029+0000 (oldest deadline 2024-04-27T00:02:27.792790+0000) 2024-04-27T00:03:00.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:00 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:00.205+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.094253+0000 front 2024-04-27T00:02:13.094226+0000 (oldest deadline 2024-04-27T00:02:37.194152+0000) 2024-04-27T00:03:00.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:00 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:00.205+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:22.995698+0000 front 2024-04-27T00:02:22.995787+0000 (oldest deadline 2024-04-27T00:02:48.895802+0000) 2024-04-27T00:03:00.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:00 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:00.205+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6822 osd.6 since back 2024-04-27T00:02:30.497796+0000 front 2024-04-27T00:02:30.497664+0000 (oldest deadline 2024-04-27T00:02:55.797482+0000) 2024-04-27T00:03:00.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:00 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:03:00.396+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:03:00.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:00 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:03:00.396+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:03:00.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:00 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:03:00.396+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.306812+0000 front 2024-04-27T00:01:55.306787+0000 (oldest deadline 2024-04-27T00:02:21.206885+0000) 2024-04-27T00:03:00.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:00 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:03:00.396+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.408538+0000 front 2024-04-27T00:02:06.408413+0000 (oldest deadline 2024-04-27T00:02:29.908238+0000) 2024-04-27T00:03:00.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:00 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:03:00.396+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:09.908699+0000 front 2024-04-27T00:02:09.908717+0000 (oldest deadline 2024-04-27T00:02:39.309343+0000) 2024-04-27T00:03:00.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:00 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:03:00.396+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:19.309951+0000 front 2024-04-27T00:02:19.309966+0000 (oldest deadline 2024-04-27T00:02:45.209887+0000) 2024-04-27T00:03:00.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:00 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:03:00.396+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6822 osd.6 since back 2024-04-27T00:02:31.611698+0000 front 2024-04-27T00:02:31.611584+0000 (oldest deadline 2024-04-27T00:02:57.511345+0000) 2024-04-27T00:03:00.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:00 smithi076 systemd[1]: Stopping Ceph osd.10 for 3c8453ae-0425-11ef-bc93-c7b262605968... 2024-04-27T00:03:00.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:00 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:03:00.674+0000 7fb63f7b0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:03:00.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:00 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:03:00.674+0000 7fb63f7b0700 -1 osd.10 97 *** Got signal Terminated *** 2024-04-27T00:03:00.677 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:00 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:03:00.674+0000 7fb63f7b0700 -1 osd.10 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T00:03:01.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:01 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:01.240+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:03:01.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:01 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:01.240+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:03:01.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:01 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:01.240+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.693019+0000 front 2024-04-27T00:02:03.693029+0000 (oldest deadline 2024-04-27T00:02:27.792790+0000) 2024-04-27T00:03:01.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:01 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:01.240+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.094253+0000 front 2024-04-27T00:02:13.094226+0000 (oldest deadline 2024-04-27T00:02:37.194152+0000) 2024-04-27T00:03:01.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:01 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:01.240+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:22.995698+0000 front 2024-04-27T00:02:22.995787+0000 (oldest deadline 2024-04-27T00:02:48.895802+0000) 2024-04-27T00:03:01.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:01 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:01.240+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6822 osd.6 since back 2024-04-27T00:02:30.497796+0000 front 2024-04-27T00:02:30.497664+0000 (oldest deadline 2024-04-27T00:02:55.797482+0000) 2024-04-27T00:03:01.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:01 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:03:01.439+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:03:01.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:01 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:03:01.439+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:03:01.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:01 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:03:01.439+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.306812+0000 front 2024-04-27T00:01:55.306787+0000 (oldest deadline 2024-04-27T00:02:21.206885+0000) 2024-04-27T00:03:01.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:01 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:03:01.439+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.408538+0000 front 2024-04-27T00:02:06.408413+0000 (oldest deadline 2024-04-27T00:02:29.908238+0000) 2024-04-27T00:03:01.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:01 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:03:01.439+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:09.908699+0000 front 2024-04-27T00:02:09.908717+0000 (oldest deadline 2024-04-27T00:02:39.309343+0000) 2024-04-27T00:03:01.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:01 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:03:01.439+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:19.309951+0000 front 2024-04-27T00:02:19.309966+0000 (oldest deadline 2024-04-27T00:02:45.209887+0000) 2024-04-27T00:03:01.927 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:01 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:03:01.439+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6822 osd.6 since back 2024-04-27T00:02:31.611698+0000 front 2024-04-27T00:02:31.611584+0000 (oldest deadline 2024-04-27T00:02:57.511345+0000) 2024-04-27T00:03:02.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:02 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:02.222+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:03:02.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:02 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:02.222+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:03:02.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:02 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:02.222+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.693019+0000 front 2024-04-27T00:02:03.693029+0000 (oldest deadline 2024-04-27T00:02:27.792790+0000) 2024-04-27T00:03:02.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:02 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:02.222+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.094253+0000 front 2024-04-27T00:02:13.094226+0000 (oldest deadline 2024-04-27T00:02:37.194152+0000) 2024-04-27T00:03:02.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:02 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:02.222+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:22.995698+0000 front 2024-04-27T00:02:22.995787+0000 (oldest deadline 2024-04-27T00:02:48.895802+0000) 2024-04-27T00:03:02.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:02 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:02.222+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6822 osd.6 since back 2024-04-27T00:02:30.497796+0000 front 2024-04-27T00:02:30.497664+0000 (oldest deadline 2024-04-27T00:02:55.797482+0000) 2024-04-27T00:03:02.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:02 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:02.222+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6820 osd.7 since back 2024-04-27T00:02:35.798135+0000 front 2024-04-27T00:02:35.798247+0000 (oldest deadline 2024-04-27T00:03:01.697968+0000) 2024-04-27T00:03:02.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:02 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:03:02.455+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:03:02.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:02 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:03:02.455+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:03:02.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:02 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:03:02.455+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.306812+0000 front 2024-04-27T00:01:55.306787+0000 (oldest deadline 2024-04-27T00:02:21.206885+0000) 2024-04-27T00:03:02.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:02 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:03:02.456+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.408538+0000 front 2024-04-27T00:02:06.408413+0000 (oldest deadline 2024-04-27T00:02:29.908238+0000) 2024-04-27T00:03:02.927 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:02 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:03:02.456+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:09.908699+0000 front 2024-04-27T00:02:09.908717+0000 (oldest deadline 2024-04-27T00:02:39.309343+0000) 2024-04-27T00:03:02.927 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:02 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:03:02.456+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:19.309951+0000 front 2024-04-27T00:02:19.309966+0000 (oldest deadline 2024-04-27T00:02:45.209887+0000) 2024-04-27T00:03:02.927 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:02 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:03:02.456+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6822 osd.6 since back 2024-04-27T00:02:31.611698+0000 front 2024-04-27T00:02:31.611584+0000 (oldest deadline 2024-04-27T00:02:57.511345+0000) 2024-04-27T00:03:03.503 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:03 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:03.240+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:03:03.503 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:03 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:03.240+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:03:03.503 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:03 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:03.240+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.693019+0000 front 2024-04-27T00:02:03.693029+0000 (oldest deadline 2024-04-27T00:02:27.792790+0000) 2024-04-27T00:03:03.504 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:03 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:03.240+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.094253+0000 front 2024-04-27T00:02:13.094226+0000 (oldest deadline 2024-04-27T00:02:37.194152+0000) 2024-04-27T00:03:03.504 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:03 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:03.240+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:22.995698+0000 front 2024-04-27T00:02:22.995787+0000 (oldest deadline 2024-04-27T00:02:48.895802+0000) 2024-04-27T00:03:03.504 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:03 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:03.240+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6822 osd.6 since back 2024-04-27T00:02:30.497796+0000 front 2024-04-27T00:02:30.497664+0000 (oldest deadline 2024-04-27T00:02:55.797482+0000) 2024-04-27T00:03:03.504 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:03 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:03.240+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6820 osd.7 since back 2024-04-27T00:02:35.798135+0000 front 2024-04-27T00:02:35.798247+0000 (oldest deadline 2024-04-27T00:03:01.697968+0000) 2024-04-27T00:03:03.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:03 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:03:03.438+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:03:03.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:03 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:03:03.438+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:03:03.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:03 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:03:03.438+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.306812+0000 front 2024-04-27T00:01:55.306787+0000 (oldest deadline 2024-04-27T00:02:21.206885+0000) 2024-04-27T00:03:03.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:03 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:03:03.438+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.408538+0000 front 2024-04-27T00:02:06.408413+0000 (oldest deadline 2024-04-27T00:02:29.908238+0000) 2024-04-27T00:03:03.927 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:03 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:03:03.438+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:09.908699+0000 front 2024-04-27T00:02:09.908717+0000 (oldest deadline 2024-04-27T00:02:39.309343+0000) 2024-04-27T00:03:03.927 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:03 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:03:03.438+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:19.309951+0000 front 2024-04-27T00:02:19.309966+0000 (oldest deadline 2024-04-27T00:02:45.209887+0000) 2024-04-27T00:03:03.927 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:03 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:03:03.438+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6822 osd.6 since back 2024-04-27T00:02:31.611698+0000 front 2024-04-27T00:02:31.611584+0000 (oldest deadline 2024-04-27T00:02:57.511345+0000) 2024-04-27T00:03:04.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:04 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:04.266+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:03:04.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:04 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:04.266+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:03:04.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:04 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:04.266+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.693019+0000 front 2024-04-27T00:02:03.693029+0000 (oldest deadline 2024-04-27T00:02:27.792790+0000) 2024-04-27T00:03:04.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:04 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:04.266+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.094253+0000 front 2024-04-27T00:02:13.094226+0000 (oldest deadline 2024-04-27T00:02:37.194152+0000) 2024-04-27T00:03:04.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:04 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:04.266+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:22.995698+0000 front 2024-04-27T00:02:22.995787+0000 (oldest deadline 2024-04-27T00:02:48.895802+0000) 2024-04-27T00:03:04.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:04 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:04.266+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6822 osd.6 since back 2024-04-27T00:02:30.497796+0000 front 2024-04-27T00:02:30.497664+0000 (oldest deadline 2024-04-27T00:02:55.797482+0000) 2024-04-27T00:03:04.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:04 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:04.266+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6820 osd.7 since back 2024-04-27T00:02:35.798135+0000 front 2024-04-27T00:02:35.798247+0000 (oldest deadline 2024-04-27T00:03:01.697968+0000) 2024-04-27T00:03:04.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:04 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:03:04.483+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:03:04.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:04 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:03:04.483+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:03:04.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:04 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:03:04.483+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.306812+0000 front 2024-04-27T00:01:55.306787+0000 (oldest deadline 2024-04-27T00:02:21.206885+0000) 2024-04-27T00:03:04.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:04 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:03:04.483+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.408538+0000 front 2024-04-27T00:02:06.408413+0000 (oldest deadline 2024-04-27T00:02:29.908238+0000) 2024-04-27T00:03:04.927 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:04 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:03:04.483+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:09.908699+0000 front 2024-04-27T00:02:09.908717+0000 (oldest deadline 2024-04-27T00:02:39.309343+0000) 2024-04-27T00:03:04.927 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:04 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:03:04.483+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:19.309951+0000 front 2024-04-27T00:02:19.309966+0000 (oldest deadline 2024-04-27T00:02:45.209887+0000) 2024-04-27T00:03:04.927 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:04 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:03:04.484+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6822 osd.6 since back 2024-04-27T00:02:31.611698+0000 front 2024-04-27T00:02:31.611584+0000 (oldest deadline 2024-04-27T00:02:57.511345+0000) 2024-04-27T00:03:05.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:05 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:05.264+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:03:05.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:05 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:05.264+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:03:05.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:05 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:05.264+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.693019+0000 front 2024-04-27T00:02:03.693029+0000 (oldest deadline 2024-04-27T00:02:27.792790+0000) 2024-04-27T00:03:05.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:05 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:05.264+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.094253+0000 front 2024-04-27T00:02:13.094226+0000 (oldest deadline 2024-04-27T00:02:37.194152+0000) 2024-04-27T00:03:05.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:05 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:05.264+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:22.995698+0000 front 2024-04-27T00:02:22.995787+0000 (oldest deadline 2024-04-27T00:02:48.895802+0000) 2024-04-27T00:03:05.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:05 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:05.264+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6822 osd.6 since back 2024-04-27T00:02:30.497796+0000 front 2024-04-27T00:02:30.497664+0000 (oldest deadline 2024-04-27T00:02:55.797482+0000) 2024-04-27T00:03:05.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:05 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:05.264+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6820 osd.7 since back 2024-04-27T00:02:35.798135+0000 front 2024-04-27T00:02:35.798247+0000 (oldest deadline 2024-04-27T00:03:01.697968+0000) 2024-04-27T00:03:05.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:05 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:03:05.532+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:35.503935+0000 front 2024-04-27T00:01:35.503995+0000 (oldest deadline 2024-04-27T00:02:05.504308+0000) 2024-04-27T00:03:05.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:05 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:03:05.532+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6804 osd.1 since back 2024-04-27T00:01:47.805067+0000 front 2024-04-27T00:01:47.805018+0000 (oldest deadline 2024-04-27T00:02:11.905117+0000) 2024-04-27T00:03:05.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:05 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:03:05.532+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:55.306812+0000 front 2024-04-27T00:01:55.306787+0000 (oldest deadline 2024-04-27T00:02:21.206885+0000) 2024-04-27T00:03:05.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:05 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:03:05.532+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:06.408538+0000 front 2024-04-27T00:02:06.408413+0000 (oldest deadline 2024-04-27T00:02:29.908238+0000) 2024-04-27T00:03:05.926 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:05 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:03:05.532+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:09.908699+0000 front 2024-04-27T00:02:09.908717+0000 (oldest deadline 2024-04-27T00:02:39.309343+0000) 2024-04-27T00:03:05.927 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:05 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:03:05.532+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:19.309951+0000 front 2024-04-27T00:02:19.309966+0000 (oldest deadline 2024-04-27T00:02:45.209887+0000) 2024-04-27T00:03:05.927 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:05 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10[57033]: 2024-04-27T00:03:05.532+0000 7fb637d9c700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.72:6822 osd.6 since back 2024-04-27T00:02:31.611698+0000 front 2024-04-27T00:02:31.611584+0000 (oldest deadline 2024-04-27T00:02:57.511345+0000) 2024-04-27T00:03:06.292 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:06 smithi076 podman[67686]: 2024-04-27 00:03:06.031551736 +0000 UTC m=+5.458063746 container died 9564dae304f814bc91a2e8f8cd8487f2e532fcd04a6fe45631f939dc034e5065 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5) 2024-04-27T00:03:06.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:06 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:06.245+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:03:06.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:06 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:06.245+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:03:06.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:06 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:06.245+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.693019+0000 front 2024-04-27T00:02:03.693029+0000 (oldest deadline 2024-04-27T00:02:27.792790+0000) 2024-04-27T00:03:06.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:06 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:06.245+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.094253+0000 front 2024-04-27T00:02:13.094226+0000 (oldest deadline 2024-04-27T00:02:37.194152+0000) 2024-04-27T00:03:06.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:06 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:06.245+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:22.995698+0000 front 2024-04-27T00:02:22.995787+0000 (oldest deadline 2024-04-27T00:02:48.895802+0000) 2024-04-27T00:03:06.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:06 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:06.245+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6822 osd.6 since back 2024-04-27T00:02:30.497796+0000 front 2024-04-27T00:02:30.497664+0000 (oldest deadline 2024-04-27T00:02:55.797482+0000) 2024-04-27T00:03:06.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:06 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:06.245+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6820 osd.7 since back 2024-04-27T00:02:35.798135+0000 front 2024-04-27T00:02:35.798247+0000 (oldest deadline 2024-04-27T00:03:01.697968+0000) 2024-04-27T00:03:06.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:06 smithi076 podman[67686]: 2024-04-27 00:03:06.391154419 +0000 UTC m=+5.817666429 container cleanup 9564dae304f814bc91a2e8f8cd8487f2e532fcd04a6fe45631f939dc034e5065 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-27T00:03:06.676 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:06 smithi076 bash[67686]: ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10 2024-04-27T00:03:07.066 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:06 smithi076 podman[67703]: 2024-04-27 00:03:06.734010246 +0000 UTC m=+0.701710567 container remove 9564dae304f814bc91a2e8f8cd8487f2e532fcd04a6fe45631f939dc034e5065 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-27T00:03:07.328 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:07 smithi076 podman[67792]: 2024-04-27 00:03:06.968899212 +0000 UTC m=+0.018329837 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-27T00:03:07.329 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:07 smithi076 podman[67792]: 2024-04-27 00:03:07.101506785 +0000 UTC m=+0.150937434 container create 2740ec546607c64bd870cd4d3eea76e6dbdc45a2321c889416716e5c8a3d524a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-27T00:03:07.329 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:07 smithi076 podman[67792]: 2024-04-27 00:03:07.328540049 +0000 UTC m=+0.377970657 container init 2740ec546607c64bd870cd4d3eea76e6dbdc45a2321c889416716e5c8a3d524a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux ) 2024-04-27T00:03:07.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:07 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:07.246+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:03:07.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:07 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:07.246+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:03:07.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:07 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:07.246+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.693019+0000 front 2024-04-27T00:02:03.693029+0000 (oldest deadline 2024-04-27T00:02:27.792790+0000) 2024-04-27T00:03:07.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:07 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:07.246+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.094253+0000 front 2024-04-27T00:02:13.094226+0000 (oldest deadline 2024-04-27T00:02:37.194152+0000) 2024-04-27T00:03:07.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:07 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:07.246+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:22.995698+0000 front 2024-04-27T00:02:22.995787+0000 (oldest deadline 2024-04-27T00:02:48.895802+0000) 2024-04-27T00:03:07.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:07 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:07.246+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6822 osd.6 since back 2024-04-27T00:02:30.497796+0000 front 2024-04-27T00:02:30.497664+0000 (oldest deadline 2024-04-27T00:02:55.797482+0000) 2024-04-27T00:03:07.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:07 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:07.246+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6820 osd.7 since back 2024-04-27T00:02:35.798135+0000 front 2024-04-27T00:02:35.798247+0000 (oldest deadline 2024-04-27T00:03:01.697968+0000) 2024-04-27T00:03:07.617 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:07 smithi076 podman[67792]: 2024-04-27 00:03:07.331831813 +0000 UTC m=+0.381262429 container start 2740ec546607c64bd870cd4d3eea76e6dbdc45a2321c889416716e5c8a3d524a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-27T00:03:07.617 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:07 smithi076 podman[67792]: 2024-04-27 00:03:07.420198881 +0000 UTC m=+0.469629503 container attach 2740ec546607c64bd870cd4d3eea76e6dbdc45a2321c889416716e5c8a3d524a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T00:03:07.905 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:07 smithi076 podman[67792]: 2024-04-27 00:03:07.617764758 +0000 UTC m=+0.667195374 container died 2740ec546607c64bd870cd4d3eea76e6dbdc45a2321c889416716e5c8a3d524a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10-deactivate, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-27T00:03:08.156 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:08 smithi076 podman[67819]: 2024-04-27 00:03:08.139786188 +0000 UTC m=+0.513794718 container remove 2740ec546607c64bd870cd4d3eea76e6dbdc45a2321c889416716e5c8a3d524a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-10-deactivate, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T00:03:08.156 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:08 smithi076 systemd[1]: ceph-3c8453ae-0425-11ef-bc93-c7b262605968@osd.10.service: Deactivated successfully. 2024-04-27T00:03:08.156 INFO:journalctl@ceph.osd.10.smithi076.stdout:Apr 27 00:03:08 smithi076 systemd[1]: Stopped Ceph osd.10 for 3c8453ae-0425-11ef-bc93-c7b262605968. 2024-04-27T00:03:08.159 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@osd.10.service' 2024-04-27T00:03:08.195 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:03:08.195 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-27T00:03:08.196 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-27T00:03:08.196 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-3c8453ae-0425-11ef-bc93-c7b262605968@osd.11 2024-04-27T00:03:08.528 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:08 smithi136 systemd[1]: Stopping Ceph osd.11 for 3c8453ae-0425-11ef-bc93-c7b262605968... 2024-04-27T00:03:08.528 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:08 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:08.287+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:03:08.528 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:08 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:08.287+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:03:08.529 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:08 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:08.287+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.693019+0000 front 2024-04-27T00:02:03.693029+0000 (oldest deadline 2024-04-27T00:02:27.792790+0000) 2024-04-27T00:03:08.529 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:08 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:08.287+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.094253+0000 front 2024-04-27T00:02:13.094226+0000 (oldest deadline 2024-04-27T00:02:37.194152+0000) 2024-04-27T00:03:08.529 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:08 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:08.287+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:22.995698+0000 front 2024-04-27T00:02:22.995787+0000 (oldest deadline 2024-04-27T00:02:48.895802+0000) 2024-04-27T00:03:08.529 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:08 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:08.287+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6822 osd.6 since back 2024-04-27T00:02:30.497796+0000 front 2024-04-27T00:02:30.497664+0000 (oldest deadline 2024-04-27T00:02:55.797482+0000) 2024-04-27T00:03:08.529 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:08 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:08.287+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6820 osd.7 since back 2024-04-27T00:02:35.798135+0000 front 2024-04-27T00:02:35.798247+0000 (oldest deadline 2024-04-27T00:03:01.697968+0000) 2024-04-27T00:03:08.529 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:08 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:08.527+0000 7fe370d90700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:03:08.529 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:08 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:08.527+0000 7fe370d90700 -1 osd.11 97 *** Got signal Terminated *** 2024-04-27T00:03:08.529 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:08 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:08.527+0000 7fe370d90700 -1 osd.11 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-27T00:03:09.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:09 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:09.262+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:03:09.625 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:09 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:09.262+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:03:09.625 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:09 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:09.262+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.693019+0000 front 2024-04-27T00:02:03.693029+0000 (oldest deadline 2024-04-27T00:02:27.792790+0000) 2024-04-27T00:03:09.625 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:09 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:09.262+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.094253+0000 front 2024-04-27T00:02:13.094226+0000 (oldest deadline 2024-04-27T00:02:37.194152+0000) 2024-04-27T00:03:09.625 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:09 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:09.262+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:22.995698+0000 front 2024-04-27T00:02:22.995787+0000 (oldest deadline 2024-04-27T00:02:48.895802+0000) 2024-04-27T00:03:09.625 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:09 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:09.262+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6822 osd.6 since back 2024-04-27T00:02:30.497796+0000 front 2024-04-27T00:02:30.497664+0000 (oldest deadline 2024-04-27T00:02:55.797482+0000) 2024-04-27T00:03:09.625 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:09 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:09.262+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6820 osd.7 since back 2024-04-27T00:02:35.798135+0000 front 2024-04-27T00:02:35.798247+0000 (oldest deadline 2024-04-27T00:03:01.697968+0000) 2024-04-27T00:03:10.797 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:10 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:10.305+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:03:10.797 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:10 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:10.305+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:03:10.797 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:10 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:10.305+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.693019+0000 front 2024-04-27T00:02:03.693029+0000 (oldest deadline 2024-04-27T00:02:27.792790+0000) 2024-04-27T00:03:10.797 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:10 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:10.305+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.094253+0000 front 2024-04-27T00:02:13.094226+0000 (oldest deadline 2024-04-27T00:02:37.194152+0000) 2024-04-27T00:03:10.798 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:10 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:10.305+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:22.995698+0000 front 2024-04-27T00:02:22.995787+0000 (oldest deadline 2024-04-27T00:02:48.895802+0000) 2024-04-27T00:03:10.798 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:10 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:10.305+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6822 osd.6 since back 2024-04-27T00:02:30.497796+0000 front 2024-04-27T00:02:30.497664+0000 (oldest deadline 2024-04-27T00:02:55.797482+0000) 2024-04-27T00:03:10.798 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:10 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:10.305+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6820 osd.7 since back 2024-04-27T00:02:35.798135+0000 front 2024-04-27T00:02:35.798247+0000 (oldest deadline 2024-04-27T00:03:01.697968+0000) 2024-04-27T00:03:11.797 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:11 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:11.320+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:03:11.797 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:11 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:11.320+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:03:11.797 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:11 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:11.320+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.693019+0000 front 2024-04-27T00:02:03.693029+0000 (oldest deadline 2024-04-27T00:02:27.792790+0000) 2024-04-27T00:03:11.798 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:11 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:11.320+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.094253+0000 front 2024-04-27T00:02:13.094226+0000 (oldest deadline 2024-04-27T00:02:37.194152+0000) 2024-04-27T00:03:11.798 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:11 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:11.320+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:22.995698+0000 front 2024-04-27T00:02:22.995787+0000 (oldest deadline 2024-04-27T00:02:48.895802+0000) 2024-04-27T00:03:11.798 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:11 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:11.320+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6822 osd.6 since back 2024-04-27T00:02:30.497796+0000 front 2024-04-27T00:02:30.497664+0000 (oldest deadline 2024-04-27T00:02:55.797482+0000) 2024-04-27T00:03:11.798 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:11 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:11.320+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6820 osd.7 since back 2024-04-27T00:02:35.798135+0000 front 2024-04-27T00:02:35.798247+0000 (oldest deadline 2024-04-27T00:03:01.697968+0000) 2024-04-27T00:03:11.798 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:11 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:11.320+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6820 osd.8 since back 2024-04-27T00:02:46.299597+0000 front 2024-04-27T00:02:46.299577+0000 (oldest deadline 2024-04-27T00:03:10.399591+0000) 2024-04-27T00:03:12.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:12 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:12.281+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:03:12.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:12 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:12.281+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:03:12.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:12 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:12.281+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.693019+0000 front 2024-04-27T00:02:03.693029+0000 (oldest deadline 2024-04-27T00:02:27.792790+0000) 2024-04-27T00:03:12.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:12 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:12.281+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.094253+0000 front 2024-04-27T00:02:13.094226+0000 (oldest deadline 2024-04-27T00:02:37.194152+0000) 2024-04-27T00:03:12.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:12 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:12.281+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:22.995698+0000 front 2024-04-27T00:02:22.995787+0000 (oldest deadline 2024-04-27T00:02:48.895802+0000) 2024-04-27T00:03:12.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:12 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:12.281+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6822 osd.6 since back 2024-04-27T00:02:30.497796+0000 front 2024-04-27T00:02:30.497664+0000 (oldest deadline 2024-04-27T00:02:55.797482+0000) 2024-04-27T00:03:12.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:12 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:12.281+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6820 osd.7 since back 2024-04-27T00:02:35.798135+0000 front 2024-04-27T00:02:35.798247+0000 (oldest deadline 2024-04-27T00:03:01.697968+0000) 2024-04-27T00:03:12.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:12 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:12.281+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6820 osd.8 since back 2024-04-27T00:02:46.299597+0000 front 2024-04-27T00:02:46.299577+0000 (oldest deadline 2024-04-27T00:03:10.399591+0000) 2024-04-27T00:03:13.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:13 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:13.254+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6806 osd.0 since back 2024-04-27T00:01:38.188370+0000 front 2024-04-27T00:01:38.188177+0000 (oldest deadline 2024-04-27T00:02:04.088022+0000) 2024-04-27T00:03:13.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:13 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:13.254+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6804 osd.2 since back 2024-04-27T00:01:54.991380+0000 front 2024-04-27T00:01:54.991429+0000 (oldest deadline 2024-04-27T00:02:20.291446+0000) 2024-04-27T00:03:13.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:13 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:13.254+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6814 osd.3 since back 2024-04-27T00:02:03.693019+0000 front 2024-04-27T00:02:03.693029+0000 (oldest deadline 2024-04-27T00:02:27.792790+0000) 2024-04-27T00:03:13.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:13 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:13.254+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6812 osd.4 since back 2024-04-27T00:02:13.094253+0000 front 2024-04-27T00:02:13.094226+0000 (oldest deadline 2024-04-27T00:02:37.194152+0000) 2024-04-27T00:03:13.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:13 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:13.254+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6812 osd.5 since back 2024-04-27T00:02:22.995698+0000 front 2024-04-27T00:02:22.995787+0000 (oldest deadline 2024-04-27T00:02:48.895802+0000) 2024-04-27T00:03:13.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:13 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:13.254+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.72:6822 osd.6 since back 2024-04-27T00:02:30.497796+0000 front 2024-04-27T00:02:30.497664+0000 (oldest deadline 2024-04-27T00:02:55.797482+0000) 2024-04-27T00:03:13.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:13 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:13.254+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.76:6820 osd.7 since back 2024-04-27T00:02:35.798135+0000 front 2024-04-27T00:02:35.798247+0000 (oldest deadline 2024-04-27T00:03:01.697968+0000) 2024-04-27T00:03:13.548 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:13 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11[57876]: 2024-04-27T00:03:13.254+0000 7fe36d384700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.136:6820 osd.8 since back 2024-04-27T00:02:46.299597+0000 front 2024-04-27T00:02:46.299577+0000 (oldest deadline 2024-04-27T00:03:10.399591+0000) 2024-04-27T00:03:14.194 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:13 smithi136 podman[68503]: 2024-04-27 00:03:13.92454417 +0000 UTC m=+5.531273639 container died 573f0f625925d5def28dc2a633ef21b6f806a9b3c65e0a58fa7f592a584ed405 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T00:03:14.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:14 smithi136 podman[68503]: 2024-04-27 00:03:14.302264215 +0000 UTC m=+5.908993684 container cleanup 573f0f625925d5def28dc2a633ef21b6f806a9b3c65e0a58fa7f592a584ed405 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-27T00:03:14.547 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:14 smithi136 bash[68503]: ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11 2024-04-27T00:03:14.915 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:14 smithi136 podman[68520]: 2024-04-27 00:03:14.561597863 +0000 UTC m=+0.632416818 container remove 573f0f625925d5def28dc2a633ef21b6f806a9b3c65e0a58fa7f592a584ed405 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-27T00:03:15.168 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:14 smithi136 podman[68609]: 2024-04-27 00:03:14.818862899 +0000 UTC m=+0.017978239 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-27T00:03:15.168 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:14 smithi136 podman[68609]: 2024-04-27 00:03:14.920619287 +0000 UTC m=+0.119734605 container create b451275f2a8120da87fb6363eaeebfe58b8281f1cdfdd1feef9dad625cba508f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True) 2024-04-27T00:03:15.168 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:15 smithi136 podman[68609]: 2024-04-27 00:03:15.165050868 +0000 UTC m=+0.364166196 container init b451275f2a8120da87fb6363eaeebfe58b8281f1cdfdd1feef9dad625cba508f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.build-date=20240424, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-27T00:03:15.168 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:15 smithi136 podman[68609]: 2024-04-27 00:03:15.168352777 +0000 UTC m=+0.367468103 container start b451275f2a8120da87fb6363eaeebfe58b8281f1cdfdd1feef9dad625cba508f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-27T00:03:15.465 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:15 smithi136 podman[68609]: 2024-04-27 00:03:15.264900686 +0000 UTC m=+0.464016005 container attach b451275f2a8120da87fb6363eaeebfe58b8281f1cdfdd1feef9dad625cba508f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T00:03:15.759 INFO:journalctl@ceph.osd.11.smithi136.stdout:Apr 27 00:03:15 smithi136 podman[68609]: 2024-04-27 00:03:15.463501673 +0000 UTC m=+0.662617002 container died b451275f2a8120da87fb6363eaeebfe58b8281f1cdfdd1feef9dad625cba508f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-27T00:03:16.042 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@osd.11.service' 2024-04-27T00:03:16.078 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:03:16.079 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-27T00:03:16.080 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-27T00:03:16.080 DEBUG:teuthology.orchestra.run.smithi072:> sudo systemctl stop ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mds.a 2024-04-27T00:03:16.540 INFO:journalctl@ceph.mds.a.smithi072.stdout:Apr 27 00:03:16 smithi072 systemd[1]: Stopping Ceph mds.a for 3c8453ae-0425-11ef-bc93-c7b262605968... 2024-04-27T00:03:17.040 INFO:journalctl@ceph.mds.a.smithi072.stdout:Apr 27 00:03:16 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-a[68103]: 2024-04-27T00:03:16.638+0000 7f6efb200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:03:17.041 INFO:journalctl@ceph.mds.a.smithi072.stdout:Apr 27 00:03:16 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-a[68103]: 2024-04-27T00:03:16.638+0000 7f6efb200700 -1 mds.a *** got signal Terminated *** 2024-04-27T00:03:20.929 INFO:journalctl@ceph.mds.a.smithi072.stdout:Apr 27 00:03:20 smithi072 podman[85709]: 2024-04-27 00:03:20.650166276 +0000 UTC m=+4.163177813 container died 97c724a655b48d7f0ef0873189c364fd980ad547f4356a39593a78778676cce9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-a, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-27T00:03:21.290 INFO:journalctl@ceph.mds.a.smithi072.stdout:Apr 27 00:03:21 smithi072 podman[85709]: 2024-04-27 00:03:21.05012806 +0000 UTC m=+4.563139587 container cleanup 97c724a655b48d7f0ef0873189c364fd980ad547f4356a39593a78778676cce9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-a, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True) 2024-04-27T00:03:21.291 INFO:journalctl@ceph.mds.a.smithi072.stdout:Apr 27 00:03:21 smithi072 bash[85709]: ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-a 2024-04-27T00:03:21.790 INFO:journalctl@ceph.mds.a.smithi072.stdout:Apr 27 00:03:21 smithi072 podman[85723]: 2024-04-27 00:03:21.419206215 +0000 UTC m=+0.748990637 container remove 97c724a655b48d7f0ef0873189c364fd980ad547f4356a39593a78778676cce9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-a, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T00:03:21.982 DEBUG:teuthology.orchestra.run.smithi072:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mds.a.service' 2024-04-27T00:03:22.088 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:03:22.088 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-27T00:03:22.089 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-27T00:03:22.089 DEBUG:teuthology.orchestra.run.smithi072:> sudo systemctl stop ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mds.d 2024-04-27T00:03:22.290 INFO:journalctl@ceph.mds.d.smithi072.stdout:Apr 27 00:03:22 smithi072 systemd[1]: Stopping Ceph mds.d for 3c8453ae-0425-11ef-bc93-c7b262605968... 2024-04-27T00:03:23.040 INFO:journalctl@ceph.mds.d.smithi072.stdout:Apr 27 00:03:22 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-d[67102]: 2024-04-27T00:03:22.678+0000 7f7ae6c00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:03:23.041 INFO:journalctl@ceph.mds.d.smithi072.stdout:Apr 27 00:03:22 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-d[67102]: 2024-04-27T00:03:22.678+0000 7f7ae6c00700 -1 mds.d *** got signal Terminated *** 2024-04-27T00:03:27.039 INFO:journalctl@ceph.mds.d.smithi072.stdout:Apr 27 00:03:26 smithi072 podman[85828]: 2024-04-27 00:03:26.695034274 +0000 UTC m=+4.143619780 container died 7714218b798c9c80bb522d36f40dc48c40d38882aad1778e8918cefac5563bc7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-d, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T00:03:27.290 INFO:journalctl@ceph.mds.d.smithi072.stdout:Apr 27 00:03:27 smithi072 podman[85828]: 2024-04-27 00:03:27.208799073 +0000 UTC m=+4.657384610 container cleanup 7714218b798c9c80bb522d36f40dc48c40d38882aad1778e8918cefac5563bc7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-d, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2) 2024-04-27T00:03:27.291 INFO:journalctl@ceph.mds.d.smithi072.stdout:Apr 27 00:03:27 smithi072 bash[85828]: ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-d 2024-04-27T00:03:27.791 INFO:journalctl@ceph.mds.d.smithi072.stdout:Apr 27 00:03:27 smithi072 podman[85841]: 2024-04-27 00:03:27.477883622 +0000 UTC m=+0.765263424 container remove 7714218b798c9c80bb522d36f40dc48c40d38882aad1778e8918cefac5563bc7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-d, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-27T00:03:27.934 DEBUG:teuthology.orchestra.run.smithi072:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mds.d.service' 2024-04-27T00:03:28.038 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:03:28.038 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-27T00:03:28.039 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-27T00:03:28.039 DEBUG:teuthology.orchestra.run.smithi072:> sudo systemctl stop ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mds.g 2024-04-27T00:03:28.290 INFO:journalctl@ceph.mds.g.smithi072.stdout:Apr 27 00:03:28 smithi072 systemd[1]: Stopping Ceph mds.g for 3c8453ae-0425-11ef-bc93-c7b262605968... 2024-04-27T00:03:29.040 INFO:journalctl@ceph.mds.g.smithi072.stdout:Apr 27 00:03:28 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-g[67714]: 2024-04-27T00:03:28.620+0000 7f6ed7200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:03:29.041 INFO:journalctl@ceph.mds.g.smithi072.stdout:Apr 27 00:03:28 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-g[67714]: 2024-04-27T00:03:28.620+0000 7f6ed7200700 -1 mds.g *** got signal Terminated *** 2024-04-27T00:03:29.911 INFO:journalctl@ceph.mds.g.smithi072.stdout:Apr 27 00:03:29 smithi072 podman[85948]: 2024-04-27 00:03:29.600939938 +0000 UTC m=+1.109810400 container died e329aec6b4c236832ea7582e1823c7a0f91a216ba9f733bb4e26a90b958ef0ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-g, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0) 2024-04-27T00:03:30.262 INFO:journalctl@ceph.mds.g.smithi072.stdout:Apr 27 00:03:30 smithi072 podman[85948]: 2024-04-27 00:03:30.034197728 +0000 UTC m=+1.543068183 container cleanup e329aec6b4c236832ea7582e1823c7a0f91a216ba9f733bb4e26a90b958ef0ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-g, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-27T00:03:30.262 INFO:journalctl@ceph.mds.g.smithi072.stdout:Apr 27 00:03:30 smithi072 bash[85948]: ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-g 2024-04-27T00:03:30.540 INFO:journalctl@ceph.mds.g.smithi072.stdout:Apr 27 00:03:30 smithi072 podman[85962]: 2024-04-27 00:03:30.261211581 +0000 UTC m=+0.645249075 container remove e329aec6b4c236832ea7582e1823c7a0f91a216ba9f733bb4e26a90b958ef0ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-g, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-27T00:03:30.784 DEBUG:teuthology.orchestra.run.smithi072:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mds.g.service' 2024-04-27T00:03:30.879 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:03:30.879 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-27T00:03:30.879 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-27T00:03:30.879 DEBUG:teuthology.orchestra.run.smithi072:> sudo systemctl stop ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mds.j 2024-04-27T00:03:31.290 INFO:journalctl@ceph.mds.j.smithi072.stdout:Apr 27 00:03:31 smithi072 systemd[1]: Stopping Ceph mds.j for 3c8453ae-0425-11ef-bc93-c7b262605968... 2024-04-27T00:03:31.791 INFO:journalctl@ceph.mds.j.smithi072.stdout:Apr 27 00:03:31 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-j[66740]: 2024-04-27T00:03:31.427+0000 7f6901600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:03:31.792 INFO:journalctl@ceph.mds.j.smithi072.stdout:Apr 27 00:03:31 smithi072 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-j[66740]: 2024-04-27T00:03:31.427+0000 7f6901600700 -1 mds.j *** got signal Terminated *** 2024-04-27T00:03:35.195 INFO:journalctl@ceph.mds.j.smithi072.stdout:Apr 27 00:03:34 smithi072 podman[86070]: 2024-04-27 00:03:34.905012087 +0000 UTC m=+3.611939464 container died 16627108e879eaa657538f08fb5e178689d377ac178a67ca122e448247477a0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-j, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-27T00:03:35.540 INFO:journalctl@ceph.mds.j.smithi072.stdout:Apr 27 00:03:35 smithi072 podman[86070]: 2024-04-27 00:03:35.307455476 +0000 UTC m=+4.014382862 container cleanup 16627108e879eaa657538f08fb5e178689d377ac178a67ca122e448247477a0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-j, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-27T00:03:35.540 INFO:journalctl@ceph.mds.j.smithi072.stdout:Apr 27 00:03:35 smithi072 bash[86070]: ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-j 2024-04-27T00:03:35.879 INFO:journalctl@ceph.mds.j.smithi072.stdout:Apr 27 00:03:35 smithi072 podman[86096]: 2024-04-27 00:03:35.627519175 +0000 UTC m=+0.700278649 container remove 16627108e879eaa657538f08fb5e178689d377ac178a67ca122e448247477a0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-j, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T00:03:35.897 DEBUG:teuthology.orchestra.run.smithi072:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mds.j.service' 2024-04-27T00:03:36.008 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:03:36.008 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-27T00:03:36.008 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-27T00:03:36.008 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mds.b 2024-04-27T00:03:36.290 INFO:journalctl@ceph.mds.b.smithi076.stdout:Apr 27 00:03:36 smithi076 systemd[1]: Stopping Ceph mds.b for 3c8453ae-0425-11ef-bc93-c7b262605968... 2024-04-27T00:03:36.549 INFO:journalctl@ceph.mds.b.smithi076.stdout:Apr 27 00:03:36 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-b[62253]: 2024-04-27T00:03:36.289+0000 7f0538f02700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:03:36.550 INFO:journalctl@ceph.mds.b.smithi076.stdout:Apr 27 00:03:36 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-b[62253]: 2024-04-27T00:03:36.289+0000 7f0538f02700 -1 mds.b *** got signal Terminated *** 2024-04-27T00:03:37.832 INFO:journalctl@ceph.mds.b.smithi076.stdout:Apr 27 00:03:37 smithi076 podman[67900]: 2024-04-27 00:03:37.550118415 +0000 UTC m=+1.352687825 container died 66c08b6d66f75919babe496dfc949337489a7d351004f15f44c2ce689b50cfb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, ceph=True, CEPH_POINT_RELEASE=) 2024-04-27T00:03:38.175 INFO:journalctl@ceph.mds.b.smithi076.stdout:Apr 27 00:03:37 smithi076 podman[67900]: 2024-04-27 00:03:37.923330527 +0000 UTC m=+1.725899927 container cleanup 66c08b6d66f75919babe496dfc949337489a7d351004f15f44c2ce689b50cfb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True) 2024-04-27T00:03:38.176 INFO:journalctl@ceph.mds.b.smithi076.stdout:Apr 27 00:03:37 smithi076 bash[67900]: ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-b 2024-04-27T00:03:38.347 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mds.b.service' 2024-04-27T00:03:38.381 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:03:38.381 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-27T00:03:38.381 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-27T00:03:38.381 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mds.e 2024-04-27T00:03:38.926 INFO:journalctl@ceph.mds.e.smithi076.stdout:Apr 27 00:03:38 smithi076 systemd[1]: Stopping Ceph mds.e for 3c8453ae-0425-11ef-bc93-c7b262605968... 2024-04-27T00:03:38.926 INFO:journalctl@ceph.mds.e.smithi076.stdout:Apr 27 00:03:38 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-e[61624]: 2024-04-27T00:03:38.676+0000 7f96c6d20700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:03:38.926 INFO:journalctl@ceph.mds.e.smithi076.stdout:Apr 27 00:03:38 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-e[61624]: 2024-04-27T00:03:38.676+0000 7f96c6d20700 -1 mds.e *** got signal Terminated *** 2024-04-27T00:03:42.318 INFO:journalctl@ceph.mds.e.smithi076.stdout:Apr 27 00:03:41 smithi076 podman[68036]: 2024-04-27 00:03:41.988450223 +0000 UTC m=+3.399017264 container died 6131b82b68ba4690383dafbaf2fce920ca9aeb94d5e6eef1d215de34e949a7a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-e, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424) 2024-04-27T00:03:42.627 INFO:journalctl@ceph.mds.e.smithi076.stdout:Apr 27 00:03:42 smithi076 podman[68036]: 2024-04-27 00:03:42.426423053 +0000 UTC m=+3.836990096 container cleanup 6131b82b68ba4690383dafbaf2fce920ca9aeb94d5e6eef1d215de34e949a7a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-e, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-27T00:03:42.627 INFO:journalctl@ceph.mds.e.smithi076.stdout:Apr 27 00:03:42 smithi076 bash[68036]: ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-e 2024-04-27T00:03:42.850 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mds.e.service' 2024-04-27T00:03:42.882 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:03:42.882 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-27T00:03:42.883 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-27T00:03:42.883 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mds.h 2024-04-27T00:03:43.176 INFO:journalctl@ceph.mds.h.smithi076.stdout:Apr 27 00:03:42 smithi076 systemd[1]: Stopping Ceph mds.h for 3c8453ae-0425-11ef-bc93-c7b262605968... 2024-04-27T00:03:43.675 INFO:journalctl@ceph.mds.h.smithi076.stdout:Apr 27 00:03:43 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-h[61930]: 2024-04-27T00:03:43.212+0000 7fdd4c4bc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:03:43.676 INFO:journalctl@ceph.mds.h.smithi076.stdout:Apr 27 00:03:43 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-h[61930]: 2024-04-27T00:03:43.212+0000 7fdd4c4bc700 -1 mds.h *** got signal Terminated *** 2024-04-27T00:03:45.926 INFO:journalctl@ceph.mds.h.smithi076.stdout:Apr 27 00:03:45 smithi076 podman[68156]: 2024-04-27 00:03:45.549036684 +0000 UTC m=+2.446448750 container died 9efa73adf79457aa174c0d2436e7f7b83a7fabfe7a2cb643c3a3a3d9fd84c224 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-27T00:03:46.297 INFO:journalctl@ceph.mds.h.smithi076.stdout:Apr 27 00:03:46 smithi076 podman[68156]: 2024-04-27 00:03:46.013291421 +0000 UTC m=+2.910703494 container cleanup 9efa73adf79457aa174c0d2436e7f7b83a7fabfe7a2cb643c3a3a3d9fd84c224 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-h, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux ) 2024-04-27T00:03:46.297 INFO:journalctl@ceph.mds.h.smithi076.stdout:Apr 27 00:03:46 smithi076 bash[68156]: ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-h 2024-04-27T00:03:46.445 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mds.h.service' 2024-04-27T00:03:46.476 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:03:46.476 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-27T00:03:46.477 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-27T00:03:46.477 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mds.k 2024-04-27T00:03:46.926 INFO:journalctl@ceph.mds.k.smithi076.stdout:Apr 27 00:03:46 smithi076 systemd[1]: Stopping Ceph mds.k for 3c8453ae-0425-11ef-bc93-c7b262605968... 2024-04-27T00:03:46.926 INFO:journalctl@ceph.mds.k.smithi076.stdout:Apr 27 00:03:46 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-k[61324]: 2024-04-27T00:03:46.807+0000 7f7eace93700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:03:46.926 INFO:journalctl@ceph.mds.k.smithi076.stdout:Apr 27 00:03:46 smithi076 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-k[61324]: 2024-04-27T00:03:46.807+0000 7f7eace93700 -1 mds.k *** got signal Terminated *** 2024-04-27T00:03:51.150 INFO:journalctl@ceph.mds.k.smithi076.stdout:Apr 27 00:03:50 smithi076 podman[68276]: 2024-04-27 00:03:50.821076068 +0000 UTC m=+4.122056924 container died 78176133a811626c2aa111917c971ad460045ac4358f6426369b2e49f9de874c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-k, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-27T00:03:51.425 INFO:journalctl@ceph.mds.k.smithi076.stdout:Apr 27 00:03:51 smithi076 podman[68276]: 2024-04-27 00:03:51.249516848 +0000 UTC m=+4.550497711 container cleanup 78176133a811626c2aa111917c971ad460045ac4358f6426369b2e49f9de874c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-k, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-27T00:03:51.425 INFO:journalctl@ceph.mds.k.smithi076.stdout:Apr 27 00:03:51 smithi076 bash[68276]: ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-k 2024-04-27T00:03:51.673 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mds.k.service' 2024-04-27T00:03:51.676 INFO:journalctl@ceph.mds.k.smithi076.stdout:Apr 27 00:03:51 smithi076 podman[68290]: 2024-04-27 00:03:51.425180393 +0000 UTC m=+0.606358857 container remove 78176133a811626c2aa111917c971ad460045ac4358f6426369b2e49f9de874c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-k, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS) 2024-04-27T00:03:51.676 INFO:journalctl@ceph.mds.k.smithi076.stdout:Apr 27 00:03:51 smithi076 systemd[1]: ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mds.k.service: Deactivated successfully. 2024-04-27T00:03:51.677 INFO:journalctl@ceph.mds.k.smithi076.stdout:Apr 27 00:03:51 smithi076 systemd[1]: Stopped Ceph mds.k for 3c8453ae-0425-11ef-bc93-c7b262605968. 2024-04-27T00:03:51.677 INFO:journalctl@ceph.mds.k.smithi076.stdout:Apr 27 00:03:51 smithi076 systemd[1]: ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mds.k.service: Consumed 1.138s CPU time. 2024-04-27T00:03:51.704 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:03:51.704 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-27T00:03:51.704 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-27T00:03:51.705 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mds.c 2024-04-27T00:03:51.982 INFO:journalctl@ceph.mds.c.smithi136.stdout:Apr 27 00:03:51 smithi136 systemd[1]: Stopping Ceph mds.c for 3c8453ae-0425-11ef-bc93-c7b262605968... 2024-04-27T00:03:52.242 INFO:journalctl@ceph.mds.c.smithi136.stdout:Apr 27 00:03:51 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-c[62838]: 2024-04-27T00:03:51.981+0000 7f3d15f25700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:03:52.242 INFO:journalctl@ceph.mds.c.smithi136.stdout:Apr 27 00:03:51 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-c[62838]: 2024-04-27T00:03:51.981+0000 7f3d15f25700 -1 mds.c *** got signal Terminated *** 2024-04-27T00:03:53.889 INFO:journalctl@ceph.mds.c.smithi136.stdout:Apr 27 00:03:53 smithi136 podman[68716]: 2024-04-27 00:03:53.559087004 +0000 UTC m=+1.669414365 container died 2a2b7325f4c05298470f7cc9e57e78c01d51fb207e59d1f1a640b82c614bc87b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-c, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240424, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-27T00:03:53.889 INFO:journalctl@ceph.mds.c.smithi136.stdout:Apr 27 00:03:53 smithi136 podman[68716]: 2024-04-27 00:03:53.889832556 +0000 UTC m=+2.000159923 container cleanup 2a2b7325f4c05298470f7cc9e57e78c01d51fb207e59d1f1a640b82c614bc87b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-c, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-27T00:03:54.291 INFO:journalctl@ceph.mds.c.smithi136.stdout:Apr 27 00:03:53 smithi136 bash[68716]: ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-c 2024-04-27T00:03:54.291 INFO:journalctl@ceph.mds.c.smithi136.stdout:Apr 27 00:03:54 smithi136 podman[68742]: 2024-04-27 00:03:54.100068753 +0000 UTC m=+0.539559745 container remove 2a2b7325f4c05298470f7cc9e57e78c01d51fb207e59d1f1a640b82c614bc87b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-c, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=) 2024-04-27T00:03:54.310 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mds.c.service' 2024-04-27T00:03:54.346 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:03:54.346 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-27T00:03:54.346 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-27T00:03:54.347 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mds.f 2024-04-27T00:03:54.661 INFO:journalctl@ceph.mds.f.smithi136.stdout:Apr 27 00:03:54 smithi136 systemd[1]: Stopping Ceph mds.f for 3c8453ae-0425-11ef-bc93-c7b262605968... 2024-04-27T00:03:55.047 INFO:journalctl@ceph.mds.f.smithi136.stdout:Apr 27 00:03:54 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-f[62530]: 2024-04-27T00:03:54.660+0000 7f68c0e01700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:03:55.047 INFO:journalctl@ceph.mds.f.smithi136.stdout:Apr 27 00:03:54 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-f[62530]: 2024-04-27T00:03:54.660+0000 7f68c0e01700 -1 mds.f *** got signal Terminated *** 2024-04-27T00:03:57.211 INFO:journalctl@ceph.mds.f.smithi136.stdout:Apr 27 00:03:56 smithi136 podman[68848]: 2024-04-27 00:03:56.931697042 +0000 UTC m=+2.372562742 container died 760eb78752d2666c87570013d9ff5e4bd8dd55b5f56d710468adb892733ab910 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-27T00:03:57.547 INFO:journalctl@ceph.mds.f.smithi136.stdout:Apr 27 00:03:57 smithi136 podman[68848]: 2024-04-27 00:03:57.310849769 +0000 UTC m=+2.751715458 container cleanup 760eb78752d2666c87570013d9ff5e4bd8dd55b5f56d710468adb892733ab910 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-f, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-27T00:03:57.547 INFO:journalctl@ceph.mds.f.smithi136.stdout:Apr 27 00:03:57 smithi136 bash[68848]: ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-f 2024-04-27T00:03:57.690 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mds.f.service' 2024-04-27T00:03:57.721 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:03:57.721 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-27T00:03:57.721 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-27T00:03:57.721 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mds.i 2024-04-27T00:03:58.015 INFO:journalctl@ceph.mds.i.smithi136.stdout:Apr 27 00:03:57 smithi136 systemd[1]: Stopping Ceph mds.i for 3c8453ae-0425-11ef-bc93-c7b262605968... 2024-04-27T00:03:58.297 INFO:journalctl@ceph.mds.i.smithi136.stdout:Apr 27 00:03:58 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-i[63154]: 2024-04-27T00:03:58.012+0000 7f4ecefd9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:03:58.297 INFO:journalctl@ceph.mds.i.smithi136.stdout:Apr 27 00:03:58 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-i[63154]: 2024-04-27T00:03:58.012+0000 7f4ecefd9700 -1 mds.i *** got signal Terminated *** 2024-04-27T00:04:01.889 INFO:journalctl@ceph.mds.i.smithi136.stdout:Apr 27 00:04:01 smithi136 podman[68969]: 2024-04-27 00:04:01.560524981 +0000 UTC m=+3.630807879 container died fb4207c7c94b57c357269e3459e25d38ec13ad58d80548cc37e42d0df28d486c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-i, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-27T00:04:01.889 INFO:journalctl@ceph.mds.i.smithi136.stdout:Apr 27 00:04:01 smithi136 podman[68969]: 2024-04-27 00:04:01.889261609 +0000 UTC m=+3.959544506 container cleanup fb4207c7c94b57c357269e3459e25d38ec13ad58d80548cc37e42d0df28d486c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-i, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-27T00:04:02.250 INFO:journalctl@ceph.mds.i.smithi136.stdout:Apr 27 00:04:01 smithi136 bash[68969]: ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-i 2024-04-27T00:04:02.250 INFO:journalctl@ceph.mds.i.smithi136.stdout:Apr 27 00:04:02 smithi136 podman[68983]: 2024-04-27 00:04:02.132101372 +0000 UTC m=+0.570086841 container remove fb4207c7c94b57c357269e3459e25d38ec13ad58d80548cc37e42d0df28d486c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-i, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-27T00:04:02.265 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mds.i.service' 2024-04-27T00:04:02.297 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:04:02.297 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-27T00:04:02.298 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-27T00:04:02.298 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mds.l 2024-04-27T00:04:02.547 INFO:journalctl@ceph.mds.l.smithi136.stdout:Apr 27 00:04:02 smithi136 systemd[1]: Stopping Ceph mds.l for 3c8453ae-0425-11ef-bc93-c7b262605968... 2024-04-27T00:04:03.047 INFO:journalctl@ceph.mds.l.smithi136.stdout:Apr 27 00:04:02 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-l[62227]: 2024-04-27T00:04:02.642+0000 7fd3f44ae700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-27T00:04:03.047 INFO:journalctl@ceph.mds.l.smithi136.stdout:Apr 27 00:04:02 smithi136 ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-l[62227]: 2024-04-27T00:04:02.642+0000 7fd3f44ae700 -1 mds.l *** got signal Terminated *** 2024-04-27T00:04:04.259 INFO:journalctl@ceph.mds.l.smithi136.stdout:Apr 27 00:04:03 smithi136 podman[69089]: 2024-04-27 00:04:03.98077011 +0000 UTC m=+1.466802475 container died f0dc138430b0426d211e3f19e8be4683bfda5f568ea1427adfd43d23e568f631 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-l, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-27T00:04:04.547 INFO:journalctl@ceph.mds.l.smithi136.stdout:Apr 27 00:04:04 smithi136 podman[69089]: 2024-04-27 00:04:04.359828285 +0000 UTC m=+1.845860712 container cleanup f0dc138430b0426d211e3f19e8be4683bfda5f568ea1427adfd43d23e568f631 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-27T00:04:04.547 INFO:journalctl@ceph.mds.l.smithi136.stdout:Apr 27 00:04:04 smithi136 bash[69089]: ceph-3c8453ae-0425-11ef-bc93-c7b262605968-mds-l 2024-04-27T00:04:04.719 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3c8453ae-0425-11ef-bc93-c7b262605968@mds.l.service' 2024-04-27T00:04:04.750 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-27T00:04:04.750 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-27T00:04:04.750 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-27T00:04:05.070 INFO:teuthology.orchestra.run.smithi072.stdout:Deleting cluster with fsid: 3c8453ae-0425-11ef-bc93-c7b262605968 2024-04-27T00:04:09.832 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-27T00:04:10.027 INFO:teuthology.orchestra.run.smithi076.stdout:Deleting cluster with fsid: 3c8453ae-0425-11ef-bc93-c7b262605968 2024-04-27T00:04:12.574 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-27T00:04:12.764 INFO:teuthology.orchestra.run.smithi136.stdout:Deleting cluster with fsid: 3c8453ae-0425-11ef-bc93-c7b262605968 2024-04-27T00:04:15.090 DEBUG:teuthology.orchestra.run.smithi072:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-27T00:04:15.184 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-27T00:04:15.216 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-27T00:04:15.241 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-27T00:04:15.243 DEBUG:teuthology.misc:Transferring archived files from smithi072:/var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675308/remote/smithi072/crash 2024-04-27T00:04:15.243 DEBUG:teuthology.orchestra.run.smithi072:> sudo tar c -f - -C /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/crash -- . 2024-04-27T00:04:15.347 INFO:teuthology.orchestra.run.smithi072.stderr:tar: /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-27T00:04:15.347 INFO:teuthology.orchestra.run.smithi072.stderr:tar: Error is not recoverable: exiting now 2024-04-27T00:04:15.349 DEBUG:teuthology.misc:Transferring archived files from smithi076:/var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675308/remote/smithi076/crash 2024-04-27T00:04:15.350 DEBUG:teuthology.orchestra.run.smithi076:> sudo tar c -f - -C /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/crash -- . 2024-04-27T00:04:15.388 INFO:teuthology.orchestra.run.smithi076.stderr:tar: /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-27T00:04:15.388 INFO:teuthology.orchestra.run.smithi076.stderr:tar: Error is not recoverable: exiting now 2024-04-27T00:04:15.390 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675308/remote/smithi136/crash 2024-04-27T00:04:15.391 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/crash -- . 2024-04-27T00:04:15.423 INFO:teuthology.orchestra.run.smithi136.stderr:tar: /var/lib/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-27T00:04:15.423 INFO:teuthology.orchestra.run.smithi136.stderr:tar: Error is not recoverable: exiting now 2024-04-27T00:04:15.424 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-27T00:04:15.424 DEBUG:teuthology.orchestra.run.smithi072:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-04-27T00:04:15.527 INFO:tasks.cephadm:Compressing logs... 2024-04-27T00:04:15.527 DEBUG:teuthology.orchestra.run.smithi072:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-27T00:04:15.531 DEBUG:teuthology.orchestra.run.smithi076:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-27T00:04:15.534 DEBUG:teuthology.orchestra.run.smithi136:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-27T00:04:15.559 INFO:teuthology.orchestra.run.smithi136.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph.log 2024-04-27T00:04:15.559 INFO:teuthology.orchestra.run.smithi136.stderr:: No such file or directory 2024-04-27T00:04:15.559 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-27T00:04:15.559 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mds.f.log 2024-04-27T00:04:15.559 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph.log: /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-27T00:04:15.559 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-27T00:04:15.560 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mds.c.log 2024-04-27T00:04:15.560 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-27T00:04:15.560 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mgr.z.log 2024-04-27T00:04:15.560 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mds.i.log 2024-04-27T00:04:15.562 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mgr.z.log: 89.0% -- replaced with /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-27T00:04:15.562 INFO:teuthology.orchestra.run.smithi136.stderr: 90.4% -- replaced with /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-27T00:04:15.567 INFO:teuthology.orchestra.run.smithi076.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph.log 2024-04-27T00:04:15.568 INFO:teuthology.orchestra.run.smithi076.stderr:: No such file or directory 2024-04-27T00:04:15.568 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-27T00:04:15.568 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-27T00:04:15.568 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mds.b.log 2024-04-27T00:04:15.568 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-27T00:04:15.568 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-27T00:04:15.568 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mds.b.log: /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-27T00:04:15.568 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mds.h.log 2024-04-27T00:04:15.569 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mgr.y.log: 89.0% -- replaced with /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-27T00:04:15.569 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-osd.10.log 2024-04-27T00:04:15.569 INFO:teuthology.orchestra.run.smithi076.stderr: 90.3% -- replaced with /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-27T00:04:15.569 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-27T00:04:15.569 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mds.h.log: /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mds.e.log 2024-04-27T00:04:15.569 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph.cephadm.log: 82.5% -- replaced with /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-27T00:04:15.569 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-27T00:04:15.574 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mds.e.log: 93.2% 94.4% -- replaced with /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mds.e.log.gz 2024-04-27T00:04:15.574 INFO:teuthology.orchestra.run.smithi076.stderr: -- replaced with /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-27T00:04:15.574 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-27T00:04:15.574 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mds.k.log 2024-04-27T00:04:15.574 INFO:teuthology.orchestra.run.smithi136.stderr: 94.5% -- replaced with /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mds.f.log.gz 2024-04-27T00:04:15.574 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-27T00:04:15.575 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-osd.11.log 2024-04-27T00:04:15.575 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-osd.8.log 2024-04-27T00:04:15.575 INFO:teuthology.orchestra.run.smithi136.stderr: 82.5% -- replaced with /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-27T00:04:15.575 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-osd.11.log: 93.3% -- replaced with /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mgr.z.log.gz 2024-04-27T00:04:15.575 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-27T00:04:15.575 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mds.l.log 2024-04-27T00:04:15.582 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-27T00:04:15.584 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-27T00:04:15.586 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mds.k.log: 91.0% -- replaced with /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-27T00:04:15.587 INFO:teuthology.orchestra.run.smithi072.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-27T00:04:15.588 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph.log 2024-04-27T00:04:15.589 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mds.l.log: 91.0% -- replaced with /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-27T00:04:15.589 INFO:teuthology.orchestra.run.smithi136.stderr: 94.5% -- replaced with /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mds.l.log.gz 2024-04-27T00:04:15.590 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-27T00:04:15.591 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/cephadm.log: 92.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-27T00:04:15.592 INFO:teuthology.orchestra.run.smithi076.stderr: 94.4% -- replaced with /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mds.k.log.gz 2024-04-27T00:04:15.593 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mds.g.log 2024-04-27T00:04:15.593 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mds.j.log 2024-04-27T00:04:15.593 INFO:teuthology.orchestra.run.smithi072.stderr: 88.9% -- replaced with /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-27T00:04:15.593 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-27T00:04:15.595 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cephadm.log: 91.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-27T00:04:15.595 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-27T00:04:15.595 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-27T00:04:15.596 INFO:teuthology.orchestra.run.smithi072.stderr: 90.2% -- replaced with /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-27T00:04:15.597 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-27T00:04:15.598 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-27T00:04:15.599 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mon.a.log: 94.6% -- replaced with /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mds.j.log.gz 2024-04-27T00:04:15.600 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-27T00:04:15.602 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mds.a.log 2024-04-27T00:04:15.603 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph.cephadm.log: 82.9% -- replaced with /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-27T00:04:15.603 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-27T00:04:15.607 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mds.a.log: /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mds.d.log 2024-04-27T00:04:15.608 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-osd.9.log 2024-04-27T00:04:15.615 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mds.d.log: /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-27T00:04:15.616 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-04-27T00:04:15.625 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/ceph-client.admin.log: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-04-27T00:04:15.626 INFO:teuthology.orchestra.run.smithi072.stderr: 94.3% -- replaced with /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mds.d.log.gz 2024-04-27T00:04:15.629 INFO:teuthology.orchestra.run.smithi072.stderr: 90.9% -- replaced with /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-27T00:04:15.633 INFO:teuthology.orchestra.run.smithi072.stderr: 93.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-27T00:04:15.637 INFO:teuthology.orchestra.run.smithi136.stderr: 94.6% -- replaced with /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mds.c.log.gz 2024-04-27T00:04:15.643 INFO:teuthology.orchestra.run.smithi076.stderr: 94.5% -- replaced with /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mds.h.log.gz 2024-04-27T00:04:15.649 INFO:teuthology.orchestra.run.smithi136.stderr: 94.5% -- replaced with /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mds.i.log.gz 2024-04-27T00:04:15.663 INFO:teuthology.orchestra.run.smithi072.stderr: 93.8% -- replaced with /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mds.g.log.gz 2024-04-27T00:04:15.683 INFO:teuthology.orchestra.run.smithi072.stderr: 94.0% -- replaced with /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mds.a.log.gz 2024-04-27T00:04:15.776 INFO:teuthology.orchestra.run.smithi076.stderr: 93.8% -- replaced with /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mds.b.log.gz 2024-04-27T00:04:15.883 INFO:teuthology.orchestra.run.smithi072.stderr: 90.5% -- replaced with /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-27T00:04:15.964 INFO:teuthology.orchestra.run.smithi076.stderr: 91.9% -- replaced with /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-27T00:04:16.130 INFO:teuthology.orchestra.run.smithi136.stderr: 91.9% -- replaced with /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-27T00:04:16.226 INFO:teuthology.orchestra.run.smithi136.stderr: 92.8% -- replaced with /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-27T00:04:16.278 INFO:teuthology.orchestra.run.smithi136.stderr: 92.6% -- replaced with /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-27T00:04:16.341 INFO:teuthology.orchestra.run.smithi136.stderr: 92.6% -- replaced with /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-27T00:04:16.412 INFO:teuthology.orchestra.run.smithi076.stderr: 92.4% -- replaced with /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-27T00:04:16.443 INFO:teuthology.orchestra.run.smithi072.stderr: 92.7% -- replaced with /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-27T00:04:16.461 INFO:teuthology.orchestra.run.smithi072.stderr: 92.6% -- replaced with /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-27T00:04:16.463 INFO:teuthology.orchestra.run.smithi076.stderr: 92.4% -- replaced with /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-27T00:04:16.637 INFO:teuthology.orchestra.run.smithi072.stderr: 90.4% -- replaced with /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-27T00:04:16.773 INFO:teuthology.orchestra.run.smithi076.stderr: 93.2% -- replaced with /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-27T00:04:17.097 INFO:teuthology.orchestra.run.smithi076.stderr: 92.8% -- replaced with /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-27T00:04:17.099 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-04-27T00:04:17.099 INFO:teuthology.orchestra.run.smithi076.stderr:real 0m1.550s 2024-04-27T00:04:17.099 INFO:teuthology.orchestra.run.smithi076.stderr:user 0m5.029s 2024-04-27T00:04:17.099 INFO:teuthology.orchestra.run.smithi076.stderr:sys 0m0.218s 2024-04-27T00:04:17.121 INFO:teuthology.orchestra.run.smithi136.stderr: 92.8% -- replaced with /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-27T00:04:17.123 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-04-27T00:04:17.123 INFO:teuthology.orchestra.run.smithi136.stderr:real 0m1.575s 2024-04-27T00:04:17.124 INFO:teuthology.orchestra.run.smithi136.stderr:user 0m4.344s 2024-04-27T00:04:17.124 INFO:teuthology.orchestra.run.smithi136.stderr:sys 0m0.170s 2024-04-27T00:04:17.220 INFO:teuthology.orchestra.run.smithi072.stderr: 92.9% -- replaced with /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-27T00:04:17.234 INFO:teuthology.orchestra.run.smithi072.stderr: 92.7% -- replaced with /var/log/ceph/3c8453ae-0425-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-27T00:04:17.237 INFO:teuthology.orchestra.run.smithi072.stderr: 2024-04-27T00:04:17.238 INFO:teuthology.orchestra.run.smithi072.stderr:real 0m1.674s 2024-04-27T00:04:17.238 INFO:teuthology.orchestra.run.smithi072.stderr:user 0m6.031s 2024-04-27T00:04:17.238 INFO:teuthology.orchestra.run.smithi072.stderr:sys 0m0.497s 2024-04-27T00:04:17.239 INFO:tasks.cephadm:Archiving logs... 2024-04-27T00:04:17.240 DEBUG:teuthology.misc:Transferring archived files from smithi072:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675308/remote/smithi072/log 2024-04-27T00:04:17.241 DEBUG:teuthology.orchestra.run.smithi072:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-27T00:04:18.136 DEBUG:teuthology.misc:Transferring archived files from smithi076:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675308/remote/smithi076/log 2024-04-27T00:04:18.137 DEBUG:teuthology.orchestra.run.smithi076:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-27T00:04:18.887 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675308/remote/smithi136/log 2024-04-27T00:04:18.888 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-27T00:04:19.601 INFO:tasks.cephadm:Removing cluster... 2024-04-27T00:04:19.601 DEBUG:teuthology.orchestra.run.smithi072:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 --force 2024-04-27T00:04:19.918 INFO:teuthology.orchestra.run.smithi072.stdout:Deleting cluster with fsid: 3c8453ae-0425-11ef-bc93-c7b262605968 2024-04-27T00:04:20.556 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 --force 2024-04-27T00:04:20.754 INFO:teuthology.orchestra.run.smithi076.stdout:Deleting cluster with fsid: 3c8453ae-0425-11ef-bc93-c7b262605968 2024-04-27T00:04:21.157 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3c8453ae-0425-11ef-bc93-c7b262605968 --force 2024-04-27T00:04:21.348 INFO:teuthology.orchestra.run.smithi136.stdout:Deleting cluster with fsid: 3c8453ae-0425-11ef-bc93-c7b262605968 2024-04-27T00:04:21.607 INFO:tasks.cephadm:Removing cephadm ... 2024-04-27T00:04:21.607 DEBUG:teuthology.orchestra.run.smithi072:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-27T00:04:21.664 DEBUG:teuthology.orchestra.run.smithi076:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-27T00:04:21.680 DEBUG:teuthology.orchestra.run.smithi136:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-27T00:04:21.695 INFO:tasks.cephadm:Teardown complete 2024-04-27T00:04:21.695 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-27T00:04:21.712 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-27T00:04:21.713 DEBUG:teuthology.orchestra.run.smithi072:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-27T00:04:21.720 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-27T00:04:21.722 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-27T00:04:21.873 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-27T00:04:21.889 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-27T00:04:21.889 DEBUG:teuthology.orchestra.run.smithi072:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-27T00:04:21.897 DEBUG:teuthology.orchestra.run.smithi076:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-27T00:04:21.900 DEBUG:teuthology.orchestra.run.smithi136:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-27T00:04:21.911 INFO:teuthology.orchestra.run.smithi076.stderr:bash: line 1: ntpq: command not found 2024-04-27T00:04:21.913 INFO:teuthology.orchestra.run.smithi136.stderr:bash: line 1: ntpq: command not found 2024-04-27T00:04:21.915 INFO:teuthology.orchestra.run.smithi076.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-27T00:04:21.915 INFO:teuthology.orchestra.run.smithi076.stdout:=============================================================================== 2024-04-27T00:04:21.915 INFO:teuthology.orchestra.run.smithi076.stdout:^* hv01.front.sepia.ceph.com 3 6 377 42 +1329us[+1334us] +/- 47ms 2024-04-27T00:04:21.915 INFO:teuthology.orchestra.run.smithi076.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 45 -502us[ -496us] +/- 62ms 2024-04-27T00:04:21.915 INFO:teuthology.orchestra.run.smithi076.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 46 -1352us[-1346us] +/- 71ms 2024-04-27T00:04:21.915 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T00:04:21.917 INFO:teuthology.orchestra.run.smithi136.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-27T00:04:21.917 INFO:teuthology.orchestra.run.smithi136.stdout:=============================================================================== 2024-04-27T00:04:21.917 INFO:teuthology.orchestra.run.smithi136.stdout:^* hv01.front.sepia.ceph.com 3 6 377 43 +1319us[+1327us] +/- 47ms 2024-04-27T00:04:21.917 INFO:teuthology.orchestra.run.smithi136.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 45 -514us[ -506us] +/- 62ms 2024-04-27T00:04:21.917 INFO:teuthology.orchestra.run.smithi136.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 41 -1361us[-1361us] +/- 71ms 2024-04-27T00:04:21.917 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T00:04:21.942 INFO:teuthology.orchestra.run.smithi072.stderr:bash: line 1: ntpq: command not found 2024-04-27T00:04:21.951 INFO:teuthology.orchestra.run.smithi072.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-27T00:04:21.951 INFO:teuthology.orchestra.run.smithi072.stdout:=============================================================================== 2024-04-27T00:04:21.951 INFO:teuthology.orchestra.run.smithi072.stdout:^* hv01.front.sepia.ceph.com 3 6 377 45 +1356us[+1368us] +/- 47ms 2024-04-27T00:04:21.952 INFO:teuthology.orchestra.run.smithi072.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 40 -421us[ -421us] +/- 62ms 2024-04-27T00:04:21.952 INFO:teuthology.orchestra.run.smithi072.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 105 -1309us[-1301us] +/- 70ms 2024-04-27T00:04:21.952 INFO:teuthology.orchestra.run.smithi072.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-27T00:04:21.953 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-27T00:04:21.972 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-27T00:04:21.973 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-27T00:04:21.982 DEBUG:teuthology.orchestra.run.smithi072:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-27T00:04:22.170 DEBUG:teuthology.orchestra.run.smithi076:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-27T00:04:22.233 DEBUG:teuthology.orchestra.run.smithi136:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-27T00:04:22.297 DEBUG:teuthology.orchestra.run.smithi072:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-27T00:04:22.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T00:04:22.393 DEBUG:teuthology.orchestra.run.smithi076:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-27T00:04:22.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T00:04:22.419 DEBUG:teuthology.orchestra.run.smithi136:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-27T00:04:22.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T00:04:22.445 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-27T00:04:22.454 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-27T00:04:22.471 INFO:teuthology.task.internal:Duration was 2428.827271 seconds 2024-04-27T00:04:22.471 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-27T00:04:22.480 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-27T00:04:22.481 DEBUG:teuthology.orchestra.run.smithi072:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-27T00:04:22.489 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-27T00:04:22.491 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-27T00:04:22.525 INFO:teuthology.orchestra.run.smithi076.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-27T00:04:22.527 INFO:teuthology.orchestra.run.smithi136.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-27T00:04:22.615 INFO:teuthology.orchestra.run.smithi072.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-27T00:04:22.962 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-27T00:04:22.963 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi072.front.sepia.ceph.com 2024-04-27T00:04:22.963 DEBUG:teuthology.orchestra.run.smithi072:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-27T00:04:23.035 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi076.front.sepia.ceph.com 2024-04-27T00:04:23.035 DEBUG:teuthology.orchestra.run.smithi076:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-27T00:04:23.064 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi136.front.sepia.ceph.com 2024-04-27T00:04:23.065 DEBUG:teuthology.orchestra.run.smithi136:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-27T00:04:23.093 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-27T00:04:23.093 DEBUG:teuthology.orchestra.run.smithi072:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-27T00:04:23.101 DEBUG:teuthology.orchestra.run.smithi076:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-27T00:04:23.106 DEBUG:teuthology.orchestra.run.smithi136:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-27T00:04:23.183 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-27T00:04:23.183 DEBUG:teuthology.orchestra.run.smithi072:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-27T00:04:23.189 DEBUG:teuthology.orchestra.run.smithi076:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-27T00:04:23.191 DEBUG:teuthology.orchestra.run.smithi136:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-27T00:04:23.700 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-27T00:04:23.710 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-27T00:04:23.710 DEBUG:teuthology.orchestra.run.smithi072:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-27T00:04:23.795 DEBUG:teuthology.orchestra.run.smithi076:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-27T00:04:23.821 DEBUG:teuthology.orchestra.run.smithi136:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-27T00:04:23.846 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-27T00:04:23.856 DEBUG:teuthology.orchestra.run.smithi072:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-27T00:04:23.865 DEBUG:teuthology.orchestra.run.smithi076:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-27T00:04:23.867 DEBUG:teuthology.orchestra.run.smithi136:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-27T00:04:23.887 INFO:teuthology.orchestra.run.smithi076.stdout:kernel.core_pattern = core 2024-04-27T00:04:23.909 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern = core 2024-04-27T00:04:23.960 INFO:teuthology.orchestra.run.smithi072.stdout:kernel.core_pattern = core 2024-04-27T00:04:24.002 DEBUG:teuthology.orchestra.run.smithi072:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-27T00:04:24.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T00:04:24.047 DEBUG:teuthology.orchestra.run.smithi076:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-27T00:04:24.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T00:04:24.061 DEBUG:teuthology.orchestra.run.smithi136:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-27T00:04:24.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-27T00:04:24.076 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-27T00:04:24.085 INFO:teuthology.task.internal:Transferring archived files... 2024-04-27T00:04:24.086 DEBUG:teuthology.misc:Transferring archived files from smithi072:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675308/remote/smithi072 2024-04-27T00:04:24.086 DEBUG:teuthology.orchestra.run.smithi072:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-27T00:04:24.191 DEBUG:teuthology.misc:Transferring archived files from smithi076:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675308/remote/smithi076 2024-04-27T00:04:24.192 DEBUG:teuthology.orchestra.run.smithi076:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-27T00:04:24.226 DEBUG:teuthology.misc:Transferring archived files from smithi136:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675308/remote/smithi136 2024-04-27T00:04:24.226 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-27T00:04:24.261 INFO:teuthology.task.internal:Removing archive directory... 2024-04-27T00:04:24.261 DEBUG:teuthology.orchestra.run.smithi072:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-27T00:04:24.269 DEBUG:teuthology.orchestra.run.smithi076:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-27T00:04:24.271 DEBUG:teuthology.orchestra.run.smithi136:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-27T00:04:24.318 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-27T00:04:24.327 INFO:teuthology.task.internal:Not uploading archives. 2024-04-27T00:04:24.327 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-27T00:04:24.337 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-27T00:04:24.337 DEBUG:teuthology.orchestra.run.smithi072:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-27T00:04:24.345 DEBUG:teuthology.orchestra.run.smithi076:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-27T00:04:24.347 DEBUG:teuthology.orchestra.run.smithi136:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-27T00:04:24.359 INFO:teuthology.orchestra.run.smithi076.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 27 00:04 /home/ubuntu/cephtest 2024-04-27T00:04:24.366 INFO:teuthology.orchestra.run.smithi136.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 27 00:04 /home/ubuntu/cephtest 2024-04-27T00:04:24.394 INFO:teuthology.orchestra.run.smithi072.stdout: 393229 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 27 00:04 /home/ubuntu/cephtest 2024-04-27T00:04:24.397 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-27T00:04:24.410 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-27T00:04:24.433 INFO:teuthology.nuke:Checking targets against current locks 2024-04-27T00:04:24.460 DEBUG:teuthology.nuke:shortname: smithi072 2024-04-27T00:04:24.460 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-27T00:04:24.483 DEBUG:teuthology.nuke:shortname: smithi076 2024-04-27T00:04:24.483 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-27T00:04:24.488 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi072.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675308', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 23:13:52.638673', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJjeDVEDNe31QcgWdx6OLTtV1zAuB+DPHQaJ6j6g+/nAjD/dAadH8llC8ys0ClkWFVLrJJhBtkVWheGRksqmgtw='} 2024-04-27T00:04:24.505 DEBUG:teuthology.nuke:shortname: smithi136 2024-04-27T00:04:24.506 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-27T00:04:24.512 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi076.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675308', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 23:13:52.645357', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEgNrepYTRHeeVzBI/2EyIIxQBwE9tGQOIgZ019w8WakdlkOJkd1u3IajgCWpBvUlces3W9hZ7ta9NssCwE0E3M='} 2024-04-27T00:04:24.531 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi136.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675308', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 23:13:52.641960', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHVtaGF43LF2b17nJkPADW9nsUJW4D5atFZYbgIcPrLZEgZwPauh8XXD1V0qvGK42lngRI4D2Aj4wyEu35o1OKw='} 2024-04-27T00:04:24.540 INFO:teuthology.orchestra.console.smithi072:Power off 2024-04-27T00:04:24.541 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-27T00:04:24.565 DEBUG:teuthology.orchestra.console.smithi072:power off output: Chassis Power Control: Down/Off 2024-04-27T00:04:24.565 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-27T00:04:24.580 INFO:teuthology.orchestra.console.smithi076:Power off 2024-04-27T00:04:24.581 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-27T00:04:24.595 DEBUG:teuthology.orchestra.console.smithi072:check power output: Chassis Power is on 2024-04-27T00:04:24.609 DEBUG:teuthology.orchestra.console.smithi076:power off output: Chassis Power Control: Down/Off 2024-04-27T00:04:24.609 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-27T00:04:24.624 INFO:teuthology.orchestra.console.smithi136:Power off 2024-04-27T00:04:24.625 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-27T00:04:24.633 DEBUG:teuthology.orchestra.console.smithi076:check power output: Chassis Power is on 2024-04-27T00:04:24.647 DEBUG:teuthology.orchestra.console.smithi136:power off output: Chassis Power Control: Down/Off 2024-04-27T00:04:24.647 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-27T00:04:24.671 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is on 2024-04-27T00:04:28.596 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-27T00:04:28.635 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-27T00:04:28.672 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-27T00:04:28.708 DEBUG:teuthology.orchestra.console.smithi072:check power output: Chassis Power is on 2024-04-27T00:04:28.745 DEBUG:teuthology.orchestra.console.smithi076:check power output: Chassis Power is on 2024-04-27T00:04:28.782 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is on 2024-04-27T00:04:32.710 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-27T00:04:32.745 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-27T00:04:32.784 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-27T00:04:32.824 DEBUG:teuthology.orchestra.console.smithi072:check power output: Chassis Power is off 2024-04-27T00:04:32.855 DEBUG:teuthology.orchestra.console.smithi076:check power output: Chassis Power is off 2024-04-27T00:04:32.894 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is on 2024-04-27T00:04:32.926 INFO:teuthology.orchestra.console.smithi072:Power off completed 2024-04-27T00:04:32.956 INFO:teuthology.orchestra.console.smithi076:Power off completed 2024-04-27T00:04:36.898 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-27T00:04:37.009 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is off 2024-04-27T00:04:37.109 INFO:teuthology.orchestra.console.smithi136:Power off completed 2024-04-27T00:04:37.215 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 806ec36b21129f113ce356c007dd59694d340b37 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/dbench}} duration: 2428.827270746231 failure_reason: 'Command failed on smithi072 with status 110: "sudo TESTDIR=/home/ubuntu/cephtest bash -c ''ceph fs subvolumegroup pin cephfs qa random 0.10''"' flavor: default owner: scheduled_rishabh@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=ea5ac8f5d9454cb0b647089e80b72433 status: fail success: false 2024-04-27T00:04:37.216 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-27T00:04:37.300 INFO:teuthology.run:FAIL